MrMagoo
Senior Member (Voting Rights)
It says they follow AMRCguidance
https://www.amrc.org.uk/listing/category/amrcs-principles-of-expert-review
https://www.amrc.org.uk/listing/category/amrcs-principles-of-expert-review
-
New Forum Software Installed (Still a few issues but reopening the forum) More details.
UK:ME Association funds research for a new clinical assessment toolkit in NHS ME/CFS specialist services, 2023
- Thread starter InitialConditions
- Start date
From the AMRC guidance:
It looks as though the MEA should publish the names of the experts who were involved in the decision-making process relating to the PROM project. @Russell Fleming, can you please give us that information?
Jonathan Edwards
Senior Member (Voting Rights)
I don't think it is as simple as that. For a charity of this size the only ongoing 'expert' may be Charles. He will then make use of expert reviewers. So we know the name - Charles Shepherd. The quoted bit is vague because it does not specific the level of involvement or relate that to charity size. Individual peer reviewers are generally not named. That tradition of anonymity may be open to question but in this particular context I think it is right. I think review comments on journal articles would be better openly authored but for review that involves decisions about money - and in effect researchers' careers - it is unreasonable to expect reviewers' names to be known.
Life just isn't this simple.
If we followed your arguments, however reasonable they may sound, we should shut down the MRC and Versus Arthritis and the NIH for completely screwing up 90% of funding for the last 50 years.
If people in science had the critical skills of people here then all biomedical problems would have been solved long ago. They don't. Most really aren't up to the job but have got themselves into a career and need funding. Peer reviewers almost by definition believe in the old stuff that science is there to show needs replacing by new stuff. If everybody else could see why the new stuff is better then it would already be the old stuff. No single argument explains why everything is a mess but put them all together and you begin to understand why most advances in science had actually been thought of 200 years earlier but didn't catch on because the community wasn't bright enough to see how it worked.
I think it would be reasonable to ask the MEA to publish the protocol or detailed funding bid from Tyson, Gladwell et al that persuaded the MEA to fund this project.
I can envisage a scenario where the bid highlighted the NICE suggestion that we need better outcome measures for clinical trials, and they may have made a persuasive case that clinicians currently working in ME/CFS clinics would find it helpful to have better tools for diagnosing and monitoring patients. They could also have put a superficially persuasive case that the team they had assembled was ideally qualified to produce such materials.
I recall when this project started we cautiously welcomed it, as we all recognise that there is a need for better tools.
We mostly didn't realise how problematic this project would be until we started reading Sarah Tyson's contributions on the forum, and becoming seriously alarmed when we saw the PASS questionnaire.
So I don't entirely blame the MEA for funding this project. With a different team it might have been possible to produce some useful materials for clinicians and patients, and to have suggested ways service evaluation should be done and more suitable outcome measures for clinical trials.
What I really cannot accept is the MEA refusing to see that there is a problem.
I can envisage a scenario where the bid highlighted the NICE suggestion that we need better outcome measures for clinical trials, and they may have made a persuasive case that clinicians currently working in ME/CFS clinics would find it helpful to have better tools for diagnosing and monitoring patients. They could also have put a superficially persuasive case that the team they had assembled was ideally qualified to produce such materials.
I recall when this project started we cautiously welcomed it, as we all recognise that there is a need for better tools.
We mostly didn't realise how problematic this project would be until we started reading Sarah Tyson's contributions on the forum, and becoming seriously alarmed when we saw the PASS questionnaire.
So I don't entirely blame the MEA for funding this project. With a different team it might have been possible to produce some useful materials for clinicians and patients, and to have suggested ways service evaluation should be done and more suitable outcome measures for clinical trials.
What I really cannot accept is the MEA refusing to see that there is a problem.
This
I think it's reasonable for members of the MEA (and any research charity for that matter) and people who donate to understand the process that the charity uses to select research projects, and to know who decides to fund a particular project. I think for transparency and accountability it would be good to know who the peer reviewers were, if that is the process that was used, for all projects that are funded.
As Trish says, it would be good to see the proposal. More transparency and accountability usually leads to better outcomes; it would be good if successful proposals were routinely published by charities.
Knowing who made the decision to support the project and why is helpful in understanding why the MEA has seemed to not be able to see the problems with the project as they have unfolded.
Actually members including Trish had concerns quite early on, although certainly Sarah Tyson was welcomed to the forum. So, I don't think the problems were impossible for the MEA to identify. As I've said, Charles surely knows the issues with BACME and subjective outcomes, so it's hard to understand why he would approve the project if he was the sole decision maker.
I'm not suggesting that the MEA should be shut down because it has made what appears to be a poor research funding decision; I don't think anyone is. But, if it is true that the MRC and Versus Arthritis and the NIH have a 50 year track record of screwing up most of their funding decisions, then they might also want to review their systems and see if they can do things better.
Yes, as a number of us have said now, the MEA's response to people's concerns about the project and the investigators (coming on the heels of the poor decision about a trustee appointment) has not inspired confidence. It all seems a bit puzzling.
Jonathan Edwards
Senior Member (Voting Rights)
Yeah and who does the reviewing of the reviewing of the reviewing?
The circular nature of the problem is something budding researchers get to understand pretty early on. As an old boss of mine said 'The system stinks'. But who decides how to unstink it if not the system?
You cannot expect referees to be named. A relatively junior person who is expert in a field may be asked for an opinion and if it is negative and their name is known you can be pretty sure that is the end of their career. Remember those little messages sent by a certain psychiatrist to employers of junior researchers in the past - one of whom nearly go fired and another of whom got hauled up in from of a committee to explain their behaviour.
No, referees who are asked to give an honest opinion on something that may matter £5M to a senior heavyweight, have to be anonymous. And if they are, everyone has to be.
Peter T
Senior Member (Voting Rights)
The MEA has historically taken the approach that avoiding direct conflict with the establishment will achieve more than sacrificing a good working relationship. It can be disputed whether this is always the right course of action, or whether at certain times a line needs to be drawn, but there are good arguments on both sides. At present BACME represent ‘the establishment’ in relation to UK specialist ME/CFS services.
Also though many of us would like to see the current approach of UK specialist services providing very selective short term rehabilitation to a narrow cross section of people with ME/CFS (ie only the mildly or moderately impaired) replaced with a radically different approach that addresses ME in its entirety (ie addressing the full symptom range and practical implications of the condition across the full spectrum of severity), it is very unlikely that in the short term we will see any substantial change in models of service delivery. Consequently there are good arguments for a close working relationship with BACME trying to influence for change from within.
So I don’t necessarily criticise the MEA for initially giving this grant if they felt it could be a way in to promote change in the current UK services, however I am not optimistic that the current research team are open to such change and so far there are no signs that the MEA will achieve via this grant any significant impact on services beyond getting them to modify their vocabulary choices to mask deviation from the ‘new’ NICE guidelines.
Also though many of us would like to see the current approach of UK specialist services providing very selective short term rehabilitation to a narrow cross section of people with ME/CFS (ie only the mildly or moderately impaired) replaced with a radically different approach that addresses ME in its entirety (ie addressing the full symptom range and practical implications of the condition across the full spectrum of severity), it is very unlikely that in the short term we will see any substantial change in models of service delivery. Consequently there are good arguments for a close working relationship with BACME trying to influence for change from within.
So I don’t necessarily criticise the MEA for initially giving this grant if they felt it could be a way in to promote change in the current UK services, however I am not optimistic that the current research team are open to such change and so far there are no signs that the MEA will achieve via this grant any significant impact on services beyond getting them to modify their vocabulary choices to mask deviation from the ‘new’ NICE guidelines.
Kitty
Senior Member (Voting Rights)
Broadly, there seem to be two problems:
- The MEA doesn't appear to accept the potential for harm if this tool goes into use as it is.
- The MEA hasn't addressed the issue of a researcher they funded asking for patient input, then making unprofessional accusations and refusing to engage further when she received it.
The second is interesting. Unless this project is an unusual setup, the grant giving body would have a relationship with Sarah as the researcher, but they wouldn't be managing the project or her work on it. There would be an explicit agreement that the project is delivered as described, on schedule, and on budget (unless variations were agreed mutually, and it's quite normal for that to happen), and an explicit or implicit expectation that it would be conducted in a professional manner and not bring the funding body into disrepute.
The MEA's options in respect of managing the accusations depend on their general policies and the specific conditions of this grant. If these don't contain anything useful about conduct, it might be impractical for them to insist, for instance, that Sarah apologises. The only real option might be the nuclear one, withdrawal of funding on the grounds of loss of confidence; that's a tough choice and may not be justified here.
But in that case, they should engage with us. The work has their badge on it, reports of the researcher's behaviour are in the public domain, and if I were leading the MEA I'd be very concerned about its credibility being undermined. It's not even an especially difficult issue to deal with it, it only needs a willingness to be open to conversation.
It's important for places who don't have specialist services . If starting from scratch this doesn't bode well
Agreed. I struggle for words that will get this across to others with the right nuance but there are certain positions and certain aspects that will necessitate people having to focus on 'defending' whether it be a decision or themselves and so on. There are other questions or issues where the focus can be on 'shall we do or note or ask x, y, z'. Obviously these areas interlink because the former issues might want to be discussions of lessons learned and if there is any clever thing that could be tightened up to avoid such actions (I'm not necessarily talking about something MEA did wrong, nor that they were being allowed to 'anticipate' until you've proof it happens etc).
If it seems like we are criticising 'both' then you end up with a reply that defends eg Sarah's actions being more likely. Of course the longer it goes on then it all becomes unavoidable circle because you have to criticise the failure to have pulled someone up.
As @Peter Trewhitt says it is worth trying things. Sometimes giving people good faith. The issue is when that isn't honoured having systems that make sure just because you did so you aren't lumbered with such a model then claiming it is 'backed by MEA and patients' - both things that can be flagged even if the funding doesn't provide any options for change (and they could do and would have the option of even if they hadn't funded it, and then it turned out to not be as expected/described). BUt it feels like the funding should provide such options on the basis of consultation being part of it.
The worry is that whilst all of this gets allowed to be pretended to be argument - and ignoring is going to allow it to make it look like that because it necessitates the other person has to then repeat themselves and so on - all the time is being lost where change to probably one of the most important things because it is a measure, isn't being discussed in a grown-up way. And I'm so bored of that game/tactic being played on us.
So what if those doing the project are just now non-communicative to changing having done what they've fancied or done, just because MEA backed it initially certainly does not leave them without options or with the only option being to try and get a little movement from something that is intransigent. If it has turned out not to be their project then they can actually step away and say things that are critiques and that is important they remember those choices. And we need to emphasise we are giving space to them of those options and are curious why it isn't happening.
Last edited:
I still think there is a third one - how did this project, which is providing tools for NHS clinics (and PROM tools that will essentially be endorsed by the MEA as developed in consultation with the ME/CFS community and may end up being used to suggest that ineffective treatments and clinics are effective) come to be funded by the MEA? If the MEA wanted to develop a productive relationship with BACME, which is a perfectly fine aim, they could have funded the development of objective outcomes, or a toolkit with a core of objective outcomes. Or, better, they could have supported a joint effort with BACME to get government funds to develop useful outcome measures for government run clinics. While making sure that an influential member of the team could keep BACME under control, in terms of treating the community with respect and producing a result that would not cause harm.
I think it is an important question to ask, because the problems were foreseeable, and, if systems aren't examined and fixed, more donor funds could be directed to counter-productive research, and, arguably, to fixing problems that aren't the ME/CFS community's responsibility. And that's not good for the MEA or people with ME/CFS.
Kitty
Senior Member (Voting Rights)
I can see an argument that the clinics may be expected by the NHS to develop these tools, and if they do, the odds on them doing it really badly are high—so why don't we pre-empt it by developing a decent one and handing it to them?
It's a strategy I've used many a time in my professional life. Okay, mine were often contracts and project partnership agreements, but the principle's the same. Drafting them is a hassle, it takes skill and experience, and if someone offers you an oven-ready one that looks thorough and fair, it's tempting to accept it. It's not really about manipulation (though there can be a bit of that in the shaping and direction of the thing), it's more about saving yourself weeks of tiresome negotiation if the other party offers up something unworkable.
Agreed there are interesting aspects to this. I just don't know how this question can be asked at the same time as the others and it not leading to us not getting an answer to the other questions which potentially something can be done about at this stage.
Because if it is worded as 'why was this project funded' then necessarily the person being asked is having to work with said person and defend in order to defend their own position - all half way house answers to this if asked right now will close down the options on the other questions.
It will also be a question that gets pointed to the person who did in the first place, who will tend to justify and answer that literally in that temporal moment (not a 'when we did we thought x, but now we realise it is y'), because it is basically a question of 'justify', rather than a request to 'fix it' or 'can x be changed' or 'do you still believe in/have confidence in/back..' or a question that might go to a team or others who could look at whether it is 'on track'. They also can't say 'we thought we'd shape it' or whatever because that will end up precluding the 'working with' option.
I think we also have to be aware that time is of the essence and whilst people are working on responses they aren't working on other things (failure demand issue), so too many questions means the one with the biggest implications to them is going to have to come ahead of the next one down, which might be the ones with the tighter deadlines.
Of course what gets complicated is that I know you are rightly asking these questions because if some of these are quite specific objectives that means certain bits can't change or put boundaries in or list what is the purpose/to be achieved in order of priority then that explains and gives clues as to what solutions are possible and other 'needs' are having to be met for it to be useful. I do know that I'd feel more confident if the first step was a pause or something similar so that procrastination doesn't lead to the status quo inadvertently getting signed off. But I guess we'd need to think through and game out how that could get 'used' / be a problem. I can think of all sorts of potential reactions and things that could be going on or suggested behind the scenes that are nudging things certain ways too.
I just don't know how to word things clearly in a question to tip toe around these in a way that isn't going to get misconstrued (other than in back of my head there is something not gamed out about asking for either a debrief to inform future once it has been completed, but how that line makes sure it doesn't close down the bits that might need to be asked if relevant to the 'solutions' part - but even the existence of that means things might need to have 'one eye to' if that has more implications for them). That doesn't mean there isn't one.
I agree, which is why I wrote the 'arguably' in the bit about doing something that isn't the MEA's responsibility. But, the argument in favour of the MEA taking action loses validity if, instead of leaving the NHS to do it really badly, the MEA uses precious donor funds to pay a team to, well, do it really badly.
Really, all the MEA Trustees need to do to deal with this issues is say something like "We have heard from the community that there are some existing and potential problems with the aims and operation of this research project. We are going to do an internal review to see if our processes for evaluating research proposals worked well in this instance and if we need to take action to ensure that the outcome of this project is indeed going to benefit people with ME/CFS. When we have done that, which we expect will take around 3 months, we will inform the ME/CFS community of our findings and our actions."
And then they need to follow through and do exactly that, avoiding producing a whitewashing report.
It certainly doesn't need to be the end of the MEA or even a terribly big problem. Just deal with the issue, find out what should have been and can be better, fix things, and people can move on.
And then they need to follow through and do exactly that, avoiding producing a whitewashing report.
It certainly doesn't need to be the end of the MEA or even a terribly big problem. Just deal with the issue, find out what should have been and can be better, fix things, and people can move on.
Jonathan Edwards
Senior Member (Voting Rights)
I would suggest that there is no such thing and never will be.
Any assessment is for a purpose. There are myriad purposes and each will be suited best by a different assessment. The whole thing is a mirage.
if we have an idea more specifically of what we think is needed /could be useful, then perhaps phrasing a question along those terms is the angle/starter for ten? or a potential one for the list to choose from
Kitty
Senior Member (Voting Rights)
Oh, I agree. I'm just trying to second guess the thinking, and the idea of producing something focused on patients' needs and realities might have looked appealing.
Many of us have probably started conversations like that at some point, but when the idea's pulled apart it becomes clear it's a non-starter. But if you don't do that, or do it without inviting input from the parties involved, you risk ending up with something like this.