Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[rvallee]

It's probably good that it's still happening but this was supposed to be a different way of doing things, one that specifically addressed the issues that lead to several junk reviews being published, somehow permanently, from biased self-serving ideologues. This is clearly the old way of doing things, in secrecy and behind closed doors.

So if the update is still happening, at least let's dispense from the BS about a new way of doing things, when the only actually different thing here is that it's not going to be published by any of the working groups, which is hard to argue is a net positive. Regardless, this has to be addressed because there is no way to claim this is a different way when it's clearly the same old Cochrane tune.

On the other hand, if this new way of doing things is impossible because of internal politics, because the contributors don't want to accept any change or because of influencing behind the scenes, we need to hear it. Because frankly it looks like the most likely explanation.

 

[Jonathan Edwards]

This is an issue with how the Cochrane editor chooses to prioritize things (leaving a harmful review up whilst allowing its replacement to drift). That may be a concious decision or it maybe that she simply does not consider it important enough to track. Either way it refects very badly on Cochrane as an organization.

Indeed. Despite the lofty objectives it seems Cochrane has no interest in patients' welfare.
We live in a real world, not the world of Jeremy: 'Ad hoc, ad loc and quid pro quo, so little time, so much to know'.

 

[Adrian]

I have said this over and over to Cochrane directly and via Hilda, but it seems to make Cochrane dig its heels in even more. Trying not to take it personally ;-)

I'm not surprised they are digging their heels in. There failures here highlight serious issues with the way the overall organization works, flaws in their processes and a lack of editorial control. It is easier for them to dig their heals in and pretend things are working than to admit to themselves that there are serious flaws and then tackle them. Unless others take notice and it seriously harms the cochrane brand they have no insentive to improve and get things right.

 

[FMMM1]

I would argue this review is important in the context of covid due to the numbers getting Long Covid and the likelihood that the Cochrane review is used to suggest ineffective and potentially harmful treatments to an increasing number of patients.

Loads of dosh for those benefitting from ineffective treatments --- of course there are incentives to leave them in place.

This is an issue with how the Cochrane editor chooses to prioritize things (leaving a harmful review up whilst allowing its replacement to drift).

With respect, I think the real issue is that now that we have an objective appraisal [see reference to Peter Barry above] i.e. that these interventions aren't effective, then public bodies, such as NICE, should be stating that clearly --- .
Cochrane can be a chat room on the outer edge of the digital universe --- it should have no role in decisions made by public bodies, such as NICE.

We've been let down by NICE's working agreement with Cochrane --- we need to challenge the public representatives (Minister etc.) ultimately responsible for decisions like that.

 

[FMMM1]

Exactly!! There is absolutely no reason for it at all. Cochrane have even signed a collaboration agreement with NICE to share data and analyses with each other to avoid the research waste this tortuous and weirdly secretive process of writing a new review with involvement of patients exemplifies. NICE involved patients in the process. Arguably more meaningfully than Cochrane is. I have said this over and over to Cochrane directly and via Hilda, but it seems to make Cochrane dig its heels in even more. Trying not to take it personally ;-)

Please keep asking and thank you for publishing it on social media --- that way it's more likely to come to public attention/decision makers attention. Ultimately that may be the way to force Ministers (and thereby NICE) to disown Cochrane ---- Cochrane can't objectively assess the quality of reviews --- if should have no role in preparing NICE guidelines.

 

[petrichor]

I thought the point of this review was so that stakeholders are kept updated so they can give input and feedback etc. at each stage of the process. If things are actually moving forward they should be updating people on the progress (if it's anything noteworthy)

It didn't take more than a couple of minutes to type that. A committee does not need more than half an hour to check the relevant data.

It ought to be kept in mind that this is a systematic review, done in accordance with the guidance in the cochrane handbook: https://training.cochrane.org/handbook/current. Even if it is obvious the evidence is worthless, all the steps of the systematic review will take a long time, apparently the average is around 6-18 months: https://libguides.massgeneral.org/c.php?g=651077&p=4565528

 

[Jonathan Edwards]

It ought to be kept in mind that this is a systematic review, done in accordance with the guidance in the cochrane handbook: https://training.cochrane.org/handbook/current. Even if it is obvious the evidence is worthless, all the steps of the systematic review will take a long time, apparently the average is around 6-18 months:

But that process has already been done. Literature has been searched and data from every study compiled. I was in a position to know the situation having had access to systematic data. The data do not change.

Systematic reviewing is a good principle in terms of making sure all available data are considered. However, the current Cochrane process is flawed because it uses GRADE. It does not have to use GRADE and it shouldn't because GRADE is rubbish.

If Cochrane insists on continuing to use a rubbish system then the whole thing is pointless. If Cochrane is not able to see that Cochrane is pointless - in fact worse than pointless.

There is nothing scientific about sticking to a process in a handbook. Science does not work by choosing answers found in a handbook, it works by making use of evidence and rational argument to get the best available answer.

 

[petrichor]

But that process has already been done. Literature has been searched and data from every study compiled. I was in a position to know the situation having had access to systematic data. The data do not change.

If you look at the link I provided: https://libguides.massgeneral.org/c.php?g=651077&p=4565528, the literature search and data collection are only some of the stages of the process, and the others also also very time consuming. I'm also not aware of the literature search and data collection for this review already being completed, so if someone could provide a source on that, that would be nice

Systematic reviewing is a good principle in terms of making sure all available data are considered. However, the current Cochrane process is flawed because it uses GRADE. It does not have to use GRADE and it shouldn't because GRADE is rubbish.

If Cochrane insists on continuing to use a rubbish system then the whole thing is pointless. If Cochrane is not able to see that Cochrane is pointless - in fact worse than pointless.

There is nothing scientific about sticking to a process in a handbook. Science does not work by choosing answers found in a handbook, it works by making use of evidence and rational argument to get the best available answer.

I agree that there are issues with systematic reviews, the cochrane handbook, and GRADE. However, the fact of the matter is that Cochrane reviews are meant to follow the Cochrane handbook: https://consumers.cochrane.org/cochrane-and-systematic-reviews, and that the purpose of Cochrane is to conduct systematic reviews: https://www.cochrane.org/evidence. (A systematic review is defined as "a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review." https://guides.temple.edu/c.php?g=78618&p=4178713) It would be extremely difficult to change those things for this review.

 

[Jonathan Edwards]

the literature search and data collection are only some of the stages of the process, and the others also also very time consuming.

But why follow some set of rules laid out by Mass General? This is real life, real ill people. Real life does not follow rules, nor does science, they follow reality.

The time consuming work has been done - collecting all the data and presenting the major parameters in such a way that reviewers can see enough about quality to make decisions. If the quality of the evidence was marginal it might take a lot more time, yes, but it isn't marginal. It is no use as evidence of a positive effect of exercise. The more interesting issues like harms actually come from surveys that would not even be included in a formal systematic trawl - i.e. real life does not work like that. And working out that PACE is a good piece of negative evidence does not come out of the rule book because the rule book does not even consider negative evidence - only the strength of positive evidence. It is crazy, and bears no relation to reality. And people having been saying this for years but those with constipated minds insist on going on using rules.

It would be extremely difficult to change those things for this review.

But that is my whole point. The only value of this review was to get some of the Cochrane people set in these ways to question what they were doing.

 

[petrichor]

But why follow some set of rules laid out by Mass General? This is real life, real ill people. Real life does not follow rules, nor does science, they follow reality.

I'm not saying they need to follow rules set out by Mass General, I'm providing a source on how long systematic reviews take. Here are some other sources too: https://guides.hsict.library.utoronto.ca/c.php?g=430254&p=5018365, https://bmjopen.bmj.com/content/7/2/e012545

The time consuming work has been done - collecting all the data and presenting the major parameters in such a way that reviewers can see enough about quality to make decisions.

Again, I'm not aware of these steps already being done in the context of this review. If you're suggesting that they take the data from other reviews or previously done reviews, then you should say that. However, that isn't standard practice in systematic reviews, and I'm not sure to what extent that is possible.

It would be extremely difficult to change those things for this review.

But that is my whole point. The only value of this review was to get some of the Cochrane people set in these ways to question what they were doing.

Well I can't speak for everyone, but I think many people are actually interested in getting the right result from the review. The right way to do that isn't to try to get Cochrane to do something in this review that isn't actually possible. I don't want people in this forum to be mislead.

I don't think people should mix up how they wish things were, and how things actually are. The fact of the matter is, Cochrane and Cochrane reviews have a particular purpose and work in a certain way, as you can see from the sources I provided earlier. These are the facts. People should not dedicate a great deal of their limited energy pursuing something that isn't actually possible to be done in this review.

 

[Jonathan Edwards]

If you're suggesting that they take the data from other reviews or previously done reviews, then you should say that. However, that isn't standard practice in systematic reviews, and I'm not sure to what extent that is possible.

All the material is available in the existing Cochrane review versions, some published and some not and in the NICE review, published in detail.

Reviews are costly. What possible justification is there for wasting money on repeating searches that could be spent on patient care or useful research?

Standard practice for systematic reviews is just how the systematic review clique like to justify spending people's money as far as I can see. It bears no relation to good practice, as many other commentators have pointed out.

I think many people are actually interested in getting the right result from the review. The right way to do that isn't to try to get Cochrane to do something in this review that isn't actually possible. I don't want people in this forum to be mislead.

I don't follow. Of course we want the right result but that is easy. If you look at all the studies so far collected none of them has the basic structure needed to provide reliable positive evidence of efficacy*. So there is no reason to recommend. Nothing more needs to be said. It is entirely possible.

I don't think people should mix up how they wish things were, and how things actually are.

Surely civilisation is based on rejecting 'how things actually are' if it stinks. They way things are is rubbish. Cochrane has revealed itself to be a self-serving irrelevance when it comes to non-pharmacological treatments. Nobody needs another review. The only people who might have benefited were people at Cochrane who had their eyes opened. I don't think now that will happen.

* I was asked to peer review the 2019 version of the review that was withdrawn following that round of peer review, unsurprisingly. There was nothing there that was usable.

 

[Caroline Struthers]

Cochrane has revealed itself to be a self-serving irrelevance

Many many times over

Nobody needs another review

I have said this over and over, and Hilda Bastian says I am wrong because Cochrane has decided it *is* needed and she is doing what they asked her to do to help make it happen. My latest suggestion, if they insist on producing a review that no one needs, is to use the NICE data and analyses used for the new ME/CFS guideline, as per the collaboration (data-sharing) agreement between Cochrane and NICE.

All my other suggestions have been rejected by Hilda without (as far as I know) being passed on to the independent advisory group or discussed with anyone at Cochrane. Understandable as my suggestions all revolve around the review being a colossal waste of everyone's time, and, don't let us forget, taxpayers' money.

The only people who might have benefited were people at Cochrane who had their eyes opened. I don't think now that will happen

Yes, I was hoping Hilda would open Cochrane's eyes, but it doesn't seem to have happened yet

 

[petrichor]

Surely civilisation is based on rejecting 'how things actually are' if it stinks. They way things are is rubbish. Cochrane has revealed itself to be a self-serving irrelevance when it comes to non-pharmacological treatments. Nobody needs another review. The only people who might have benefited were people at Cochrane who had their eyes opened. I don't think now that will happen.

Yes people should advocate for better things if they don't like them. But people should be aware of what is possible and what isn't, and how things work. I don't want people to believe that certain things are possible in this cochrane review which are simply not possible. And the fact of the matter is, Cochrane produces systematic reviews, which work in a certain way (which can be seen in the handbook: https://training.cochrane.org/handbook/current) If people want to advocate for things to be done in different ways generally or in specific instances, that's fine. But I want people to be properly informed about how all of this works. I should not feel any hesitation about stating things which are simple facts.

 

[Jonathan Edwards]

But people should be aware of what is possible and what isn't, and how things work. I don't want people to believe that certain things are possible in this cochrane review which are simply not possible.

But I want people to be properly informed about how all of this works.

Yes, but things are more complicated.

The original Larun review was very nearly withdrawn. When David Tovey was editor he made it clear that it should be withdrawn. Moreover, he said that internal review within Cochrane by their own experts had confirmed that. People like Bob, Tom, David Tuller and others were directly involved in getting to stage where Cochrane admitted that its methods had failed.

For some reason everything slipped, but it was clear that it was possible to change things. At least the 2019 review was binned. What happened with NICE has also demonstrated that you can change things. An old guideline was eventually overturned through the determination of people, many of whom are members here.

And 'the way the system works' in medicine includes academics who have pointed out the weaknesses of current systematic review methods. Even Lynne Turner-Stokes wrote to the BMJ to say GRADE was no good (for the wrong reasons).

How things work is that patients and their supporters have huge power to change things and have changed things. Maybe in reality the patients have won out with this re-review. The people at Cochrane have realised that they are not going to be able to re-do a review saying exercise works and to say it doesn't would just further tarnish the brand - in their eyes. They seem unaware that the way to untarnished their brand and serve patients' interests is to learn how to do things properly.

If anyone was actually expecting this review to produce a worthwhile document, I agree they should forget it. But we were led to believe that Hilda Bastion had some idea of the realities involved and might at least make people at Cochrane aware that things needed to change.

 

[Trish]

I had hoped that the introduction of the advisory group as a Cochrane innovation for this review would mean that Cochrane was open to the possibility of their standard procedures being questioned and if found inadequate or inappropriate, being modified and new more scientifically and clinically valid approaches being used.

If we are now being told that regardless of any patient and outside expert input, exactly the same processes and decision making procedures as failed before have to be applied, and may as a result reach the same flawed conclusions as before, then what's the point?

 

[petrichor]

If we are now being told that regardless of any patient and outside expert input, exactly the same processes and decision making procedures as failed before have to be applied, and may as a result reach the same flawed conclusions as before, then what's the point?

Well, as Jonathan Edwards has correctly pointed out previously there is actually quite a bit of flexibility in these processes. I am not an expert on this, but I believe there may be enough flexibility to accomodate people's concerns to an extent. And as other various people have pointed out, the last time the review was done, those processes weren't followed correctly. If those processes had been followed correctly, the review would have said the quality of evidence was Low or Very Low.

I think it's good for people to advocate for their concerns, I'm not opposing that at all. I fully encourage it. But that should be balanced with an awareness of how this works.

 

[MSEsperanza]

I'm still too slow to take into account everything that has been said. So apologies for potential redundancy.

I think if Cochrane takes the criticism seriously they have to conclude that a new approach is not only needed to the "publication of evidence" (Soares-Weiser), but also to the way how they generally use to assess evidence.

It has become increasingly difficult to dismiss most of the criticism on Cochrane's exercise review.

It's clear from the correspondence between Cochrane's then editor-on-chief and the last review authors that GRADE and their general methodological approach doesn't properly take into account bias such as expectation bias both from researchers and patients and also report bias. Also, that they are not primarily interested in the question whether statistical results reflect clinical significance. But isn't that the gist of the criticism? I can't see how it's possible to both acknowledge that criticism and further apply Cochrane's usual tools to assess evidence.

To acknowledge that criticism automatically questions much of Cochrane's other past and ongoing work and thus their legitimacy. But it's not Cochrane alone that is questioned.

With regard to non-pharmacological treatments there seems to be a more widely shared awareness of the uselessness of unblinded trials that use solely self-reported outcomes only for surgical procedures. Trials investigating physical therapy/ exercise and psychological treatments rarely apply sound methods to generate robust evidence, and hardly anyone seems to bother.

Now there's the chance to do better, but that would mean that many people must acknowledge to have been wrong, and also that by dismissing justified criticism they added to peoples' suffering, severe suffering. That's not an easy thing to acknowledge. But if people at Cochrane don't acknowledge that, how can they any longer take themselves seriously?

 

[MSEsperanza]

I understand all that, CRG. So why not a brief update saying that instead of silence.

I compare this to DecodeME which has had delays too, some due to the pandemic and some to the team including pwME. All this has been explained with regular updates and none of us are complaining that it's taking a bit longer than originally hoped.

Tweeted vague reassurance that something with the Cochrane review is happening is not adequate.

If we aren't worth that Cochrane let us know every two or three months that they're still working on this, if we aren't worth to be reminded of the occasions to be heard and get proper answers, if the involvement will kept restricted to a very small group of patients and carers chosen by Cochrane and Hilda, I have very little hope that mistakes will actually be acknowledged.

It would be great if Cochrane (and Hilda) were able to accept the chance they have with this pilot' review to get things right not only for people with ME/CFS but for patients with many other illnesses too.

I don't know if an updated review makes sense before we have a better evidence base. But perhaps it could be possible to acknowledge how badly done most of the clinical trials on non-pharmacological treatments are and in addition, that results in this field are often reported in a misleading way?

I think the best way for Cochrane to acknowledge previous mistakes would be to encourage researchers to generate robust evidence by taking into account our criticism and engaging with us to develop better measurements -- see the diverse forum discussions, e.g.:

https://www.s4me.info/tags/research-methodology/

https://www.s4me.info/tags/outcome-measures/

 

[Caroline Struthers]

But perhaps it could be possible to acknowledge how badly done most of the clinical trials on non-pharmacological treatments are and in addition, that results in this field are often reported in a misleading way?

I think the best way for Cochrane to acknowledge previous mistakes would be to encourage researchers to generate robust evidence by taking into account our criticism and engaging with us to develop better measurements -- see the diverse forum discussions, e.g.:

Absolutely. But they refuse to do either of these things, because that is not how their reviews are currently formatted. They are obsessed with their own manualised process for producing the same unreadable and unusable format for reviews. It does not currently have a slot for exposing individual appallingly useless trials like PACE (and most trials of social and psychological interventions). They say advocate for evidence, but by that they mean Cochrane evidence in the form of systematic reviews - they advocate for themselves. I have tried to get them to think about re-purposing reviews to focus on explicitly and graphically demonstrating evidence gaps in areas of importance to patients. This includes where there are no trials at all and/or a preponderance of poor trials like PACE. They have told me that that is beyond their remit, their (volunteer) authors would not want to or have time do it, and so its a job for another agency yet to emerge. At least that is the message I have taken away from banging my head against a brick wall with them over the past few years.

 

[Lou B Lou]

Why are we waiting?
Oh wh-y are we wai-ai-ting?

Feel free to sing along

Every kid knows the tune