Caroline Struthers' correspondence and blog on the Cochrane Review: 'Exercise therapy for chronic fatigue syndrome, 2017 and 2019, Larun et al.

 
Last edited by a moderator:
Just wanted to drop an older not-yet-posted comment I found in my drafts folder (on David Tovey's answer regarding objective outcomes in his response to Caroline Struthers' complaint) :

@Caroline Struthers:
"Using only subjective outcomes in unblindable trials is terrible methodology. This is even more inexplicable in a review. And avoidable. Some of the included trials, including PACE, did measure objective outcomes, such as employment levels. They didn’t support the positive findings from subjective measures."

David Tovey:
"The review was conducted in a manner that was consistent with the review protocol. In fact, objective measures such as resource use were reported when they were identified. To describe this as ‘terrible methodology’ is simply an opinion and it is not shared by independent methodologists who we consulted, or the Cochrane Handbook.

"It is common for Cochrane Reviews to use patient-reported outcomes as primary outcomes for chronic conditions. When the primary complaint is a private experience (e.g., pain, fatigue, anxiety) the most appropriate outcome is frequently the direct observation and report of patient experience by the patient, for whom it matters most. Obviously, there are limitations and risk of bias"

Others have pointed out and criticized the gist of this answer -- the idea that using only subjective outcomes in unblindable trials is OK and to describe this as terrible methodology is just an opinion. I think Jonathan's comment is most illustrative:
Jonathan Edwards said:
Today I managed to canvas the opinion of the members of the University College London Department of Medicine at Grand Rounds. The attendance was good, about 80, for a presentation on medical negligence. I asked of the statement:

An experiment with subjective outcome measures not blinded to test versus control is unreliable and therefore unsatisfactory.

Do you agree or disagree?

Nobody disagreed. All agreed bar one abstention.
I deliberately made the statement general because this is a general principle for science, not just for trials. The abstainer pointed out that more detail might affect his opinion and it is true that there can be mitigating factors.

So although this is an opinion it is more or less universally held by a body of academics with no vested interest in any particular case.
Click to expand...

I realized only now that, in addition, Tovey rejected Caroline's criticism by saying objective measures actually were reported. However, the only objective measure he mentions is resource use.

Apologies if that also has already been mentioned -- I now wonder, on the one hand, if this is true: Which objective outcomes did the review take into account?

On the other hand, I wonder about resource use as an adequate indicator of treatment effect, especially if one of the treatment effects is that patients learnt that they won't get any help from health services but instead get worse.

Reduced resource use might even correlate with giving positive feedback about one's health -- just say that you're better so the therapist or doctor won't keep misunderstanding and insisting you need more of their treatment.

(Put his here since I am still not able to read much and don't want to jam the active threads where people keep repeating their and others' substantial criticism as well as adding new valuable comments that might help improve things with regard to the Cochrane saga.)
 
Last edited:
MSEsperanza said:
I realized only now that, in addition, Tovey rejected Caroline's criticism by saying objective measures actually were reported. However, the only objective measure he mentions is resource use.
Click to expand...
Using and reporting objective outcomes is only half the story. They have to be taken fully into account, not ignored, downplayed, misrepresented, and then finally claimed to be irrelevant anyway.
 
MSEsperanza said:
Apologies if that also has already been mentioned -- I now wonder, on the one hand, if this is true: Which objective outcomes did the review take into account?
Click to expand...
These are the measures used in the review. Nothing objectively measured except potentially adverse events - which weren't reported properly in the trials, dropouts, also not reported properly I don't think, sleep if measured by polysomnography, and resource use. I was a "consumer" referee on a Cochrane review of a drug for chronic pain once. I mentioned that they hadn't included outcomes I considered important - it might have been work/study status or something like that, and maybe quality of life... ? Can't remember. I was told there was no point including these outcomes because the studies didn't measure them. That's the whole point isn't it?! Cochrane really don't get that they could and should be highlighting these sorts of omissions to help improve primary research so future studies do measure meaningful outcomes. In this case, some of the studies did measure objective outcomes, but the Cochrane reviewers chose not to include them in the review.
upload_2020-5-25_10-52-36.png
 
Caroline Struthers said:
I mentioned that they hadn't included outcomes I considered important - it might have been work/study status or something like that, and maybe quality of life... ? Can't remember. I was told there was no point including these outcomes because the studies didn't measure them. That's the whole point isn't it?!
Click to expand...
That's like a car MOT tester not testing the brakes because the car doesn't have any!

Cochrane seem content to be driven by the scientists and not by the science.
 
Last edited:
Trish said:
Cochrane seem content to be driven by the scientists quacks and not by the science.
Click to expand...
Yes, but my meaning is that even where trials have been run by good scientists genuinely striving for good science, there are still going to be trials where they have messed up. Cochrane is supposed to take a wholly independent approach, and not be influenced by the scientists themselves, else Cochrane cease to be independent and frankly lose their right to claim any sort of credibility.

So when reviewing any and all trials, no matter who the scientists are, Cochrane must always guard against blindly transferring bias resident in trials under review, out into their own review. @Caroline Struthers example of Cochrane deliberately ignoring the absence of something crucial - because it was not there! - is a pantomime-absurd example of what a strangely unscientific collective mindset Cochrane seem to have.
 
Barry said:
Cochrane is supposed to take a wholly independent approach, and not be influenced by the scientists themselves
Click to expand...
Is it really? Giving veto to the authors of a review over retraction certainly doesn't support that, especially when it's clear that the review is completely unreliable and does not meet any acceptable standards, especially a veto over a review that was essentially rating their own grading of their own homework on a course they invented themselves with zero supervision over any part of that process. The authors basically own the review.

Cochrane seems content with "don't ask me I just work here" and behaving as if the authors are threatening their puppies or something. Same thing with most BPS papers frankly, editors are content with pretending that they have no control over what they publish and because the concerns were dismissed by the authors it's out of their hands. Horton basically did exactly that with the PACE papers, taking no editorial position as if the authors were in charge. Authority without responsibility.

What seems apparent is that there simply is no accountability or quality control for BPS "research". Everything goes. No one takes any position other than leaving everything to the authors and if they choose to dismiss the concerns then that's all there is to it. The substance of the papers isn't even evaluated, it's entirely a style review. If there is a review at all beyond reading the abstract or being a like-minded pal of the authors. Or when the authors themselves rate their own grading of their own work of their own invented subject matter.

It's all a mockery of the scientific process.
 
Latest blog from @Caroline Struthers
On 18 November I wrote a rapid response on the BMJ article “NICE backtracks on graded exercise therapy and CBT in draft revision to CFS guidance” It called for Cochrane to withdraw the reviews, as following the conclusions of both reviews could lead to patient harm.

According to the Cochrane policy on reporting serious errors in reviews, I notified the Editor in Chief Karla Soares-Weiser on 19 November.

Our correspondence is below. I was informed today that Cochrane intend to conclude the investigation into the errors in both reviews by January 2021.
Click to expand...
https://healthycontrolblog.wordpress.com/2020/11/24/request-to-withdraw-cochrane-reviews/
 
Have written again to the Cochrane Editor in Chief this morning re-submitting my notification of a serious error which should lead to withdrawal of the reviews. I also forwarded it to Hilda Bastian so she can add it to the correspondence about the Exercise review update.


Subject: Notification of a serious error in reviews Exercise therapy for chronic fatigue syndrome and CBT for chronic fatigue syndrome: resubmission

Dear Karla (cc Rachel Marshall)

According to Cochrane’s policy on withdrawing published reviews (1), and the procedure which states that The Editor in Chief must be notified of all suspected or confirmed serious errors (2) I am resubmitting my notification to you of a serious error in the Cochrane reviews Exercise therapy for chronic fatigue syndrome (3) and CBT for chronic fatigue syndrome (4) which should lead to them being withdrawn.

For the avoidance of doubt, the criteria met for withdrawal in this case is one of the examples of a serious error specified in Cochrane’s own policy documents (1, 2)

“Following the conclusions of the review could result in harm to patients”

As I informed you in my original notification on 19 November 2020, the NICE draft guidelines (5) have reversed the recommendations for Exercise therapy and CBT to treat ME/CFS because of lack of good quality evidence of benefit and recent evidence of harm (6, 7). Therefore, following the conclusions of the Cochrane reviews which state that Exercise therapy and CBT may be helpful for ME/CFS is continuing to put patients in harm’s way.

Please will you acknowledge receipt of this notification and let me know when you anticipate concluding your investigation and deciding a course of action.

With best wishes

Caroline

1. Withdrawing published Cochrane Reviews last modified on Jul 30, 2019 [cited 5 Feb 2021] https://documentation.cochrane.org/display/EPPR/Withdrawing+published+Cochrane+Reviews
2. Serious errors in published Cochrane Reviews last modified on Jul 05, 2019 [cited 5 Feb 2021 https://documentation.cochrane.org/display/EPPR/Serious+errors+in+published+Cochrane+Reviews
3. Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database of Systematic Reviews 2019, Issue 10. Art. No.: CD003200. DOI: 10.1002/14651858.CD003200.pub8.
4. Price JR, Mitchell E, Tidy E, Hunot V. Cognitive behaviour therapy for chronic fatigue syndrome in adults. Cochrane Database of Systematic Reviews 2008, Issue 3. Art. No.: CD001027. DOI: 10.1002/14651858.CD001027.pub2.
5. Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management In development [GID-NG10091] Expected publication date: 21 April 2021 https://www.nice.org.uk/guidance/indevelopment/gid-ng10091/documents
6. Geraghty K, Hann M, Kurtev S. Myalgic encephalomyelitis/chronic fatigue syndrome patients’ reports of symptom changes following cognitive behavioural therapy, graded exercise therapy and pacing treatments: Analysis of a primary survey compared with secondary surveys. Journal of Health Psychology. 2019;24(10):1318-1333. doi:10.1177/1359105317726152
7. ME Association. Consolidated report: Evaluation of a survey exploring the experiences of adults and children with ME/CFS who have participated in CBT and GET interventional programmes, April 2019: https://meassociation.org.uk/wp-con...nd-GET-Final-Consolidated-Report-03.04.19.pdf
 
Hi All

I have heard back from Cochrane about my complaint to the "Cochrane Library Oversight Committee".

They have changed the policy so that complaints about the Editor in Chief are now handled by the CEO (her boss) Mark Wilson - see his letter attached. The main upshot is MW has recommended add an extra bit to the editorial note saying the review should not be used for clinical decision-making.

upload_2021-2-26_9-50-25.png

Pretty ironic as Cochrane's strap line is "Trusted evidence, informed decisions, better health"

Even if they follow his recommendation, the note will not visible here https://www.cochrane.org/CD001027/DEPRESSN_cognitive-behaviour-therapy-chronic-fatigue-syndrome which is the main window to the review

So my next step is to complain to COPE https://publicationethics.org/facilitation-and-integrity-subcommittee which I will do asap.
 

Attachments

Caroline Struthers said:
Yes, I will definitely wait. I am anticipating COPE not doing anything very much, so thinking about the next step regulator-wise
Click to expand...
Well, "not very much", or rather "nothing" is exactly what COPE are doing. It's out of scope, apparently. Editorz rule OK! I have attached their ruling on my complaint, and my response to it. I have also attached the report which contains Cochrane's answers to their queries. Onwards and upwards.
 

Attachments

I just wrote to Cochrane again, mainly in response to the announcement that NICE and Cochrane were going to be teaming up. I have sent it to Hilda to add to her database of articles about the Exercise review. It covers a lot of bases and it will be interesting to see what response I get, if any. https://healthycontrol.org/2021/11/26/letter-to-toby-lasserson-cochrane/ (NB I address Toby by his first name as I know him from when I worked at Cochrane)
 
I have had a reply to my letter to Toby Lasserson. I was planning to follow up today to ask why they have removed the "used in" link from the reviews to the NICE Clinical Knowledge Summary but not to the new NICE ME/CFS guidelines which explicitly say the Cochrane reviews were excluded. I will do that now.

Dear Caroline,

Thank you for your recent message addressed to the Deputy Editor in Chief of The Cochrane Library.

As you will be aware, Cochrane continues to support the ongoing review of exercise therapy with a new review team, in conjunction with an Independent Advisory Group. The decision to retain the amended review from 2019 will not be reconsidered by the Editor in Chief, nor will any further action be taken relating to other reviews implied by the Editorial Note posted in the review of cognitive behavioural therapy. We are in discussion with colleagues at Cochrane UK as to the how links to guidelines and other dissemination products such as the Clinical Knowledge Summaries for the reviews of exercise therapy and Cognitive Behavioural Therapy are best displayed on the Cochrane Library.

NICE have confirmed that details of the collaborative approach to support the use of both Cochrane reviews and the topic expertise of Cochrane review authors, groups and networks in the development of NICE guidelines will be included the planned update to the NICE Guidelines Manual expected in 2022.

Best wishes,

Katie Abbotts
Communications and External Affairs
Cochrane

Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX UK

www.cochrane.org

Trusted evidence. Informed decisions. Better health.
 
Have finally got round to starting work on the complaint to Cochrane about the Editor in Chief ignoring the advice of arbitrator Gordon Guyatt. He said allowing the exercise review authors to continue to rate the quality of evidence on one outcome (fatigue) as moderate was contingent on highlighting the lack of important effect. I want assurance from the new CEO (Catherine Spencer) that the complaint will be investigated properly, so I wrote to her yesterday (3 January 2023) as follows


Dear Catherine

I am writing in a personal capacity and not as a representative of my employer.

I wish to make a complaint about Dr Karla Soares-Weiser, Editor in Chief of the Cochrane Library. Your procedure stipulates that the complaint should be directed to you. In the past I have complained both to and about the Editor-in-Chief, including the previous incumbent, Dr David Tovey.

In a quest for an independent investigation I have (I think) twice appealed to a committee called the Cochrane Library Oversight Committee. When it was formed in around 2014 it seemed to be the ideal way for complaints about the Editor in Chief to be dealt with independently. Now the Editor in Chief has been made a member of that committee so the independence is gone. https://community.cochrane.org/organizational-info/people/cochrane-library-oversight-committee. I would be interested to know what if any issues have ever been resolved by this committee. I don’t understand why minutes from their meetings are not made available. Please could you chase up my question about its membership which still does not include “at least one health consumer advocate” despite assurances that this was being addressed (see attached email correspondence. I have not had any response since May 2021 despite polite chasing)

Please can you let me know how a complaint about the Editor in Chief will be investigated independently? I need more information than “Cochrane will be guided by COPE's core practices and guidelines when handling complaints”. Please point me to the exact COPE practice guidelines you will follow when you investigate my complaint. I think it would be a good idea to develop this more explicitly and post it on the complaints page to aid people wishing to raise concerns https://www.cochranelibrary.com/help/complaints. The HRA seem to have a reasonably good procedure in place https://www.hra.nhs.uk/about-us/gov...concerns/hra-complaints-policy-and-procedure/

As advised on your complaint page, I have also appealed to COPE on a related but separate complaint which was not handled independently and it seems COPE ensure journals have followed their own procedures and investigating editorial misconduct is beyond their remit.

Once I have assurance the complaint will be investigated independently and how it will be conducted, I will submit it to you.

With best wishes


Caroline
 
This is the latest correspondence with Cochrane after I wrote to say I was preparing a complaint on 3 January. Getting hold of the email address of the new CEO sped things up considerably.



So I now have to get on and submit the complaint itself. To remind everyone, it will be specifically about Karla allowing the authors of the exercise review to ignore Guyatt's instruction that they explicitly state that exercise has no meaningful positive effect on fatigue.
 
Caroline Struthers said:
I have had no reply to this sent to the IAG address on 24 Feb. I could now forward it to the CEO?
Click to expand...
Have just written to Catherine Spencer

Dear Catherine (cc Cochrane.iag@gmail.com, exerciseforme-cfsupdates@cochrane.org)

I am writing in a personal capacity and not as a representative of my employer.

I wish to complain about the poor management of Cochrane’s “Stakeholder engagement in high-profile reviews pilot” (https://community.cochrane.org/orga...eholder-engagement-high-profile-reviews-pilot)

Despite Karla Soares-Weiser’s assurances before the process began (https://www.cochrane.org/news/cfs), there has been slow progress, lack of clarity, and no update AT ALL since 2021. This is unacceptable. As far as I can tell, all attempts to find out what’s going on have been unanswered or received a stalling response. My latest email to the Independent Advisory Group was 24 February (see below) and has also remained unanswered. I wrote in 2022 and received no response (see attached). I have also emailed the Cochrane Editorial address which is supposes to keep us updated on progress, and received no response (see attached).

As you know, I am planning to send another complaint about Karla Soares-Weiser which is related to this review, but not connected with this complaint. I will do this in the next few days.

Best wishes

Caroline
 
You must log in or register to reply here.
Back
Top Bottom