David Tuller: Trial By Error: Cochrane's Report on Courtney's Complaint

Kalliope

Kalliope

Senior Member (Voting Rights)
Cochrane's Report on Courtney's Complaint

I now have a copy of Cochrane’s report about Courtney’s complaint. It is clear from the analysis that Cochrane’s concerns involved the scientific accuracy and integrity of the exercise review. Contrary to the claims of the exercise review’s defenders, Cochrane was not responding to a hysterical mob of patients but to the cogent and well-reasoned argumentation of one patient in particular.
 
What are Cochrane's reasons to reject and bury this report? It makes for a completely different review.

Where does it come from? Or is it sources and methods?

How does it respect Cochrane's policies to bury evidence of possible harm and deterioration and the true nature of bias and strength of evidence? This is basically complete distortion of the quality of evidence, the difference between high and low overall quality.

How likely is it we are going to find the same from the Lancet about PACE and PLOSone, especially about the release of the data on the latter?

Are there any steps we can take collectively to accelerate something, anything? There are cascading consequences that can come as a result of this move and the time for denial is over. Someone needs to answer all the questions and fast.
 
rvallee said:
What are Cochrane's reasons to reject and bury this report? It makes for a completely different review.

Where does it come from? Or is it sources and methods?

How does it respect Cochrane's policies to bury evidence of possible harm and deterioration and the true nature of bias and strength of evidence? This is basically complete distortion of the quality of evidence, the difference between high and low overall quality.

How likely is it we are going to find the same from the Lancet about PACE and PLOSone, especially about the release of the data on the latter?

Are there any steps we can take collectively to accelerate something, anything? There are cascading consequences that can come as a result of this move and the time for denial is over. Someone needs to answer all the questions and fast.
Click to expand...

I'm not sure what "sources and methods" mean, but it is from a reliable source. Why Cochrane isn't open with this kind of stuff is a question, but I guess they would call it "internal work" or something like that. Cochrane doesn't like a PR scandal.

Someone did leak e-mails to Reuters last fall. I don't think Cochrane wasn't very happy with that. The troubling part is that only one side of the story got out. Seems like the Reuters journalist didn't read this complaint. That is a problem.
 
rvallee said:
What are Cochrane's reasons to reject and bury this report? It makes for a completely different review.

Where does it come from? Or is it sources and methods?

How does it respect Cochrane's policies to bury evidence of possible harm and deterioration and the true nature of bias and strength of evidence? This is basically complete distortion of the quality of evidence, the difference between high and low overall quality.
Click to expand...

To me, it looks like Cochrane is trying to avoid embarrassing Larun and acknowledging that they got things wrong and that their processes failed. Courtney's complaint was so solid that they can't pretend he was wrong, but they certainly don't seem overly concerned that Larun was wrong in the first-place and then responded to Courtney pointing out her failings with dismissive "we must agree to disagree" nonsense. I don't think that they've 'buried' the report recognising Courtney was right (a report that was really unnecessary imo - Courtney clearly explained himself so that anyone could see for themselves that he was right) but they don't seem overly concerned by the fact that Larun has shown herself to be, at best, incompetent, or that her Cochrane review is continuing to cause patients problems.

dave30th said:
I've just added an update to the post. Norway has a website where agencies list their documents and correspondence and people--anyone--can ask to see them.
Click to expand...

That is a level of transparency that amazed me!

I just searched for 'lightning process'... 90 results: https://www.einnsyn.no/sok?offset=25&searchTerm=lightning process
 
Last edited:
Thank you, @dave30th, for bringing this report to light!
I wish Cochrane added this report at a prominent place to the review.

Have I understood properly that the person in charge of assessing Bob's criticism agreed with all of the key points Bob made?

P.S.: Typos in the added update: "That is also the month when Dr Tovey leaves his position as Cochrane’s as editor-in-chiefl it..."

Edit: I think I don't understand this paragraph:

While the Cochrane report spells out many of the issues, it unfortunately does not include discussion of the authors’ decision to omit objective outcomes from the review. In responding to this criticism, Larun justified the omission by noting that these outcomes were not included in the review protocol.
Click to expand...

Does this mean Bob's formal complaint included the omission of objective outcomes, but this point was not recognized in the Cochrane report as a key issue because Larun's justification had been accepted? Or was this issue not addressed in the report because all other comments, regardless how substantial they were, were not assessed with the same thoroughness as the formal complaint was?
 
Last edited:
So lets see Alem made himself much worse getting the PACE data, Bob dies from bad healthcare and perhaps his imprisonment and forced exercise.
Funny how both cases are about pushing beyond our envelope, what the PACErs claim should be curing us. Both these people should be alive and healthy by their energy expenditures :emoji_face_palm:
These two paid the price while the perpetrators claim they are the real victims

@dave30th A story idea?
 
Last edited:
"Authors must avoid giving direct recommendations, or referring to any intervention as ‘safe’
or ‘effective’. Specifically, the following statement must be removed;

“We think the evidence suggests that exercise therapy might be an effective and safe
intervention for patients able to attend clinics as outpatients.” "

If only this were implemented (not just on the Cochrane review).
 
If you have a look at the amended abstract they propose (last page), it's not revolutionary (nothing like "CBT and GET are ineffective", but the accumulation of "mays" seriously undermines the claims of success). That Larun et al. couldn't sign this is telling.
upload_2019-3-13_10-50-21.png
 
Last edited:
Cheshire said:
If you have a look at the amended abstract they propose (last page), it's not revolutionary (nothing like "CBT and GET are ineffective", but the accumulation of "mays" seriously undermines the claims of success). That Larum et al. couldn't sign this is telling.
View attachment 6350
Click to expand...

For comparison, this is what the Authors' conclusion of the abstract *still* says:
Authors' conclusions

Patients with CFS may generally benefit and feel less fatigued following exercise therapy, and no evidence suggests that exercise therapy may worsen outcomes. A positive effect with respect to sleep, physical function and self‐perceived general health has been observed, but no conclusions for the outcomes of pain, quality of life, anxiety, depression, drop‐out rate and health service resources were possible. The effectiveness of exercise therapy seems greater than that of pacing but similar to that of CBT. Randomised trials with low risk of bias are needed to investigate the type, duration and intensity of the most beneficial exercise intervention.
Click to expand...

Have they provided any reason at all why this hasn't been changed?
 
So in summary, it seems that Cochrane agreed with Robert Courtney on using the FINE trial unpublished data and the PACE trial selective reporting bias. But it's not clear if this will significantly alter the results. It seems mostly an illustration of Larun et al.'s bias in conducting this review.

The most important critique - leaving out all objectives outcomes - is not mentioned. Vinks analysis showed that most of the GET-trials had objective outcomes. Some found improvements, but most of them didn't. So the conclusion of that would probably be that GET fails to show improvement on objective outcomes.

As I see it the main critique addressed in this document, is the fact that at follow-up, GET no longer showed a significant improvement on all outcomes except for sleep. So normally you would expect the review to write something like: "GET showed significant improvements post-treatment but no longer at follow-up". But instead of simply saying that, the Cochrane commentator suggests it is best to state that no reliable conclusion can be made of the follow-up data. So he/she wants the review to state:
“When exercise therapy was compared with 'passive control,' fatigue may be reduced at end of treatment (Analysis 1.1), however we are uncertain if there is any difference between groups at follow up”
Click to expand...

So the outcomes at post-treatment are deemed reliable to mention, but the data at follow-up isn't. What the differences between the two?
  • If we take fatigue as the main outcome, 453 patients in the intervention group and 387 patients in the control group gave data post-intervention. Data came from 7 studies.
  • At follow-up there was data for 373 patients in the intervention group and 297 patients in the control group. Data came from 4 studies.

That doesn't seem like a large difference to me. It's confusing to only state the post-treatment outcomes and act like the follow-up doesn't provide reliable outcomes. There ain't much of a difference between the two. In both cases, the majority of patients came from three studies: Powell et al. 2001, the FINE trial (Wearden et al. 2010) and the PACE trial (White et al. 2011). The interesting thing is that those three trial reported the follow-up data as their main outcome! So I think it's misleading, as the Cochrane commentator suggests, to state the post-treatment data and consider the follow-up data too unreliable to mention. That's contrary to how these trials were conducted and reported. And for obvious reasons: the response bias is probably much larger shortly after treatment. That's why well-conducted trials measure their main outcome some weeks/months after the end of treatment.
 
Last edited:
It seems that the whole methodology of Cochrane fails to give an accurate view of what actually happened in these trials. Anyone looking at the data (not the propaganda of the authors) of the GET-trials neutrally would come to the following conclusion: "there seemed to be an improvement on subjective outcomes post-treatment but that disappears at follow-up and objective outcomes showed no improvement. The initial improvement was probably due to lack of blinding and response bias. So GET doesn't work." The Cochrane method of doing a review seems to avoid that conclusion by taking each part of the outcomes separately. The lack of blinding, for example, should be used to interpret the results. Instead, Cochrane simply addresses it by downgrading the certainty of their conclusions.

If they were to take each objective outcome (VO2 max, walking test, fitness test, employment, disability payments) separately, they would probably conclude that for each outcome there isn't enough reliable data to reach a conclusion. But that would miss the obvious conclusion that these trial do not show objective improvement if you look at all these objective outcomes together. I hope this will not be the case, if they decide to include objective outcomes.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom