Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic re..., 2022, Schmidt et al

Full title: Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis

https://www.sciencedirect.com/science/article/abs/pii/S0895435622001007


Highlights

• In this large sample of 1567 interventions studied within Cochrane reviews, effects of most interventions (94%) interventions were not supported by high quality evidence.

• Potential harms of healthcare interventions were measured more rarely than benefits.

• Patients, doctors, and policy makers should consider the lack of high-quality evidence supporting the benefits and harms of many interventions in their decision-making.​

Results

Of 1567 eligible interventions, 87 (5.6%) had high quality evidence on first-listed primary outcomes, positive, statistically significant results and were rated by review authors as beneficial. Harms were measured for 577 (36.8%) interventions, 127 of which (8.1%) had statistically significant evidence of harm. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was a potential limitation of our study.​

Conclusion

Most healthcare interventions studied within recent Cochrane Reviews are not supported by high quality evidence, and harms are under-reported.
​

 

Going for the jugular there?

 

Probably needs a mountain of salt, as this review includes John Ioannidis, who has been critical of Cochrane for all the wrong reasons.

Although none of this is a surprise to us, the few Cochrane reviews published on subject matters that are known to this forum all fail to meet minimal standards of basic competence and scientific literacy, and have reached false, even misleading to the point of harmful, conclusions. They are all also heavily biased, to the point of being suspiciously like a marketing pitch.

This is a systematic review of the process of making systematic reviews. Can a failed process evaluate itself? Can we trust the conclusions of a process that so often reaches invalid conclusions? Can a systematic review that finds most systematic reviews to be flawed be trusted? If so, can systematic reviews continue to be trusted? If not, the same, since this is a systematic review.

 

I have been through it. It is pretty much what it says on the tin - very few treatments come out with high quality evidence on Cochrane reviews and harms are under-reported and common.

But the pinch of salt is that they think GRADE is great and maybe even a bit strict.

On the other hand that makes things even easier - things are probably worse than they say.

Having said that, my impression is that therein problem with GRADE and RoB is that they are too kind when it comes to rating very low. I think they may pick out enough to reduce from high to moderate. So at least for this exercise they may be adequate.

 

Haven't read yet, but....

.... this worries me too.

All the more as all the co-authors seem to be some of the most active proponents of 'alternative' medicine and psychosomatic pseudoscience,

e.g. Harald Walach:

https://en.wikipedia.org/wiki/Harald_Walach

Such a pity when justified critique of some of the most harmful research practices happens to be uttered only by some of the most questionable researchers...

 

David Gorski, MD, PhD on Twitter: " Cochrane reviews are only as good as the evidence that goes into them (and sometimes not even then). Their fatal flaw is that they do not take into account prior plausibility based on basic science, which can lead to false positives in the case of highly improbable treatments."

Aren't these two different things?

Testing highly implausible treatments I think is wasting resources.

Sound assessment of those treatments should be able to conclude that they aren't effective though? Aren't false positives more likely to result from bad assessment, e.g. turning a blind eye to the combination of solely subjective outcomes in unblinded trials?