Concerns about Cochrane

This is so much what it is all about, I agree, @Esther12. One has to keep asking oneself how rational one is really being. After all it might be crazy to think that Simon Wessely had no idea what he was talking about. And it might be equally crazy to think 100 academics writing to the Lancet had no idea what they were talking about ... and so on.

But sometimes all I feel I need to do is stop and think how the people around me at work have behaved for the last thirty years. Do they fiddle the data? Yes, all the time. Do they know the rules? Yes. But do they apply the rules? No. Can you tell if they are a died in the wool bullshitter if you listen to them presenting their work for five minutes? By and large yes, although one or two are wilier than that.

In a sense I agree that one should stick to the evidence, but so often even raw data comes so overlain with the baleful effects of human nature that one has to put on different tinted glasses, at least to see the wood for the trees.

 

Do you think that NICE are aware of these problems, as I presume they will rely heavily on Cochrane reviews.? Surely part of the problem is vested self interest because doctors need publications for career advancement. Wessely has authorship of 800 papers but one good paper in a whole career is better than 800 dubious ones. I also think peer review amounts to little more than cronyism and nepotism. Until the rush to publish changes these problems will persist.

 

A bit OT but as the theme is Cochrane and this is a patient forum, I thought someone here might be interested to have a listen:
https://twitter.com/user/status/1040871935767445504

 

https://twitter.com/user/status/1040862794395082753

 

The BMJ: Cochrane HPV vaccine review: BMJ journal defends "inconvenient criticisms"

Defending the paper this week in an editorial in BMJ Evidence-Based Medicine,5 Heneghan and Onakpoya said, “We acknowledge that articles in our journal will seek to hold organisations to account and should not shrink from offering criticisms that may be considered inconvenient.

“Academic freedom means communicating ideas, facts and criticism without being censored, targeted or reprimanded. We believe that the article . . . provokes healthy debate and poses important questions about the need to ensure that all available evidence is included in systematic reviews to properly inform healthcare decisions.”

The disagreement over the review touches a sensitive nerve because two of the critics, Lars Jørgensen and Peter Gøtzsche, are from the Cochrane Nordic Centre. It thus has something of the bitter quality of a civil war, with both sides claiming to better represent the ideals of the Cochrane Collaboration for exhaustive analysis of all appropriate data.

 

More fun drama:

https://twitter.com/user/status/1041055036820869122


https://twitter.com/user/status/1037333923871768588

 

This drama may be related to Gotzsche criticising the Cochane review of the HPV vaccine https://ebm.bmj.com/content/early/2018/07/27/bmjebm-2018-111012

I found it a little shocking to be honest. If this critique is valid then the Cochrane review was biased junk that probably obscured harms and may have exaggerted benefits.

Relevant to ME/CFS: the HPV vaccine may increase the risk POTS and complex regional pain syndrome.

 

This looks to be the typical mess where one has to be careful trying to sort good guys from bad guys. I am not sure anyone comes out well. My impression is that Gotzsche may well have a bee in his bonnet about various things that are unhelpful, but on the other hand his critique of the executive is highly plausible.

The Cochrane executive leadership almost always uses the commercial terms of “brand”, “products” and “business” but almost never describes what is really a collaborative network with the values of sharing, independence and openness.

As one might have predicted. Cochrane is now a Murdoch style 'sell them what they want' enterprise. For me Cochrane as an authority is now dead. So in relation to:

Message from the Governing Board of @cochranecollab. Investigation ongoing; we should all reserve judgement and give them some space.

I am afraid I have seen enough to know that I do not need to reserve judgement. Without a new CEO and board this is a dead duck. There is no reason to think the 'ongoing investigation' will address the problem. The message really seems to say 'please look away while we are stabbing each other in the back, we don't want anyone to rock the share price'.

 

So the way we can deal with Cochrane is to stop waiting for them to get their shit together and instead just point out that it's no longer an unbiased and trustworthy source of information.

 

Yes, it is a pity that when I wrote back to Fiona Watt saying that there was some concern within Cochrane that all was not well I was not in a position to say that the whole thing is in a state of civil war about exactly this sort of drop in standards.

 

Sir Ian Chalmers is currently on the front page of the Norwegian broadcasting company in an article about being critical to research claims. I wonder if the timing is entirely coincidental?

NRK: Slik unngår du å bli lurt av dårlig forskning
google translation: How to avoid being fooled by bad research

Being critical of research is not hard to learn, says one of the founders of The Cochrane Collaboration, Iain Chalmers.
The Cochrane Collaboration is a recognized research collaboration that compiles good quality research results to determine the extent to which treatments work or not. The results are presented both in a medical language and in an easy-to-understand language.

 

Pity that Chalmers was dismissive when I suggested it was odd that the recent exercise for ME review draft was full of flaws despite being written by the guy who writes these articles with Chalmers about how easy it is to spot bad science. He was particularly affronted that I raised conflict of interest. These people don't actually understand the realities behind the recipes they recommend.

 

My experience from whenever ME finds its way onto subreddits with physicians present is that Cochrane is held in higher status that the NIH, CDC and NAM combined. I don't know if it's just an exception they do for us by cherry-picking a source that agrees with their opinions or a general sentiment, but Cochrane seems to be held to a near infaillible credibility so that may a tall order.

However it's clear that Cochrane practices a garbage in, garbage out approach where it's better to have something that relies on placebo effect and confirmation bias than to have nothing at all. That's just bound to naturally fail over time and maybe that's what's happening now, especially with the convergence of the crisis of replicability and the creeping rot of for-profit scientific publishing.

One would think that given the Lancet-Wakefield debacle, the controversy over PACE would have been taken more seriously if it were not for the garbage in, garbage out approach. It seems that medical science in the UK is undergoing a crisis of sort, driven largely by domestic politics of austerity.

 

I am not sure that there is so much a crisis as a silent slipping into a swamp of mediocrity. The crisis may be coming now - in the sense of confronting just how low things have sunk.

Cochrane had a unique reputation as honest broker, but it is pretty clear that this is now defunct. I had been writing some stuff about the role of Cochrane in ME and was thinking it might be novel and exciting to raise question marks. It seems the question marks have now landed like a ton of bricks.

Austerity and commerciality may have something to do with it but a much bigger issue that I see is intellectual laziness on the part of medics and scientists.

 

It's possible to leave comments on this article for those who want to draw parallels with the Cochrane review of CBT/GET.

Tagging @dave30th and @Jonathan Edwards and @Graham and @Tom Kindlon (and am sure I'm leaving loads of useful people out!).