Concerns about Cochrane

My understanding is that NICE does a review based on other meta reviews, so sort of a meta meta review, and bases its conclusions from that.

But I could be, and probably am, wrong, my impression is just based upon my understanding of what I've read, which is always a dubious method of forming a reliable conclusion upon which to make pronouncements.

 

NICE gathers all relevant publications but that is different from a meta-analysis. Inmost cases you look at the publications and if there is solid evidence from any of them that is enough. Most drug regulatory authorities require more than one study showing efficacy but NICE has on occasion agreed to recommend use of new drugs prior to licensing.

 

NICE used York for the last ME guideline but it is something that was discussed at the table I was at for the last stakeholder meeting.

This function has been outsourced (with many other things) to the National Clinical Guideline Centre based at the Royal College of Physicians.

They will be doing the evidence search using the Grade method

About NCGC (from the RCP website)
Who we are
The NCGC was formed in April 2009 following the merger of the National Collaborating Centres for Acute Care, Chronic Conditions, Nursing and Supportive Care and Primary Care. Hosted by the Royal College of Physicians, the NCGC is one of the largest clinical guideline development organisations in the world.
The work of the NCGC is overseen by a governance partnership between the Royal Colleges of General Practitioners, Nursing, Physicians and Surgeons. Each college is represented on the NCGC management board, alongside representatives from the Royal College of Physicians Patient and Carer Network, the UK Cochrane Centre, and NHS England.
What we do
The NCGC is commissioned to develop NICE clinical practice guidelines. NICE guidelines provide care standards within the NHS for healthcare professionals, patients and their carers on the prevention, treatment and care of people with specific diseases and conditions. The NCGC specialises in guidance for acute and chronic conditions and delivers a large work programme covering a wide variety of clinical and service delivery topics.
How we work
The NCGC is a vibrant, dedicated and enthusiastic team with a staff of 70 people. The majority are research specialists in systematic reviewing, health economics and information science, supported by operational and clinical directors, project managers and project co-ordinators.


and from their manual (attached)

Evidence Searches
Developing a strategy to search for evidence will include the following:
 developing search questions based on the clinical questions the guideline will address search terms using key words derived from the questions to be answered
 databases to be consulted such as the Cochrane Library, Medline, Embase, and US National Guideline Clearing House.
 the range of publication years included in the search (e.g. 1995 to present)
 filters to screen out references which are not relevant based on the keywords or headings by which the reference has been cateogorised. This may be, for example, exclude case studies, articles not published in the English language or animal studies. There may be specific filters for each question, for example if there is likely to sufficient RCT evidence, the search may exclude all observational studies.

 

If Cochrane reviews are just a rubber-stamping exercise, then it defeats the purpose. Cochrane was founded to rock the boat, though it seems many don't like it that way.

 

Paul Whiteley's take on the whole Cochrane saga:
https://questioning-answers.blogspo...f-facts-politics-evidence-based-medicine.html

Nothing very original, but a nice overview.

 

but from what I remember reading they only look at the abstracts which ,certainly in the case of CBT or GET 'research', almost always spin the results as positive.

 

I usually go by abstracts. But if an abstract 'spins' in the sense of coming to a conclusion that is not already evident from the data presented then for me it is heavily marked down, or very likely binned. Good scientific studies are not hard to assess from an abstract. The conclusion is not what one looks at, other than as a give away that the data might not be quite what they seem.

The real problem is people assessing studies who do not not know about the reality of clinical research. It is a bit like people assessing a situation faced by soldiers without any training in combat. Reality is not to be found in books. That is why I worry when I see terms like 'research specialists in systemic reviewing'.

 

it could be more than that when considering Cochrane. It looks as if NICE not only weights Cochrane Highly but actively recruits them for updates. This is from the Surveillance paper in 2017 on the ME/CFS Guideline.

"The latest Cochrane review of CBT for CFS in adults was published in 2008. NICE has approached Cochrane about the possibility of updating this review."

 

It does not really get to the meat of the matter for me.

The sentence: Such a public spat however, is unlikely to be good for science or evidence-based medicine, and indeed may have some wider implications for some fundamentals of science and science communication... seems to me wrong. It is likely to be very good for evidence-based science because it makes us all wake up the fact that these people are shamateurs.

We need Gotzsche but is he really a method purist? From what I have heard him say recently my impression is that he is as sloppy as Wessely is when it comes to it. He has his agenda, which involves being critical and that might be good in some ways. But his agenda seems to be to knock high tech medicine and replace it by comfy natural things like exercise and maybe even talking therapy.

If you want rigorous argument it seems to me you need to come to S4ME. None of these people are free of their own brand of bias.

 

Evidence based only has good value if it can withstand scrutiny. Otherwise its a sham.

 

Nobody is free of bias, which is why argument and debate are critical to science. However rhetorical argument, rather than factual or rational arguments, do not count, aside from a political debate.

 

by chance came across this paper from 2012
Quality and acceptability of patient-reported outcome measures used in chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME): a systematic review.

"
CONCLUSIONS:
The quality and acceptability of reviewed PROMs is limited, and recommendations for patient-reported assessment are difficult. Significant methodological and quality issues in PROM development/evaluation were identified by the appraisal framework, which must be addressed in future research. Clear discrepancies exist between what is measured in research and how patients define their experience of CFS/ME. Future PROM development/evaluation must seek to involve patients more collaboratively to measure outcomes of importance using relevant and credible methods of assessment."

just thought it might be of interest and didn't know where to put it!
https://www.ncbi.nlm.nih.gov/pubmed/21590511

 

And why do they keep saying "method purist" as it's a bad thing - shouldn't all researchers strive to be....?

Well, I do realize they're implying "method therrorist" pestering other people about their research practices. (Or, also known as scientific debate.)

But as a bystander, all I'm left with is the question - "Why aren't you?" - and the impression of a person doing sloppy research.

 

New blog from Hilda Bastian giving her view on things: https://blogs.plos.org/absolutely-m...out-the-evidence-but-its-critical-to-science/

I'm a bit unsure on some of this. I think that it's bad to try to use 'academic freedom' as an excuse to justify the promotion of misleading medical claims that will affect the lives of others, but I'm also wary of academics complaining of 'harassment' without giving details of exactly what behaviour was unacceptable. We've seen with PACE how a debate in the House of Lords was presented as an example of 'harassment' of researchers.

edit: she posts more info in the comments too.

 

Bastian is still talking of limiting damage but to my mind no damage has been done. On the contrary, the civil war at Cochrane has reminded us not to take it as an authority.

People might say that some sources have to be treated as more reliable than others, or we have no idea what to think. My take on that is that a source like Cochrane should make us think "Well, there is at least a chance that this is not just the usual drivel you find in the literature today, but let's read it carefully and see if the evidence presented is cogent." So there is no room for Fiona Watt implying that because a Cochrane review says something that is right. That is the biggest problem of all. The Cochrane wars have helped in showing that all of them, including Gotzsche, have axes to grind.

 

Precisely. Its a flag to possible usefulness, but does not remove the need for critical scrutiny.

 

Challenges of independent assessment of potential harms of HPV vaccines
BMJ 2018; 362 doi: https://doi.org/10.1136/bmj.k3694 (Published 24 September 2018)

1. Lars Jørgensen, researcher,1,
2. Peter Doshi, assistant professor2,
3. Peter Gøtzsche, professor1,
4. Tom Jefferson, researcher1
   
   "
   In our view, independent researchers ought to be able to obtain
   complete and unredacted clinical study reports within a
   reasonable time frame without too many constraints or
   limitations—especially when potential serious harms are
   reported after regulatory approval."
   
   "
   In her decision, the ombudsman concluded that EU rules on
   access to documents “are ill-suited to the purpose of making
   (large amounts of) scientific data available to researchers.”32
   The rules clearly need amending. Urgent changes are essential
   for open and transparent assessment of the harms and benefits
   of interventions."
   
   available on sci-hub:
   http://sci-hub.tw/10.1136/bmj.k3694
   
   
   

 

https://twitter.com/user/status/1044460974403719168

 

This statement says absolutely nothing. They're just throwing him under the bus without revealing any details of the complaints they got about him.

This Board decision is not about freedom of speech.
It is not about scientific debate.
It is not about tolerance of dissent.
It is not about someone being unable to criticize a Cochrane Review.

It is about a long-term pattern of behaviour that we say is totally, and utterly, at variance with the principles and governance of the Cochrane Collaboration. This is about integrity, accountability and leadership.

 

This bit:

Is really the essence here, isn't it? Both sides probably agree on that point - even we as bystanders.

To me, that statement reads as this has everything to do with scientific debate etc - but loyalty to Cochrane and the board must come first and foremost, is the remaining boards point of view. Goetzhe and friends have the oposite position.