Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

This is terrible. Please can you share the Tweet that was liked? Was it Signe Flottorp who liked it? The NICE process was demonstrably transparent and scientific - exactly the same process as they use for all guidelines.

I think Cochrane may be risking legal action against them by the authors (or the NIPH?) if they withdraw a review without permission of the authors. I can't think what else is stopping them.

 

If you click on the likes to this tweet, you'll see it. And yes, it was Flottorp.

 

@rvallee has just posted a retweet of this by Carson
https://www.s4me.info/threads/uk-ni...lication-discussion.22996/page-34#post-395460

 

It has to be the authors themselves as NIPH is governmental and is answerable to the minister of Health. They cant independently threathen to sue Cochrane. Perhaps Cochrane should move this up on political levels in Norway.

 

Folks, got this reply from NICE i.e. to my email setting out my concerns re NICEs use of GRADE/Cochrane. I haven't even read it yet

@Jonathan Edwards @Caroline Struthers


"Dear Francis.

Thank you for your follow up email.

The Cochrane review Exercise therapy for chronic fatigue syndrome<https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full> (Larun et al. 2019), and the evidence reviews on exercise therapy in support of NICE guideline NG206 Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management<https://www.nice.org.uk/guidance/ng206>, are separate entities. The Cochrane review by Larun et al. was examined by the NG206 guideline committee<https://www.nice.org.uk/guidance/ng...acological-management-of-mecfs-pdf-9265183028> (see p.7, section 1.1.3.2 ‘Excluded studies’) who noted that it “…pooled all exercise therapies irrespective of the type of exercise therapy, and also pooled all control arms considered ‘passive’ (including treatment as usual, relaxation and flexibility). We did not consider this methodology appropriate for decision-making.”

Each review question in the NICE guideline has its own review protocol agreed by the guideline committee. These protocols are different to those used by Cochrane reviews (for example, different inclusion and exclusion criteria), and the NICE guideline conducted its own independent systematic reviews based on Developing NICE guidelines: the manual<https://www.nice.org.uk/process/pmg20/chapter/introduction>.

In terms of quality assessment of the evidence, the NICE guideline committee exercised their clinical judgement to decide which components of GRADE were important for downgrading evidence quality, for example, risk of bias, population directness, and threshold(s) for minimal clinically important differences. The full details of the review protocol and data analysis relating to exercise therapy in NG206 can be found in Evidence review [G] for the non-pharmacological management of ME/CFS<https://www.nice.org.uk/guidance/ng...acological-management-of-mecfs-pdf-9265183028>. The clinical evidence summary tables for exercise interventions, including GRADE analysis, begin on p.147.

These differences between the methods and processes of Cochrane reviews and NICE guidelines can result in different quality ratings of the evidence in GRADE, as in this case with exercise therapy for ME/CFS.

Please also note that in Oct 2019 Cochrane announced<https://www.cochrane.org/news/cfs> that “a new approach to the publication of evidence in this area is needed; and, today we are committing to the production of a full update of this Cochrane Review”.

Regarding the announcement in Sept 2021 that NICE have signed a collaborative agreement with Cochrane. This is a high level agreement, and NICE is in the process of working with Cochrane in an attempt to align processes and methods in the future, and this will take time.

I hope the above information is useful for you.

Kind regards
Katy

 

Not to me I am afraid. None of it is new.

 

Well done on getting this response. I will soon write to NICE and ask how they are going to align the GRADE rating process with Cochrane so that the situation where Cochrane were unable exert any authority over the Exercise review authors to downgrade the evidence won't happen again.

 

Trial By Error by David Tuller Lightning Process Star Complains About NICE; Struthers Nudges Cochrane to Keep Up

quote:
It is important to hold organizations to account when they fall short of expectations, as Cochrane has in how it has handled these two published reviews. Until recently, it had allowed the GET/CBT ideological brigades full authority over how the illness was handled. That has changed. In her latest letter, Struthers is appropriately asking Cochrane to clean up after itself and keep up with developments, among other requests. You can read it here.

ETA: Article is also shared at the post NICE guideline publication thread here

 

Thanks @Caroline Struthers i.e. for your assistance and to @Michiel Tack too.

Caroline, I imagine NICE's reaction when you ask how they [NICE] are going to align the GRADE rating process with Cochrane so that the situation where Cochrane were unable exert any authority over the Exercise review authors to downgrade the evidence won't happen again.

I think the last quote captures the reality i.e. NICE are engaging with Cochrane in an attempt to align processes and methods in the future, and this will take time. While NICE may well be engaged in such an attempt I don't hold out much hope that Cochrane is. E.g. where a Cochrane review is relevant to a NICE guideline, then will Cochrane consult on the guideline, and publish the responses, along with any revisions and the reasons? Caroline's point comes to mind - Cochrane won't do revisions ---- so Cochrane basically doesn't do what it says on the can --- review the evidence.
NICE's attempt at revising Cochrane may well take time; however, it really doesn't need to - Cochrane isn't set up to deliver a review the evidence.

Perhaps there are hints in NICE's guarded statement about high level agreement --- an attempt to align processes and methods in the future, and this will take time.
Translation - NICE has gone for the photo opportunity [agreement with Cochrane] without thinking through what they were getting involved in.

I think we need to keep an eye on NICE's use of GRADE/Cochrane and to try to get others [Long covid ++] to challenge it.

I'll try to draft a response to NICE [Katy]

Thanks to everyone for their assistance.

 

what could possibly go wrong? --- thanks for the laugh!

Yea they may be randomized and non-randomized studies; however, as I recall @Caroline Struthers pointing out --- many of the Cochrane systematic reviews ignore the quality of the base studies --- unreliable derivates of unreliable studies ---- evidence syntheses!

 

I thought GRADE was led by Guyatt of McMaster?

 

Yes it is

 

Repeating myself ---

I recall watching a film, with some other folks, about a fictional TV presenter who basically took his guests apart - one of the other folks pointed out that a show like that would only last a few episodes - no one would be willing to be a guest.

If you proposed to do a systemic study on an intervention and all of the available research was:

• unblinded; and
• used subjective outcomes (questionnaires);

then you either publish a flawed study i.e. GRADE/Cochrane approach, or you publish a study showing that all of the available research is unreliable. If you go down the latter route then your opportunities to collaborate with other researchers may be more limited!

So even if Cochrane could downgrade a study on an intervention, there is no incentive for it to do so - if it did then the pool of researchers willing to do a Cochrane review would shrink considerably. Possibly you could get a few retired researchers to do a review if you paid them, and they didn't mind loosing some friendships among their former colleagues; however, that would require Cochrane to have money to do reviews --- does it?

So Cochrane looks like something that is compelled to go on talking the talk about doing a review/revision e.g. to avoid unblinded studies with subjective outcome indicators being labelled as moderate quality. However, it can't actually do it ---- it has to stick with just talking the talk.

Sadly(?) I think researchers should join the bandwagon i.e. submit their old grant applications but include objective outcome indicators (FitBit type devices). The funding bodies, particularly NIH, might just fund them. Of course they'd probably show that the intervention doesn't work so the gravy train of repeated (pointless) unblinded studies, with subjective outcome criteria, would finally come to an end ---- where would the money come from then?

So Cochrane looks like a dud ---- one to be watched and challenged

EDIT -
@It's M.E. Linda posted this:
Did you ever get a reply from the APPG and/Or Lady Cumberlege @Caroline Struthers please?
APPG looks like a good route to challenge GRADE/Cochrane.

Further EDIT - of course, if Cochrane want to know the unadulterated truth about their reviews, then they could send them to @Jonathan Edwards before they publish them ---- can't imagine Jonathan will be getting that call though!

 

No. well it does get money but seems to spend it on editors, systematic review methods development, and tech - they don't pay the hands on reviewers anything directly. So they can't threaten not to pay them if they do a bad job. They don't even recognize (or seem to much care) when they have done a bad job. The NICE approach of having technical reviewers who are paid and are carefully vetted to eliminate bias seems very good - but it's expensive. I think they are trying to save money by teaming up with Cochrane who don't pay their reviewers anything. I will do my best to get across what a terrible idea this is.

Exactly this.

Yep. Plan to carry on watching and challenging

 

Certainly seems so, in fact we can rely on Paul Garner et al to come out and tell us what idiots we are [including NICE] for thinking it's a bad job---!

Conversation recently on BBC Radio Ulster - more than half of the NI Block Grant (£6.5 billion ish of the total £13 billion ish) is spent on health - there are 2 million people in NI. So, spending a few pounds on paying NICE technical staff, so that guidelines are reliable, looks like next to nothing i.e. compared to the cost of implementing those guidelines. Also, I don't think people know that typically you can get good scientists EDIT relatively cheaply ---- look at the salaries for teachers ---- that indicates the ball park --- so technical staff aren't going to be that expensive.

Sometimes you can get something for nothing; however, this doesn't look like one of them. Yes, it certainly looks like a terrible idea; @Barry has highlighted* that we have some unlikely allies in demonstrating that - Paul Garner launching a public attack on NICE i.e. based on the usual unevidenced statements --- I shouldn't compliment him --- he might stop/slack off.

*https://www.s4me.info/threads/uk-ni...lication-discussion.22996/page-37#post-395873

Thank you very much --- please do.

 

Thanks @Hutan. This brought back some deeply unpleasant information I’d read and forgotten.


“With medication, treatment and a rehabilitation programme, I am now looking at a full-time return to work in the summer.”

I expect we can imagine the rehabilitation program. But what medication and treatment could this be that raised one up and back into full time work?

I think access to effective medical care would make this patient an outlier. Probably not a person in a position to have the most realistic assessment of the overall experience for patients.