Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Caroline Struthers]

Only yesterday the head of research in the Norwegian institute of public health liked and shared an article by a norwegian LP coach about how unscientific the NICE process was.

This is terrible. Please can you share the Tweet that was liked? Was it Signe Flottorp who liked it? The NICE process was demonstrably transparent and scientific - exactly the same process as they use for all guidelines.

I think Cochrane may be risking legal action against them by the authors (or the NIPH?) if they withdraw a review without permission of the authors. I can't think what else is stopping them.

 

[Kalliope]

This is terrible. Please can you share the Tweet that was liked? Was it Signe Flottorp who liked it? The NICE process was demonstrably transparent and scientific - exactly the same process as they use for all guidelines.

I think Cochrane may be risking legal action against them by the authors (or the NIPH?) if they withdraw a review without permission of the authors. I can't think what else is stopping them.

If you click on the likes to this tweet, you'll see it. And yes, it was Flottorp.

 

[Sly Saint]

If you click on the likes to this tweet, you'll see it. And yes, it was Flottorp.

@rvallee has just posted a retweet of this by Carson
https://www.s4me.info/threads/uk-ni...lication-discussion.22996/page-34#post-395460

 

[Joniper]

This is terrible. Please can you share the Tweet that was liked? Was it Signe Flottorp who liked it? The NICE process was demonstrably transparent and scientific - exactly the same process as they use for all guidelines.

I think Cochrane may be risking legal action against them by the authors (or the NIPH?) if they withdraw a review without permission of the authors. I can't think what else is stopping them.

It has to be the authors themselves as NIPH is governmental and is answerable to the minister of Health. They cant independently threathen to sue Cochrane. Perhaps Cochrane should move this up on political levels in Norway.

 

[FMMM1]

Folks, got this reply from NICE i.e. to my email setting out my concerns re NICEs use of GRADE/Cochrane. I haven't even read it yet

@Jonathan Edwards @Caroline Struthers


"Dear Francis.

Thank you for your follow up email.

The Cochrane review Exercise therapy for chronic fatigue syndrome<https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full> (Larun et al. 2019), and the evidence reviews on exercise therapy in support of NICE guideline NG206 Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management<https://www.nice.org.uk/guidance/ng206>, are separate entities. The Cochrane review by Larun et al. was examined by the NG206 guideline committee<https://www.nice.org.uk/guidance/ng...acological-management-of-mecfs-pdf-9265183028> (see p.7, section 1.1.3.2 ‘Excluded studies’) who noted that it “…pooled all exercise therapies irrespective of the type of exercise therapy, and also pooled all control arms considered ‘passive’ (including treatment as usual, relaxation and flexibility). We did not consider this methodology appropriate for decision-making.”

Each review question in the NICE guideline has its own review protocol agreed by the guideline committee. These protocols are different to those used by Cochrane reviews (for example, different inclusion and exclusion criteria), and the NICE guideline conducted its own independent systematic reviews based on Developing NICE guidelines: the manual<https://www.nice.org.uk/process/pmg20/chapter/introduction>.

In terms of quality assessment of the evidence, the NICE guideline committee exercised their clinical judgement to decide which components of GRADE were important for downgrading evidence quality, for example, risk of bias, population directness, and threshold(s) for minimal clinically important differences. The full details of the review protocol and data analysis relating to exercise therapy in NG206 can be found in Evidence review [G] for the non-pharmacological management of ME/CFS<https://www.nice.org.uk/guidance/ng...acological-management-of-mecfs-pdf-9265183028>. The clinical evidence summary tables for exercise interventions, including GRADE analysis, begin on p.147.

These differences between the methods and processes of Cochrane reviews and NICE guidelines can result in different quality ratings of the evidence in GRADE, as in this case with exercise therapy for ME/CFS.

Please also note that in Oct 2019 Cochrane announced<https://www.cochrane.org/news/cfs> that “a new approach to the publication of evidence in this area is needed; and, today we are committing to the production of a full update of this Cochrane Review”.

Regarding the announcement in Sept 2021 that NICE have signed a collaborative agreement with Cochrane. This is a high level agreement, and NICE is in the process of working with Cochrane in an attempt to align processes and methods in the future, and this will take time.

I hope the above information is useful for you.

Kind regards
Katy

 

[Jonathan Edwards]

I hope the above information is useful for you.

Not to me I am afraid. None of it is new.

 

[Caroline Struthers]

Regarding the announcement in Sept 2021 that NICE have signed a collaborative agreement with Cochrane. This is a high level agreement, and NICE is in the process of working with Cochrane in an attempt to align processes and methods in the future, and this will take time.

Well done on getting this response. I will soon write to NICE and ask how they are going to align the GRADE rating process with Cochrane so that the situation where Cochrane were unable exert any authority over the Exercise review authors to downgrade the evidence won't happen again.

 

[Kalliope]

Trial By Error by David Tuller Lightning Process Star Complains About NICE; Struthers Nudges Cochrane to Keep Up

quote:
It is important to hold organizations to account when they fall short of expectations, as Cochrane has in how it has handled these two published reviews. Until recently, it had allowed the GET/CBT ideological brigades full authority over how the illness was handled. That has changed. In her latest letter, Struthers is appropriately asking Cochrane to clean up after itself and keep up with developments, among other requests. You can read it here.

ETA: Article is also shared at the post NICE guideline publication thread here

 

[FMMM1]

Well done on getting this response. I will soon write to NICE and ask how they are going to align the GRADE rating process with Cochrane so that the situation where Cochrane were unable exert any authority over the Exercise review authors to downgrade the evidence won't happen again.

Thanks @Caroline Struthers i.e. for your assistance and to @Michiel Tack too.

Caroline, I imagine NICE's reaction when you ask how they [NICE] are going to align the GRADE rating process with Cochrane so that the situation where Cochrane were unable exert any authority over the Exercise review authors to downgrade the evidence won't happen again.

These differences between the methods and processes of Cochrane reviews and NICE guidelines can result in different quality ratings of the evidence in GRADE, as in this case with exercise therapy for ME/CFS.

Please also note that in Oct 2019 Cochrane announced<https://www.cochrane.org/news/cfs> that “a new approach to the publication of evidence in this area is needed; and, today we are committing to the production of a full update of this Cochrane Review”.

Regarding the announcement in Sept 2021 that NICE have signed a collaborative agreement with Cochrane. This is a high level agreement, and NICE is in the process of working with Cochrane in an attempt to align processes and methods in the future, and this will take time.

I think the last quote captures the reality i.e. NICE are engaging with Cochrane in an attempt to align processes and methods in the future, and this will take time. While NICE may well be engaged in such an attempt I don't hold out much hope that Cochrane is. E.g. where a Cochrane review is relevant to a NICE guideline, then will Cochrane consult on the guideline, and publish the responses, along with any revisions and the reasons? Caroline's point comes to mind - Cochrane won't do revisions ---- so Cochrane basically doesn't do what it says on the can --- review the evidence.
NICE's attempt at revising Cochrane may well take time; however, it really doesn't need to - Cochrane isn't set up to deliver a review the evidence.

Perhaps there are hints in NICE's guarded statement about high level agreement --- an attempt to align processes and methods in the future, and this will take time.
Translation - NICE has gone for the photo opportunity [agreement with Cochrane] without thinking through what they were getting involved in.

I think we need to keep an eye on NICE's use of GRADE/Cochrane and to try to get others [Long covid ++] to challenge it.

I'll try to draft a response to NICE [Katy]

Thanks to everyone for their assistance.

 

[Hutan]

I think we need to keep an eye on NICE's use of GRADE/Cochrane and to try to get others [Long covid ++] to challenge it.

GRADE Guidance 24. Optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelinesCuello-Garcia, Carlos A, Santesso, Nancy, Morgan, Rebecca L, Verbeek, Jos, Thayer, Kris, Ansari, Mohammed T, Meerpohl, Joerg, Schwingschackl, Lukas, Katikireddi, Srinivasa Vittal, Brozek, Jan L, Reeves, Barnaby, Murad, Mohammad H, Falavigna, Maicon, Mustafa, Reem, Regidor, Deborah L, Alexander, Paul Elias, Garner, Paul, Akl, Elie A, Guyatt, Gordon and Schünemann, Holger J (2021) 'GRADE Guidance 24. Optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines'. Journal of Clinical Epidemiology. (In Press)

When you've got Paul Garner co-writing GRADE Guidance, what could possibly go wrong?

 

[FMMM1]

GRADE Guidance 24. Optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelinesCuello-Garcia, Carlos A, Santesso, Nancy, Morgan, Rebecca L, Verbeek, Jos, Thayer, Kris, Ansari, Mohammed T, Meerpohl, Joerg, Schwingschackl, Lukas, Katikireddi, Srinivasa Vittal, Brozek, Jan L, Reeves, Barnaby, Murad, Mohammad H, Falavigna, Maicon, Mustafa, Reem, Regidor, Deborah L, Alexander, Paul Elias, Garner, Paul, Akl, Elie A, Guyatt, Gordon and Schünemann, Holger J (2021) 'GRADE Guidance 24. Optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines'. Journal of Clinical Epidemiology. (In Press)

When you've got Paul Garner co-writing GRADE Guidance, what could possibly go wrong?

what could possibly go wrong? --- thanks for the laugh!

Yea they may be randomized and non-randomized studies; however, as I recall @Caroline Struthers pointing out --- many of the Cochrane systematic reviews ignore the quality of the base studies --- unreliable derivates of unreliable studies ---- evidence syntheses!

 

[Hutan]

Financial support: The Cochrane Methods Innovation grant, the National Toxicology Program in the U.S., and the McMaster GRADE centre provided support for this project.

(From that GRADE Guidance - note the 'McMaster GRADE centre' providing financial support. Does that mean GRADE has a relationship with McMaster University? I don't think I knew that. Also does not bode well.)

 

[Jonathan Edwards]

I thought GRADE was led by Guyatt of McMaster?

 

[Caroline Struthers]

I thought GRADE was led by Guyatt of McMaster?

Yes it is

 

[Sean]

re: evidence synthesis

A polished turd is still a turd.

 

[FMMM1]

This is terrible. Please can you share the Tweet that was liked? Was it Signe Flottorp who liked it? The NICE process was demonstrably transparent and scientific - exactly the same process as they use for all guidelines.

I think Cochrane may be risking legal action against them by the authors (or the NIPH?) if they withdraw a review without permission of the authors. I can't think what else is stopping them.

Repeating myself ---

I recall watching a film, with some other folks, about a fictional TV presenter who basically took his guests apart - one of the other folks pointed out that a show like that would only last a few episodes - no one would be willing to be a guest.

If you proposed to do a systemic study on an intervention and all of the available research was:

• unblinded; and
• used subjective outcomes (questionnaires);

then you either publish a flawed study i.e. GRADE/Cochrane approach, or you publish a study showing that all of the available research is unreliable. If you go down the latter route then your opportunities to collaborate with other researchers may be more limited!

So even if Cochrane could downgrade a study on an intervention, there is no incentive for it to do so - if it did then the pool of researchers willing to do a Cochrane review would shrink considerably. Possibly you could get a few retired researchers to do a review if you paid them, and they didn't mind loosing some friendships among their former colleagues; however, that would require Cochrane to have money to do reviews --- does it?

So Cochrane looks like something that is compelled to go on talking the talk about doing a review/revision e.g. to avoid unblinded studies with subjective outcome indicators being labelled as moderate quality. However, it can't actually do it ---- it has to stick with just talking the talk.

Sadly(?) I think researchers should join the bandwagon i.e. submit their old grant applications but include objective outcome indicators (FitBit type devices). The funding bodies, particularly NIH, might just fund them. Of course they'd probably show that the intervention doesn't work so the gravy train of repeated (pointless) unblinded studies, with subjective outcome criteria, would finally come to an end ---- where would the money come from then?

So Cochrane looks like a dud ---- one to be watched and challenged

EDIT -
@It's M.E. Linda posted this:
Did you ever get a reply from the APPG and/Or Lady Cumberlege @Caroline Struthers please?
APPG looks like a good route to challenge GRADE/Cochrane.

Further EDIT - of course, if Cochrane want to know the unadulterated truth about their reviews, then they could send them to @Jonathan Edwards before they publish them ---- can't imagine Jonathan will be getting that call though!

 

[Caroline Struthers]

however, that would require Cochrane to have money to do reviews --- does it?

No. well it does get money but seems to spend it on editors, systematic review methods development, and tech - they don't pay the hands on reviewers anything directly. So they can't threaten not to pay them if they do a bad job. They don't even recognize (or seem to much care) when they have done a bad job. The NICE approach of having technical reviewers who are paid and are carefully vetted to eliminate bias seems very good - but it's expensive. I think they are trying to save money by teaming up with Cochrane who don't pay their reviewers anything. I will do my best to get across what a terrible idea this is.

So Cochrane looks like something that is compelled to go on talking the talk about doing a review/revision e.g. to avoid unblinded studies with subjective outcome indicators being labelled as moderate quality. However, it can't actually do it ---- it has to stick with just talking the talk.

Exactly this.

So Cochrane looks like a dud ---- one to be watched and challenged

Yep. Plan to carry on watching and challenging

 

[FMMM1]

They don't even recognize (or seem to much care) when they have done a bad job.

Certainly seems so, in fact we can rely on Paul Garner et al to come out and tell us what idiots we are [including NICE] for thinking it's a bad job---!

The NICE approach of having technical reviewers who are paid and are carefully vetted to eliminate bias seems very good - but it's expensive.

Conversation recently on BBC Radio Ulster - more than half of the NI Block Grant (£6.5 billion ish of the total £13 billion ish) is spent on health - there are 2 million people in NI. So, spending a few pounds on paying NICE technical staff, so that guidelines are reliable, looks like next to nothing i.e. compared to the cost of implementing those guidelines. Also, I don't think people know that typically you can get good scientists EDIT relatively cheaply ---- look at the salaries for teachers ---- that indicates the ball park --- so technical staff aren't going to be that expensive.

I think they are trying to save money by teaming up with Cochrane who don't pay their reviewers anything. I will do my best to get across what a terrible idea this is.

Sometimes you can get something for nothing; however, this doesn't look like one of them. Yes, it certainly looks like a terrible idea; @Barry has highlighted* that we have some unlikely allies in demonstrating that - Paul Garner launching a public attack on NICE i.e. based on the usual unevidenced statements --- I shouldn't compliment him --- he might stop/slack off.

*https://www.s4me.info/threads/uk-ni...lication-discussion.22996/page-37#post-395873

Yep. Plan to carry on watching and challenging

Thank you very much --- please do.

 

[Hutan]

(Sorry, I first posted this on the FOI thread, but realised it would be off-topic there.)

When we see the brazen attempts to influence the NICE Guideline, it makes me even more concerned about the Cochrane Exercise therapy review development process. Given the prevailing views about ME/CFS in Cochrane, I do not envy @Hilda Bastian's task in keeping the process uncorrupted and the new review solidly evidence based. I have no doubt that the BPS elite are working hard to ensure that the Cochrane review does not slip through the net as the NICE Guideline did.

Just as a reminder, @Hilda Bastian told us that she found Kay Hallsworth, the only ME/CFS patient in the review writing team, through an internet search. Kay's primary qualifications to be writing a review of exercise therapy appeared to be that she has ME/CFS and has not been 'an activist'.

Yes she knows of Wessely and likes his work as she was one of four people (including Wessely himself) who retweeted this Tweet
(Tweet about the Advance Study - armed services trauma rehabilitation, with a picture of the group involved including Wessely)

No one in Cochrane seems to know anything about ME. This is why the first review was done by the Mental Disorders Review Group. Wessely was the founding Editor of that group, although that fact seems to have been airbrushed out of history.

I think it's fine to ask for ME/CFS-related information about patient representatives on a review that will affect the management of patients around the world. If the representatives are not widely known and it's not yet obvious why they are qualified to do the job, I think it's reasonable to ask for that information. A lack of information does breed suspicion, especially when we consider Wessely's relationship with defence forces, and the defence force funding of a number of poor studies that we have reviewed.

https://civilservice.blog.gov.uk/2019/05/03/personal-disability-stories-10-kays-story/

Royal Navy Commander Kay Hallsworth, who has kindly been in contact to share her story of working in the Armed Forces whilst living with Myalgic Encephalomyelitis (ME)...
I have always had exceptional support through Defence Medical Services...
Thankfully, my fantastic GP at HMS Excellent, and the team at DMRC (Defence Medical Rehabilitation Centre) Stanford Hall, identified the underlying cause of my relapse. With medication, treatment and a rehabilitation programme, I am now looking at a full-time return to work in the summer.

https://www.gresham.ac.uk/professors-and-speakers/colonel-john-etherington-obe/

Colonel John Etherington is the Director of Defence Rehabilitation and a Consultant in Rheumatology and Rehabilitation at the Defence Medical Rehabilitation Centre, Headley Court in Epsom, Surrey. He has recently been appointed the National Clinical Director for Rehabilitation and Recovering in the Community for NHS England.

Colonel Etherington has been an officer commanding the Medical and Rehabilitation Divisions and Research Department at Headley Court. He is the Defence Consultant Advisor in Rheumatology and Rehabilitation and Sport and Exercise Medicine, and the Chairman of the Surgeon General's Training and Exercise Medicine Advisory Group. Colonel Etherington has been the chair of the Royal College of Physicians of London Committee on Sport and Exercise Medicine, and he is the Vice President of the Faculty Sport and Exercise Medicine, UK. He was the chair of the Department of Health’s Rehabilitation sub-group for Trauma Networks and he also chaired the RCP’s working party on ‘Exercise for Life’.

Adverse outcomes in trials of graded exercise therapy for adult patients with chronic fatigue syndrome
P.D.White a
J.Etherington b
a
Wolfson Institute of Preventive Medicine, St Bartholomew's and The London School of Medicine and Dentistry, Queen Mary University of London, London, UK
b
Pure Sports Medicine, 12 Finsbury Square, London, UK

(Lest you be thinking J. Etherington of Pure Sports Medicine, surely that's not the same person as Colonel Etherington CBE Director of Defence Rehabilitation?...
https://puresportsmed.com/team/dr-john-etherington-cbe)

 

[Ash]

Thanks @Hutan. This brought back some deeply unpleasant information I’d read and forgotten.


“With medication, treatment and a rehabilitation programme, I am now looking at a full-time return to work in the summer.”

I expect we can imagine the rehabilitation program. But what medication and treatment could this be that raised one up and back into full time work?

I think access to effective medical care would make this patient an outlier. Probably not a person in a position to have the most realistic assessment of the overall experience for patients.