BMJ: Rapid response to 'Updated NICE guidance on CFS', 2021, Jason Busse et al, Co-chair and members of the GRADE working group

So a study which found a negative outcome [ability to return to work] was adopted as the basis for Government policy. I'm tempted to say unbelievable - but then it (unfortunately) is believable. I've previously highlighted the old internal joke about the Governments official line being "evidence based policy development" then there's the other sort "policy based evidence development" - generally where the evidence doesn't support the Minister's/systems preferred outcome --- here?

The other thing is that the study was very poorly designed i.e. if a previous study had used objective measures (activity monitors) then why weren't the used here - conspiracy or cock-up? Obviously the Government officials should have raised the issue - submit a freedom of information request to check that? Also, the academics weren't exactly up front when they submitted a flawed study i.e. they should have known about activity monitors/objective assessment - caveat emptor - or the academics didn't give a shit --- they after all are healthy! From the outside I'd say that these guys shouldn't be getting any more public money --- but they may be mixing in the right circles!

 

Thank you.

"meaningless games with words and statistics" ---
"It is not clear that patients' self-perception has actually changed, or just their scoring behaviour, which are two different things."

seems to sum it up - damming.

EDIT - if the intervention doesn't work, and the exercise component is potentially harmful, then rather than roll out the (ineffective) intervention just give the folks with ME the money the intervention costs ---- spa treatments or whatever --- OK I'm being flippant. Just fund a GWAS study, expanded proteomics study (Hanson's recent one) ---

 

Don't think so. The main difference seems to be that the Cochrane review has thrown all of the different types of GET interventions into one big comparison while NICE has made different tables for many of them.

The approach by Cochrane is useful if you want to see the effect when all interventions are combined but there are also some arguments why they shouldn't be combined. Some say for example that the Wallman trial was more like pacing than GET. Other trials used pragmatic rehabilitation which included not only GET but also patients education inspired by CBT. Some trials used treatment as usual as comparison while others relaxation therapy etc. The results in the Cochrane meta-analysis suffers from high heterogeneity, possibly because they combined these different approaches.

The NICE report splits many of these into different meta-analyses. It makes some comments about this. for example in the general introduction:

And in the section on GET:

Unfortunately, the NICE overview does not mention which studies are included in the meta-analysis but I could take an educated guess.
The Cochrane review includes eight studies:

Fulcher 1997
Wearden 1998
Powell 2001
Wallman 2004
Moss Morris 2005
Jason 2007
Wearden 2010 (FINE)
White 2011

I think these were subgrouped in the NICE report as follows:

Table 30: Clinical evidence summary: Pragmatic rehabilitation versus Supportive listening: adults, severity mixd or unclear
1 trial: Wearden 2010

Table 31: Clinical evidence summary: Pragmatic rehabilitation versus Usual care: adults, severity mixed or unclear
1 trial: Wearden 2010

Table 37: Clinical evidence summary: Graded exercise therapy versus standard care: adults, severity mixed or unclear
2 trials: Moss-Morris 2005, White 2011

Table 38: Clinical evidence summary: Graded exercise therapy versus flexibility/relaxation treatment: adults, severity mixed or 3 unclear
2 studies: Fulcher 1997, Wallman 2004

Table 42: Clinical evidence summary: GET versus Activity diaries: adults, severity mixed or unclear
1 study: Wearden 1998

Table 43: Clinical evidence summary: GET versus Standard care: age and severity mixed or unclear
1 study: Powell 2001 (I think the title here should be GET + patient education versus standard care)

Table 49: Clinical evidence summary: Anaerobic activity therapy versus relaxation techniques: adults, moderate severity
Jason 2007

 

As Esther noted, NICE rated all the evidence in support of GET as low to very low quality. But the Cochrane review actually did the same with only one exception: fatigue measured post-treatment which was rated as moderate quality. So not exactly a big difference. According to Gordon Guyatt, however, Cochrane is an example of an appropriate application of GRADE while NICE is an example "disastrous misapplication of GRADE methodology". The emotive language is quite misguided.

The NICE report explains why it downgraded the quality of evidence.

Indirectness.
NICE downgraded for indirectness because the patient population in studies is not representative of ME/CFS patients for which the document was made. It explains that the Oxford and Fukuda criteria do not require PEM, even though this is now seen as a characteristic feature of ME/CFS:

Guyatt and colleagues that the Cochrane review carried a subgroup analysis, and there was little or no difference based on different diagnostic criteria. But this was a comparison between the Oxford and Fukuda criteria, not between those criteria and those that require PEM. So their comment is a bit off the mark.

Risk of bias - blinding
NICE has also downgraded because most studies used subjective outcomes and patients nor therapists were blinded. it explains:

Guyatt and colleagues argue that lack of blinding should not lead to downgrading the certainty of evidence. That's rather an unusual way of thinking. It either means that drug trials should no longer bother to blind patients and therapists because it does not influence the certainty of evidence, or it means that behavioural interventions are treated as an exception where blinding is not an issue while it is in drug trials.

Imprecision
In some cases, NICE also downgraded for imprecision. It used the following rule:

MID probably stands for minimal important difference. I had come to the same rule after reading the GRADE handbook- it's one of the reasons why I thought the certainty of evidence for fatigue in the Cochrane review should be downgraded: because the lower bound of the confidence interval is below the MID.

Guyatt and colleagues use a different rule: they argue that the lower bound of the confidence interval is still higher than 0.2 standard deviations, generally considered a small effect. Standard deviations in these randomized trials however were curtailed because of the use of entry criteria, so I think the MID approach is more reliable here. It's a matter of interpretation though.

 

Policy based evidence development for sure.

Don't forget that PACE was also part funded by......the Department for Work and Pensions.

Beyond part funding it and allowing the work and benefits outcomes to be downplayed, I think they just stood back and let the run amok.

I believe we found out about the dropping of the activity monitoring when some of their meeting notes came to light.

 

But surely "all studies gave the same direction and because the observed heterogeneity (80%) was mainly caused by a single outlier" suggests that in an unblinded intervention, with subjective outcomes (questionnaires), the Hawthorne effect is consistent - people respond positively to attention
https://en.wikipedia.org/wiki/Hawthorne_effect

At the very least they (can't use the word "researchers") should explain why it's not Hawthorne effect ---- of course it could be that they are ignoring evidence that doesn't support their case!

 

If people submit a rapid response, I think this paragraph from the authors should be highlighted as being outright false and dishonest (bolding mine):

The quotation about the use of CBT and GET comes from the introduction of the evidence review of non-pharmacological management for ME/CFS (p. 5). It is just introductory text, and nowhere do NICE say that it is a justification for including or excluding any kind of information from the evidence review.

And as pointed out by @Esther12 and @Michiel Tack, NICE exhaustively reviewed many different outcomes and provided GRADE evidence summary tables for each comparison of interventions. Does that really qualify -- almost 1050 pages of evidence review across both appendices G and H -- as "reject[ing] the randomized trial evidence (...) on the basis of theoretical arguments and anecdote"?

In fact, Busse et al. seem to be the ones who are rejecting the randomized trial evidence. Clearly, they cherry-picked the fatigue outcome post-treatment because it was the only one that the Cochrane review, of which they fancy the application of their GRADE methodology, rated as 'moderate' quality (and that is arguable). But included in the evidence summary tables provided by NICE are "patient-important" outcomes like quality of life, physical functioning and pain. This cherry-picking reeks of dishonesty, in a poor attempt to find a way to simultaneously promote GET and CBT for CFS while saving the face of the GRADE metholodogy (all very contrived).

Had they been honest, they would have entirely cited Cochrane's evidence summary of findings table for fatigue, which states:
- Fatigue (12-26 weeks post-treatment) outcome, moderate quality: "SMD is reduced to −0.44 if Powell 2001 is excluded from the analysis" (95% CI −0.63 to −0.24, but no longer clinically meaningful when considering the Minimal Important Difference instead of the 0.2 lower bound as a small effect size, as shown by @Michiel Tack in his commentary to Cochrane)
- Long-term fatigue (52-70 weeks post-treatment) outcome, very low quality: "SMD is reduced to −0.27 if Powell 2001 is excluded from the analysis" (95% CI −0.54 to 0.00, not clinically meaningful)

Their dishonesty is also highlighted by ignoring PEM as a "theoretical argument" and thus misrepresenting ME/CFS to lump it up with other "MUS" conditions: "Regarding directness, changes to diagnostic criteria for chronic fatigue syndrome (...) and other complex conditions that lack pathognomonic findings may or may not affect results."

In fact, besides NICE, when the Institute of Medicine used GRADE to rate the evidence they reviewed for their report on ME/CFS in 2015, they concluded that: "There is sufficient evidence that PEM is a primary feature that helps distinguish ME/CFS from other conditions". Here is the description of the IOM panel's methodology:

Adequately, the methodology for NICE's evidence review, based on GRADE, downgraded studies that did not require PEM due to indirectness (Appendix H, p. 15);

Their cherry-picking of the fatigue outcome blends in well with their omission of PEM, as evidenced by them pointing out that different diagnostic criteria do not affect the fatigue outcome in the Cochrane review even though none (Oxford and Fukuda) require PEM. But as I mentioned in my previous post, this is contradicted by the Agency for Healthcare Research and Quality's addendum to their review of treatments for ME/CFS, and there is no evidence available for physiotherapy interventions on patients with PEM (nor psychotherapies to the best of my knowledge).

 

I wonder if that could be highlighted in the media i.e. if the objective activity monitoring was dropped - pretty damning!
By the way who dropped the activity monitoring i.e. the "researchers" or the civil servants/Government?
If there's a link to the specific text then I'd be interested.

Can't help thinking @Jonathan Edwards is right i.e. this particular can of worms (PACE) might claim other "victims" --- no loss though, it might mean that public money is targeted towards research which has real benefits [GWAS ---]. Also, it might mean that evidence for public policy (including access to benefits) is subject to scrutiny.

 

That was the researchers themselves, I believe.

I know we discussed it on here somewhere. Who got hold of the TSG meeting notes, I can't quite remember?

@Robert 1973, maybe?

 

It was @JohnTheJack.

There is a good blog on “whatever happened to actigraphy?” by @Lucibee: https://lucibee.wordpress.com/2018/05/09/pace-trial-whatever-happened-to-actigraphy/

 

The randomized trial evidence is rejected because the quality of its evidence is dire, pure and simple, due to appalling trial methodology, erroneous presumptions, etc. And criticised not only by patients, but many scientists, clinicians, etc. A very selective bit of quoting I think.

 

Thanks for the helpful suggestions @Esther12 @cassava7 @MSEsperanza . I have adjusted my commentary accordingly.

The more I look into this, the more have the impression that Guyatt and colleagues view is directly at odds with what the GRADE handbook recommends.

As Jonathan has pointed out, the GRADE system has weaknesses that make it easy for researchers to present evidence as more robust than it actually is. Rather than critically reviewing the evidence, it encourages researchers to quickly check some boxes and follow the algorithm. But as the NICE committee has shown, one doesn't have to break the GRADE rules to come to a critical assessment of the evidence. One can also apply them thoughtfully.

Therefore I have adjusted my commentary to focus more on the fact that the NICE committee does follow GRADE appropriately. Here's what I got:

Is mesmerism effective after all?
I would like to respond to the comment by Jason Busse and colleagues as it includes some remarkable statements. The authors criticize the NICE guideline committee on myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) for employing “a disastrous misapplication of GRADE methodology.” In a draft document, the committee rated the quality of evidence for GET as low to very low.

As an example of an “appropriate” application of GRADE, Busse and colleagues refer to a contested Cochrane review on GET for ME/CFS. This review, however, also rated the quality of evidence in support of GET as low to very low with the sole exception of post-treatment fatigue where the quality of evidence was rated as moderate. At follow-up, however, the Cochrane review also rated the evidence that GET reduces fatigue as very low quality. This suggests that the difference between both assessments was rather small.

The NICE guideline committee gave several arguments for downgrading the certainty of evidence, which are all in accord with GRADE methodology.

1) Indirectness. The committee argues that post-exertional malaise (PEM) - a worsening of symptoms following exertion - is a characteristic feature of ME/CFS. Trials on GET used case definitions, such as the Oxford and Fukuda criteria, that were created in the 1990s and that do not require patients to experience PEM. The committee agreed that a population diagnosed with such criteria may not accurately represent the ME/CFS population and that people experiencing PEM are likely to respond differently to treatment than those who do not experience it. It was therefore agreed to downgrade the evidence for population indirectness. This is in agreement with other systematic reviews, which also differentiate between case definitions that require PEM and those that do not. [1, 2] Busse and colleagues refer to the Cochrane review which performed a subgroup analysis showing “little or no difference between subgroups based on different diagnostic criteria.” All included studies in this review, however, used the Oxford or Fukuda criteria which do not require PEM, making the statement rather misleading.

2) Imprecision. The committee downgraded for imprecision when a confidence interval crossed the minimally important difference (MID). This is in agreement with the GRADE handbook which suggests downgrading for imprecision “if a recommendation would be altered if the lower versus the upper boundary of the CI represented the true underlying effect.” Busse and colleagues, however, argue that researchers should not rate down for imprecision if the lower boundary of the confidence interval excludes a difference of 0.2 standard deviations. The authors do not clarify why such a small effect should be regarded as the clinical decision threshold. Independent estimates of the MID are a more appropriate choice.

3) Heterogeneity: The Cochrane review compared different forms of exercise therapy. In the trial by Wallman et al., for example, patients could reduce their activity level if exercise made them feel unwell while other forms of GET were strictly time-contingent. The trial by Jason et al. used anaerobic exercise, while others focused on aerobic exercise. The FINE trial did not prescribe GET but ‘pragmatic rehabilitation’: an intervention that was delivered by nurses at home. The trial by Powell et al. tested exercise therapy combined with patient education based on cognitive-behavioral principles. By combining these different interventions in one meta-analysis, the estimates found in the Cochrane review resulted in high heterogeneity. The NICE committee, therefore, decided to make greater differentiation between these different forms of exercise therapy by performing multiple meta-analyses.

4) Risk of bias. The committee noted that GET-trials used subjective outcomes even though patients nor therapists could be blinded to treatment allocation. This combination was considered an important limitation when interpreting the evidence. The figures cited by Busse and colleagues compare GET to a passive control condition where patients received less time and attention from healthcare providers. Patients in the GET-group also received instructions to interpret their symptoms as less threatening and more benign. According to one therapist manual on GET “participants are encouraged to see symptoms as temporary and reversible, as a result of their current physical weakness, and not as signs of progressive pathology.” Treatment manuals also included strong assertions designed to strengthen patients’ expectations of GET. One patient booklet stated: “You will experience a snowballing effect as increasing fitness leads to increasing confidence in your ability. You will have conquered CFS by your own effort and you will be back in control of your body again.” Patients in the control group received no such instructions. There is therefore a reasonable concern that the reduction on fatigue questionnaires in the GET group reflects response bias rather than a genuine reduction in fatigue. Other reviews have previously come to a similar conclusion. [3, 4]

The recommendation by Busse and colleagues that lack of blinding should not result in downgrading quality of evidence, even if subjective outcomes are used, is at odds with current understanding [5] and has far-reaching implications. It would either mean that drug trialists should no longer attempt to blind patients and therapists (because this wouldn’t affect the quality of evidence) or that behavioral interventions should be treated as an exception where risk of response bias can freely be ignored because it is practically not feasible to blind patients and therapists. Additionally, if the GRADE system was used as Busse and colleagues recommend, there would be a high risk that quack treatments and various forms of pseudo-science also provide 'reliable' evidence of effectiveness in randomized trials. All that is needed is an intervention where therapists actively manipulate how patients interpret and report their symptoms. One example should suffice to clarify this point.

Suppose an intervention based on ‘neurolinguistic programming’ where therapists assume that saying one is fatigued, reinforces neural circuits that perpetuate fatigue. The intervention consists of breaking this vicious cycle by encouraging patients to no longer see or report themselves as fatigued. This example is not that far-fetched as there are already behavioral interventions for ME/CFS that are based on similar principles. [6] According to the GRADE methodology specified by Busse et al., however, such attempts to manipulate how patients report their symptoms, form no reason to downgrade the quality of evidence of randomized trials, even if fatigue questionnaires are used as the primary outcome.

The first and foremost principle of rating quality of evidence should be to understand the specifics of what is being assessed. One has to understand the intervention and the way it impacts patients. By providing a standardized checklist and algorithm to assess quality of evidence, the GRADE methodology discourages researchers from studying the details of what happens in randomized trials. The rapid response by Busse and colleagues is an example of how this approach might result in questionable treatment recommendations.

References
1. Smith MEB, Nelson HD, Haney E, Pappas M, Daeges M, Wasson N, et al. Diagnosis and Treatment of Myalgic Encephalomyelitis/ Chronic Fatigue Syndrome. Evidence Report/Technology Assessment Number 219. July 2016 Addendum. Agency for Healthcare Research and Quality (US); 2016. https://www.ncbi.nlm.nih.gov/books/NBK293931/pdf/Bookshelf_NBK293931.pdf. Accessed 20 Apr 2020.

2. Wormgoor MEA, Rodenburg SC. The evidence base for physiotherapy in myalgic encephalomyelitis/chronic fatigue syndrome when considering post-exertional malaise: a systematic review and narrative synthesis. J Transl Med. 2021;19:1.

3. Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018;5:2055102918805187.

4. Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue: Biomedicine, Health & Behavior. 2020;8:181–92.

5. Hróbjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol. 2014;43:1272–83.

6. Reme SE, Archer N, Chalder T. Experiences of young people who have undergone the Lightning Process to treat chronic fatigue syndrome/myalgic encephalomyelitis--a qualitative study. Br J Health Psychol. 2013;18:508–25.

 

It would be good to add here that the committee's view on PEM being a characteristic feature of ME/CFS is generally in line with the view of ME/CFS experts and the IOM report. This shows that the NICE committee is following scientific mainstream opinion and not making up arbitrary rules to exclude evidence they don't like. It could also be good to point out that abnormal physiological responses in patients with PEM have been demonstrated,showing that these patients do not respond to exercise in the same way as others.

 

Here's an extract but it's all pretty daming.

“Although we originally planned to use actigraphy as an outcome measure, as well as a baseline measure, we decided that a test that required participants to wear an actometer around their ankle for a week was too great a burden at the end of the trial. We will however test baseline actigraphy as a moderator of outcome.“

From the original FAQs:



However, looking at the trial management committee minutes, the decision to drop actigraphy as an outcome measure seems to have been made because they knew it wouldn’t confirm that participants had increased their activity, and not because it was “too great a burden” for participants. If anything, it seems that it was too great a burden for the analysis team, as there seemed to be problems extracting the baseline data and getting it into an analysable format. Five pages of data needed to be completed by the centre staff for each participant, so I suspect there were loads of issues with missing or incomplete data that hampered any chance of getting any useful measurements even at baseline. There were also issues with the availability of the actiwatches themselves (too few per centre had been ordered). It very much seems that use of actigraphy was not properly field tested before the trial started. For such an important trial, this seems to be a massive oversight.

From TMG minutes (5 Nov 2004):"


Umh ---- your contacted to do a study, paid for out of public money [who - which Government Department "oversaw" the contract?], and which would become the basis of Government policy, and you and a charity [Action for ME doubtless] decide to scrap objective monitoring. Sorry why did they get paid? You don't do what your contracted to do and you get paid --- OK I've done that in a private capacity but this was public money.

Was the change to the project signed off by the management group i.e. Government side? I wonder if the public accounts committee looked at this - seems that the contract wasn't properly completed but it was paid for!

 

Yes, I would succinctly add, after "is a characteristic feature of ME/CFS", the following: "in line with the US National Academy of Medicine's evidence review for its 2015 report on ME/CFS [x]"

And after "This is in agreement with other systematic reviews (...)":

"and the recommendations from the NIH that "studies [on ME/CFS] should report findings according to important features of ME/CFS, such as postexertional malaise" and that the Oxford criteria should be retired [y]".

[x] Institute of Medicine. 2015. Beyond Myalgic Encephalomyelitis/Chronic Fatigue Syndrome: Redefining an Illness. Washington, DC: The National Academies Press. https://doi.org/10.17226/19012

[y] Haney E, Smith ME, McDonagh M, Pappas M, Daeges M, Wasson N, Nelson HD. Diagnostic Methods for Myalgic Encephalomyelitis/Chronic Fatigue Syndrome: A Systematic Review for a National Institutes of Health Pathways to Prevention Workshop. Ann Intern Med. 2015 Jun 16;162(12):834-40. doi: 10.7326/M15-0443. PMID: 26075754.

 

I wondered about adding a quickie comment something like:

The contribution to this debate by Busse et al. is interesting. Turner-Stokes and Wade argued that GRADE is too strict. Busse et al. say no, and (re: assessment of evidence for therapist-delivered treatments for ME/CFS) 'An appropriate application of GRADE would have come to a very different conclusion.' - apparently an endorsement. Yet, as indicated in my expert witness statement, included in the draft guideline, any reasonably sensible person can see that we have factual evidence for both the unreliability of the methods used and the trivial nature of any apparent positive signal. This must mean that GRADE, interpreted by its own people, is not fit for purpose.

Both Turner-Stokes and Busse blame pressure from patients. I doubt either is privy to what went on and the draft guideline does not, as incorrectly implied by Busse, tell us. From what I see, the committee was influenced by the evidence, including that in my testimony, despite their being tied to the pseudo-arithmetic of GRADE. I am not a patient and have no competing interest. I was invited to give testimony as I have experience with trials and their problems. I am simply appalled at the poor quality of analysis of trials by people who are supposed to understand these things. Have they read the trials (and critiques)?



Comments?

 

I think you should post. And you should ask whether they have read the trials and the critiques. The people commenting on this topic don't seem to have a good understanding of PACE and ME/CFS. I think they do not understand what they are defending.

The comment they make about case definition not making a difference, when it's based on a comparison of Oxford and Fukuda criteria, is probably not intentionally misleading. They probably just don't know that there's this thing called PEM or that it's important, that later case definitions exist, what case definitions are considered good, what mainstream view of all this is.