The difficulties of conducting intervention trials for the treatment of [ME/CFS]: Expert testimony to NICE guidelines committee by Jonathan Edwards

Thought it could be helpful to have a thread on Jonathan's expert testimony for any further use and citation.

Edwards, Jonathan CW (2020), The difficulties of conducting intervention trials for the treatment of myalgic encephalomyelitis/chronic fatigue syndrome: Expert testimony presented to the NICE guideline committee, 06.09.2019, https://www.nice.org.uk/guidance/GID-NG10091/documents/supporting-documentation-3

Summary

Therapeutic trials for ME/CFS are particularly at risk from expectation bias because of the subjective core features and fluctuating nature of the illness. Therapist-delivered treatments, with lack of blinding, and particularly those involving cognitive intervention, are even more at risk because of role-playing and deliberate influence on attitudes to health status.

Trials of drug treatments have tended to navigate difficulties adequately but have yielded negative (or equivocal) results. Trials of therapist-delivered treatments to date have not met basic requirements for minimising bias and are therefore unsuitable as an evidence base for treatment recommendations. Inasmuch as they are interpretable, they suggest that although attitudes to health status maybe influenced this does not lead to significant improvement in objectively measures of disability.

There are also specific ethical concerns with trials of treatments that aim to persuade patients of an unproven theoretical model for their illness and for which there is prima facie evidence of significant harm (for exercise therapy).

The failure to meet basic requirements indicates a problem with quality control through peer review in this area, as illustrated with problems with Cochrane reviews (see bibliography: Vink and Vink-Niese, 2018; Vink and Vink-Niese, 2019).

Trials to date have failed to make use of methods that can significantly mitigate difficulties with bias from subjective outcomes, but it may be that for unblindable treatments new forms of trial methodology will be needed if useful assessment is to be achieved. In my view, the uninterpretability of evidence from trials that fail to overcome basic difficulties, combined with ethical concerns, means that it is inappropriate to recommend either cognitive behavioural therapy or exercise therapies for ME/CFS.

(Paragraphing inserted.)

Please let me know if the title/ citation needs any correction. Also, feel free to move it to another place. Perhaps a new subforum on trial methodology added to 'ME/CFS News and Research' forums could be a good thing?

 

https://twitter.com/user/status/1326633085274877955

Code:

https://twitter.com/RobertHMcMullen/status/1326633085274877955

 

My flippant initial response was 'Ouch!'...but it's also really interesting to see the language, approach, and style of argument used in this kind of review.

Huge thank you to @Jonathan Edwards for the experience and wisdom you've brought to this, the time you've given up – and for the patience and humour whilst listening to us chuntering and wittering over the last five years!

 

Thank you so much for such a detailed, informative, and very definitive, strongly-worded review, @Jonathan Edwards! I was especially pleased to see the inclusion of the paper co-authored by the sadly-late Robert Courtenay, aka @Bob. How I wish he were still here...

 

Thank you, Prof Jonathon Edwards...

For your excellent testimony to the NICE Draft Guidelines.

I have had severe ME since 1998 and rarely able to read, let alone comment. But I have managed to read your submission to NICE and would personally like to thank you for your interest and insight, and valuable contribution to ME advocacy.

Thank you so much for giving your time to help us. It is greatly appreciated, especially to those of us, housebound and bedbound for decades.

Tamzin Mole aka Fero

https://www.nice.org.uk/guidance/GID-NG10091/documents/supporting-documentation-3

 

@Simon M there is an error in the picture. It says "blinded trials, subjective outcomes" when it should say "unblinded trials, subjective outcomes".

 

@Simon M , I've spotted a mangled word in the quote at top of article

"it should not recommend eitehr CBT"

And did you mean to insert more links in the section at the bottom of the article, the number of non-hyper linked authors makes it look like you did.

 

Trying again:

The expert testimony to NICE that took apart the case for CBT

Professor Jonathan Edwards told NICE it should not recommend eitehr CBT or graded exercise as all the trial evidence for them used subjective outcomes in unblinded trials, giving unreliable results. He showed why blinding in ME/CFS trials is essential, despite PACE supporters claiming it is not. Edwards said it was also unethical to tell patients that the treatmens are safe, effective and based on sound theories as there are serious doubt about all three claims. Finally, he argued that quality control in research for these kind of studies is broken and needs fixing.​

In 2007, NICE recommended cognitive behavioural therapy (CBT) and graded exercise therapy as the only suitable treatments for ME/CFS. Its new draft guidelines, published in November 2020, say no to all graded exercise. And no to any CBT that assumes ME/CFS is caused by patients’ flawed beliefs or behaviours.

The driving force for this U-turn was the guideline committee’s evidence review, which rated almost all of the evidence for CBT or graded exercise as very low quality. The remainder was rated as low quality.

Probably the main reason for these damning ratings was Professor Jonathan Edwards’ expert testimony to the ME/CFS guidelines committee.

His testimony brilliantly exposed the deep flaws in the evidence for CBT and graded exercise. Effectively, Edwards, a specialist in clinical trial design, hammered a box of nails into the coffins of CBT and graded exercise for ME/CFS.

Who is Professor Jonathan Edwards? An independent expert clinician and researcher
The central problem: flawed ME/CFS treatment studies tell us nothing useful
Edwards highlighted the central problem, that CBT and graded exercise studies have relied on biased methods that produce unreliable results.

Most clinical trials of drugs are run blinded, which means that patients don’t know if they are getting the active drug or are receiving a dummy pill in the “control” group.

Success is defined not as the improvement in the drug group alone, but whether or not people in the drug group improve by substantially more than those in the placebo group.

Such blinding is usually required in biomedical treatment studies because it is well known that if patients are asked how they feel, their answers can be biased because they expect to feel better or they want to please researchers. This is known as expectation bias and blinding is there to protect against this.

Blinding is much less of an issue when a study uses objective outcomes, such as changes in blood sugar levels or exercise capacity. However, for subjective outcomes, such as patients scoring their own fatigue levels, expectation bias is a major problem.


Edwards told the committee that bias “is an aspect of human nature that affects all types of scientific experiment, whether in the lab or in the clinic. If outcome measures are truly objective blinding is not necessary, and vice versa. For this reason, unblinded trials with subjective outcomes are specifically considered unreliable.”

The PACE trial as well as all the other psychosocial treatment trials in ME/CFS were unblinded, and so problems of bias, said Edwards, “make more or less all the trials to date unsuitable as a basis for treatment recommendation.”

But does blinding really matter for ME/CFS?
Despite the compelling case for bias undermining trials of CBT and graded exercise, the PACE trial authors and their supporters have tried to justify the continued use of flawed methods.

...

The blinding conundrum – no free pass for psychosocial research
Edwards argued throughout his testimony that the high standards required for drug trials should apply equally to psychosocial treatment trials.

Of course, it simply isn’t possible to conceal from a patient that they are getting CBT or graded exercise. That blinding is not possible does not make the problem go away. Instead, says Edwards, it is down to the researchers to solve this problem in order to get reliable results.

He added, “Unreliable results cannot be seen as reliable just because it is difficult to get more reliable ones.”

E...

Unethical to continue using CBT and graded exercise
Edwards pointed to three ethical concerns with using these treatments.

...
Given the problems with the claims made for CBT and graded exercise, Edwards said, it is unethical to tell patients the treatments are based on sound theory, unethical to say they are safe and unethical to claim they work.

Quality control: the “broken field” and the unsayable criticism
That CBT and graded exercise continue to be used and recommended despite the evidence shows, said Edwards, that quality control has broken down.

It became clear then that the existing approach of using unblinded studies with subjective outcomes couldn’t produce useful results. So Edwards was surprised to discover years later that researchers of psychosocial therapies for ME/CFS were still pursuing such a flawed approach.

“the disciplines involved need to take a long hard look at themselves.”

Research quality is supposed to be policed by peer review, where fellow researchers check the standards of studies submitted for funding or publication.

This requires that the peer reviewers understand the problems and reject work that is flawed. It appears that they don’t, as Edwards illustrated with three examples.

...

Why CBT and graded exercise have to go

Edwards concluded by telling the NICE guidelines committee he believed NICE should not recommend either CBT or graded exercise for ME/CFS. He said this applies to both adults and children. The evidence for the therapies is unreliable because of bias and there are significant ethical problems with their continued use.

Professor Jonathan Edwards’ expert testimony is a brilliant and brutal demolition of the case for CBT and graded exercise as treatments for ME/CFS. While he drew on the evidence of many others, his testimony is perhaps the most impressive single critique of flaws in the evidence used to justify these treatments. It deserves to get a wider audience.

===

Researchers whose work pointing out flaws in ME/CFS research was referenced by Professor Edwards
Lead authors: Tom Kindlon, Dr Carolyn Wilshire, Mark Vink, Alexandra Vink-Niese, Dr Keith Geraghty, Professor Brian Hughes. Co-authors: Robert Courtney (RIP), Alem Matthees, David Tuller, Professor Bruce Levin. Links are to work referenced by Edwards but not included in my blog.

 

in a comment in reply to Jerome Burns recent piece on new guidelines:

https://www.s4me.info/threads/nice-...-10th-november-2020.17687/page-26#post-313059

 

It also does not factor in externalities, the hidden costs borne by others (think pollution, it's cheaper for a factory to dump toxic chemicals in a waterstream but someone will have to pay to clean it up at some point), or the opportunity cost, the benefits lost by making choices that close the door to other choices. Most of the externalities are borne by us, medicine is fully blind to it, doesn't even consider it. The opportunity costs are mainly the fact that medicine has made exactly zero progress, has in fact regressed, in the area of chronic illness. On purpose, by choosing delusional fantasies. This is one of the main arguments against alternative medicine. Also why Steve Jobs died.

The entire basis of cost-benefit analysis in the BPS ideology is to only count some things, the accounting equivalent of committing fraud by doing things like passing income as losses or burying expenses so as not to appear connected to their purpose. When considering the societal costs, you can't focus on some pockets and ignore the others. They all add up in the end.

The economic costs have been calculated and vary from a recent estimate of, IIRC, $13B per year in Australia and $17-20B in the US. I think the Australian estimate is more likely to be accurate, the IOM estimate was a very conservative one. This amounts to far more than $100B per year in direct and indirect economic losses, which adds up to more than $1T per decade.

When a cost-benefit analysis does not even consider decadal losses on the scale of trillions, it is simply not a serious analysis. This is made with extreme bias in a way that would make any serious economist cringe with despair. The entire basis for the economic benefits of medical austerity is a mix of fraud and delusion. Only a tiny few actually benefit. It is a bona fide Big Lie.

 

And yet time and time again the PACE authors and their endorsers have said that GET/CBT should remain in use because "it is the best there is". A bit like saying you'll have to eat their sh*t because there's nothing else in the fridge.

 

I hope we will also see an influential article on the economic losses caused by ME/CFS and the extreme foolishness of not even trying to address this. We've had some academic papers but we need something for a broad audience.

 

I wonder how much influence Jonathan's excellent testimony had an influence on the grading of GET/CBT trials as low to very low-quality evidence.

Because his testimony was before the guideline development group but the summary and grading of evidence was prepared by other people if I understand correctly (Can't remember their name)

 

Initial grading was prepared by the in house NICE team but was then reviewed by the committee.
The final grading in the draft guideline came much later.
I could not possibly comment on what caused what!

 

In which case I would be astonished if @Jonathan Edwards's testimony did not help NICE set the bar for evidence quality at a proper scientific level, when the evidence gradings were being established.

I am also hopeful that even though the guideline is only at the draft stage, it would be extremely difficult for NICE to now backtrack on their evidence gradings. I mean ... it really would make them look like they didn't know what they were doing wouldn't it .

 

I'm not sure but 'influence' seems to have different meanings or connotations.

E.g. it could just mean that someone gave input and that people learned something from that input and reflected on their knowledge and their task.

I assume that the people in charge at NICE for overseeing and doing the grading of evidence are many.

I'm still concerned that NICE guidelines on other illnesses where therapist-delivered treatments are taken into account, the grading of evidence looks a bit different. I checked that a while back and don't remember now for sure, but my impression was that for other illnesses NICE didn't apply the standards needed to grade the evidence of psychological interventions and physiotherapy properly.

So to me it seems interesting whether the insights and results of the grading process for the NICE guideline on ME -- the fact that grading of evidence from trials on therapist-delivered treatment can't be allowed to apply less rigorous methods than trials on drug treatments -- are new. And if they are new, how will the revision of other guidelines look like in the future?

Given that some people didn't like it when Cochrane suddenly questioned the former grading of their trials(*), and given that not only science journals but also diverse oversight committees aren't always convinced by facts and reason, to see that now also NICE don't buy their attempts at exculpating bad science, I'm afraid that there is no reason to be self-assured regarding the draft guideline on ME.

Having said that, I think Jo's expert testimony was extremely important and share the positive views on the draft guideline as a major step forwards. I just think people need to be prepared that there will be more and more influential attempts at dismissing and discrediting this major success for science and for pwME than the attempts we are seeing on Twitter and by the BMJ (see https://www.s4me.info/threads/bmj-e...-fatigue-syndrome-2020-stokes-and-wade.18233/ )

For me, the most encouraging part of the story so far is that people at an important health care institution in the UK seem to have chosen facts and reason over influence and eminence.

Sorry for the long sentences, not able to word that better at the moment.

Only now I see that fine wording by Barry:

I so much hope you are right.

(*) Edited to add: (finally, after Cochrane had ignored valid criticism by pwME and their allies for years.)

 

Thanks for clarifying.

There are quite a lot of proponents of behavioural interventions for ME/CFS in the guideline development group, so your testimony must have made quite an impression.

 

Allow me to speculate.

Presumably, NICE undertook some kind of evidence review in 2007 and found the evidence (pre-PACE, even lower quality) for CBT and GET to be acceptable.

Whatever system was used in 2007, in 2020 NICE used the GRADE system. Interestingly, Cochrane had used exactly the same system to give PACE and similar a high rating for research quality. So there are reasons to suspect then may be some new reason why NICE rated the evidence as low quality or very low-quality this time around.

It's notable that the main reason given for the raging of low and very low-quality was bias. And the problems of bias in psychosocial research was the central theme of @Jonathan Edwards evidence.

Anyway: draw your own conclusions.


PS Given that NICE had invited this expert testimony and was, I think, obliged to publish it, it would have been difficult for proponents of psychosocial research on the guidelines committee to ignore problems of bias.


Given that NICE had invited this expert testimony and was, I think, obliged to publish it, it would have been difficult for proponents of psychosocial research on the guidelines committee to ignore problems of bias.