A general thread on the PACE trial!

[Jonathan Edwards]

It was clear from the beginning that the outcomes differed from the protocol ones.

It was clear from the beginning that as an open trial with subjective outcomes it was dead in the water, but my understanding is that the pushback from organisations like MEA was on grounds that were pretty debatable.

 

[Evergreen]

Thanks for your reply.
Interesting @Jonathan Edwards plausibly suggested that PACE may have had its origins in the preparation of the previous NICE Guideline [experts from a UK University - which uni?]. Basically all of the studies indicated that these interventions [exercise (& CBT?)] worked; however, one of those on the panel had published a research paper highlighting that they were all methodologically flawed - so potentially they highlighted that a better study was required!
What is the earliest business case/protocol in the public domain and what does it say?

I don't know, perhaps people like @Dolphin would recall? Certainly the earliest document in the Trial Management Group meeting minutes - from June 2002 - references previous meetings and a protocol. So there were drafts pre- June 2002. At a meeting in September 2003, the primary outcome measures were already nailed down:

11. Primary outcome measures
Our current positive symptom outcome is less than 4 on the 11 item Chalder fatigue questionnaire. This is equivalent to normal healthy population levels of fatigue, so this would represent recovery rather than clinically significant improvement. It was agreed that we should propose changing the positive symptom outcome to a 50 per cent reduction in baseline score as a better representation of improvement, with a score of 3 or less being a secondary outcome measure.
We noted that a score of 75 or more on the SF-36 physical function sub-scale represents a score of one standard deviation below the working adult UK population score, which was considered reasonable.

But the criteria for positive outcomes were changed both between this point and the published protocol, and between the published protocol and the published results, if I've understood correctly

What does "protocol paper---does not list actigraphy as an outcome measure, only as a predictor" mean in practice?

Outcome measures are measured twice, before therapy and after therapy, to see whether someone improved.

Predictors are measured once, before therapy, so that you can see if any of them are useful for predicting whether someone improves or not. For example, you might want to know if people who are sick for less than 2 years and more likely to improve than people sick for longer, because it could help you direct services towards the people most likely to benefit. So their interest in actigraphy was whether a particular activity pattern - eg boom and bust - predicted whether someone improved or not in the trial.

Some things are both outcome measures and predictors, because you might want to know whether someone's depression improved with therapy AND whether being depressed at baseline made you more or less likely to improve with therapy.

The predictors from the published protocol of the PACE trial were:

Predictors
1. Sex
2. Age
3. Duration of CFS/ME (months)
4. 1 week of actigraphy [18] (as initiated at visit 1 with the
research nurse)
5. Body mass index (measure weight in kg and height in
metres)
6. The CDC criteria for CFS [1]
7. The London criteria for myalgic encephalomyelitis [40]
8. Presence or absence of "fibromyalgia" [41]
9. Jenkins sleep scale of subjective sleep problems [37]
10. Symptom interpretation questionnaire [34]
11. Preferred treatment group
12. Self-efficacy for managing chronic disease scale [32]
13. Somatisation (from 15 item physical symptoms PHQ
sub-scale) [35]
14. Depressive disorder (major and minor depressive disorder,
dysthymia by DSMIV) (from SCID) [30]
15. The Hospital Anxiety and Depression Scale (HADS)
[38] combined score
16. Receipt of ill-health benefits or pension
17. In dispute/negotiation of benefits or pension
18. Current and specific membership of a self-help group
(specific question)

Wow - would you expect a contractor to decide what you should get as an outcome? As per my previous comment I'd like to see the original business case and see what the intention re outcome indicators was. It seems the contract was shambolic - you can trust an inhouse (civil servant in this case) expert - but relying on the contractor to set outcome indicators/purpose of the study --- beggars belief!

Sorry I don't understand this bit.

Begs the question what the propose of the study was!

They didn't seem to see any problem with having only subjective primary outcome measures in an unblinded/unblindable trial. Nor do countless other researchers. Their original goal was to see which was better between CBT and GET.

 

[Evergreen]

I wonder if the campaign to stop PACE originally was based on the poor trial design?

My guess is that it was about negative patient feedback on exercise and CBT, and possibly about physical vs psychological, but I haven't found the relevant documents.

 

[rvallee]

Also since FINE was mentioned, a timely reminder that this is the trial that (IIRC) had a primary objective outcome that was swapped for a subjective one, and where the published paper had the famous quote about how "the bastards just don't want to get better". It was a terrible offensive trial of no credible or scientific purpose and it should never have happened.

It was also a complete bust, showed no positive results, and yet still formed the basis for a NHS module on CFS. The whole thing was even more shambolic than PACE, and I still maintain that PACE was fraudulent in the way it was reported, had a piss-poor design and served no legitimate professional or scientific purpose either.

So opposition to both was well-warranted, they served no legitimate purpose and ultimately set everything back.

 

[rvallee]

Interesting @Jonathan Edwards plausibly suggested that PACE may have had its origins in the preparation of the previous NICE Guideline [experts from a UK University - which uni?]. Basically all of the studies indicated that these interventions [exercise (& CBT?)] worked; however, one of those on the panel had published a research paper highlighting that they were all methodologically flawed - so potentially they highlighted that a better study was required!

Maybe I'm misremembering, but I think that the 2007 guideline actually mentioned an upcoming trial that would prove it correct, likely referencing PACE. It was really all set up together, with the main piece of evidence happening in the future, so that piece of evidence could not fail, hence they were allowed to blatantly cheat.

 

[FMMM1]

Thank you.

I think that the 2007 guideline actually mentioned an upcoming trial

Interesting - so maybe check the previous Guideline.

 

[Evergreen]

The working group that produced the 2002 report to the Chief Medical Officer included Shepherd, Pheby as well as Chalder, Chris Clark (AfME), White. Names we'd know like Anne Macintyre, Speight and Weir were also involved. So these quotes might give an idea of what early objections to the PACE and FINE trials would have been:
(full report here http://www.erythos.com/gibsonenquiry/docs/cmoreport.pdf)

[Re GET]
One key controversy that exists over graded exercise rests on whether the nature of
the treatment is appropriate for the nature of the disease, at least in some individuals.
Existing concerns from voluntary organisations and some clinicians include
the view that patients have a primary disease process that is not responsive to or
could progress with graded exercise, and that some individuals are already
functioning at or very near maximum levels of activity.

[Re CBT]
It is important to note that a specific or shared belief system is not essential to
apply the principles of cognitive behavioural therapy to CFS/ME. However, the
wider uncertainty surrounding the nature of CFS/ME does impact on perceptions
and delivery of the therapy in individuals. Difficulties can also arise when therapist
and patient share differing beliefs about the individual’s illness, and the nature of
CFS/ME. Patients may have an understandable apprehension about increasing
activity, so it is important that changes are mutually agreed and the patient is
supported through the process.

 

[Medfeb]

I wonder if the campaign to stop PACE originally was based on the poor trial design?
My memory is that the failings trial design did not really surface until 2015. Prior to that most of the complaints were about the diagnostic criteria for entry, which may not have suited PWME/CFS but were scientifically legitimate. Even the US committee that advised against GET in ?2015 seemed to base its decision on that.

In the US in 2015, the CDC was still saying that PACE-style GET could help some patients. CDC dropped that in 2017 as a result of advocacy on all the issues with trial design - including how they selected patients.

Regarding the diagnostic criteria - wouldn't there be a question of whether its scientifically legitimate in this specific case to generalize the exercise recommendations from cohorts who were not required to have PEM to people who all have an abnormal response to exertion?

 

[Jonathan Edwards]

So these quotes might give an idea of what early objections to the PACE and FINE trials would have been:

This would presumably be at the stage of planning to set up a trial. The quotes are quite interesting in that they suggest that those with inside knowledge of ME/CFS saw CBT and GET as pretty unlikely to help on the basis that people had already tried exercise mostly and not managed and that CBT was a bit of a theoretical muddle.

It is interesting to note that this is only a few years (?4) after Chalder first proposed an optimal method for CBT for ME/CFS - without having done any trials of course.

 

[Jonathan Edwards]

Regarding the diagnostic criteria - wouldn't there be a question of whether its scientifically legitimate in this specific case to generalize the exercise recommendations from cohorts who were not required to have PEM to people who all have an abnormal response to exertion?

That can be raised as a concern for those wishing to have advice specifically for those with PEM but the trial would still be entirely scientifically legitimate - on its own terms. There are always caveats about subsets.

 

[Medfeb]

That can be raised as a concern for those wishing to have advice specifically for those with PEM but the trial would still be entirely scientifically legitimate - on its own terms. There are always caveats about subsets.

So in a very general sense, I can see the point that ME is a "subset" of the condition of "chronic fatigue." And presumably, there are many different conditions that have chronic fatigue as a feature. But PACE was not claiming to be studying exercise in all such conditions. They were specifically claiming to be studying the group of people who have ME.

I can imagine one reason they are objecting to NICE requiring PEM is because they know NICE's new criteria scientifically invalidates the way they've been studying ME all along...

Edited to add
Of course, this on top of all the other problems with study design and conduct

 

[Sean]

Charity changes position after data released and reanalysed. Investigators change original protocol after themselves working in clinics and meeting others who had done so and after all the data had been collected.

Which ones are allowing their ideological bias to trump their respect for science?

The PACE authors and their supporters (including Wessely) were openly saying that they changed the outcome analysis protocol post-hoc because the results didn't match up with the results from earlier trials and their clinical experience.

Except that the whole purpose of PACE was supposed to be testing those earlier, and less methodologically robust, trials and clinical experience, which is how it should be.

I don't know how much more brazenly anti-science and fraudulent it could get, including on harms measures.

The fact that they were not openly laughed out of the room and immediately removed from all positions of power and influence shows just how broken the rest of medicine is on this issue.

 

[Evergreen]

This would presumably be at the stage of planning to set up a trial.

Exactly. The CFS/ME Working Group began their work in 1998, culminating in their report to the CMO published in January 2002. The first minuted PACE trial meeting was in June 2002, and those minutes reference earlier meetings. In July 2004, the MEA publishes this: ME Association. MEA calls for PACE trial to be scrapped. ME Essential, July 2004: 91: 3–4. The PACE trial management group meeting minutes refer to a campaign to stop the FINE and PACE trials from 2004. Recruitment for the FINE trial starts in February 2005 and for the PACE trial in March 2005.

 

[Evergreen]

I’ve always thought the timing of the PACE protocol changes need to be seen in the context of the earlier-finishing FINE trial.

The protocols for the FINE and PACE trials had the same criterion for improvement on the SF36PF scale (having agreed to swap protocols in a May 2003 TMG meeting). The FINE protocol confirms using the 11-point Chalder questionnaire, but does not specify what will count as improvement on that scale.

This table, from Wilshire et al 2018 (full reference below) shows the switch in criteria for improvement on the SF36PF and CFQ that was made between protocol and publication of the PACE trial ie 2007 vs 2011:



From: Wilshire CE, Kindlon T, Courtney R, Matthees A, Tuller D, Geraghty K, Levin B. Rethinking the treatment of chronic fatigue syndrome-a reanalysis and evaluation of findings from a recent major trial of graded exercise and CBT. BMC Psychol. 2018 Mar 22;6(1):6. doi: 10.1186/s40359-018-0218-3. PMID: 29562932; PMCID: PMC5863477.

On 29th April 2009, the PACE statistical analysis plan is discussed at a TSC meeting, but there is no mention of changing the criteria for improvement at this point.
On 13th May 2009 the FINE trial results were presented to the FINE TMC.
On 17 June 2009 the FINE trial results were presented to the PACE TMG.

The FINE trial results as published in 2010:


From: Wearden AJ, Dowrick C, Chew-Graham C, Bentall RP, Morriss RK, Peters S, Riste L, Richardson G, Lovell K, Dunn G; Fatigue Intervention by Nurses Evaluation (FINE) trial writing group and the FINE trial group. Nurse led, home based self help treatment for patients in primary care with chronic fatigue syndrome: randomised controlled trial. BMJ. 2010 Apr 23;340:c1777. doi: 10.1136/bmj.c1777. PMID: 20418251; PMCID: PMC2859122.

The FINE trial summarised their findings as follows:

Our study shows that, when compared with treatment as usual, pragmatic rehabilitation has a statistically significant but clinically modest beneficial effect on fatigue at the end of treatment (20 weeks), which is mostly maintained but no longer statistically significant at one year after finishing treatment (70 weeks). Pragmatic rehabilitation did not significantly improve our other primary outcome, physical functioning, at either 20 weeks or 70 weeks.

On 4 November 2009, less than 5 months after the FINE results were presented to the PACE TMG, TMG meeting minutes note changes to the analysis plan:

TSC/ASG[Redacted] fed back that the analysis strategy was close to completion. There had been a number of final issues to resolve. It was highlighted that an important change has been made to the reporting of the primary outcome measures. Previously it had been decided that the results would be presented categorically using thresholds derived from the binary scoring of the Chalder questionnaire and the continuous SF36 scale. It has since been decided that the original question posed by the study would be better answered by comparing the continuous scores on both the Chalder and SF36 scales. The originally planned categorical scores will also be reported in the main paper, as a secondary analysis, reflecting clinically important differences.

A few concerns were raised that making a change at this stage may invite criticism. It was highlighted that the change will increase the sensitivity of the study and that the changes have been made before the reviewing of any data, and that the change will be reported in the paper. It was agreed by the TMG that these changes should go ahead.[my bold]

White et al. describe the change in 2015 in BJPsych Bull. 2015 Feb; 39(1): 24–27:

We originally planned to use a composite outcome measure of the proportions of participants who met either a 50% reduction in the outcome score or a set threshold score for improvement. However, as we prepared our detailed statistical analysis plan, we quickly realised that a composite measure would be hard to interpret, and would not allow us to answer properly our primary questions of efficacy (i.e. comparing treatment effectiveness at reducing fatigue and disability). Before any examination of outcome data was started, and after approval by our independent steering and data monitoring committees, we decided to modify our method of analysis to one that simply compared scores between treatments at follow-up, adjusting the analysis by baseline scores.

 

[Jonathan Edwards]

They were specifically claiming to be studying the group of people who have ME.

The PACE trial claimed to be on chronic fatigue syndrome and their definition did not include PEM. That is scientifically legitimate. The term ME had always been ambiguous and confusing, with a variety of interpretations. I think the present term ME/CFS is very useful but I am increasingly doubtful that ME was a very useful medical term. It tended to assume a causal homogeneity that we don't have support for.

 

[Adrian]

So in a very general sense, I can see the point that ME is a "subset" of the condition of "chronic fatigue." And presumably, there are many different conditions that have chronic fatigue as a feature. But PACE was not claiming to be studying exercise in all such conditions. They were specifically claiming to be studying the group of people who have ME.

I can imagine one reason they are objecting to NICE requiring PEM is because they know NICE's new criteria scientifically invalidates the way they've been studying ME all along...

Edited to add
Of course, this on top of all the other problems with study design and conduct

I would describe chronic fatigue as a symptom so ME isn't a subset it is something where chronic fatigue is one of the symptoms. And chronic fatigue is a common symptom with a number of diseases. Perhaps an issue comes in that as a symptom 'chronic fatigue' is not well defined or perhaps has subclasses in terms of what exactly is experienced. If this is the case then it may be meaningless as a symptom and more work needs to be done on definition.

The PACE trial claimed to be on chronic fatigue syndrome and their definition did not include PEM. That is scientifically legitimate.

It feels irrelevant for PACE since the big issues are around bad methodology rather than who it applied to.

It is legit as long as clearly defined and that the group referred to is very clear. One of the parts of the definition is often exclusions and in the PACE trial I was never clear how these were done - I seem to remember lots of patients being rejected for the trial and lack of clarity on why. For real legitimacy there needs to be a clear definition both both of selection criteria and how they were operationalized. There is of course another big issue here if you take a large group of people with a few symptoms then try X on them then that group could be very diverse. Thus some may gain, some may loose and an aggregate could be hard to interpret. But from what I remember of looking at the PACE date results were remarkable consistent - I had assumed there would be multi-modal distributions and I don't recall finding any. But that, perhaps, is due to the dominant feature of the trial being psychological priming to change questionnaire scores.

 

[Adrian]

The whole point of his tweets is to say AfME had ideological bias, and despite designing part of the PACE trial, after the results were in, they distanced themselves.

Apart from that not necessarily being what happened, what’s the point of telling people that AfME is “ ideologically biased” citing a widely-derided trial, whose very name is shorthand for “here’s what NOT to do”.

It's a bit like criticising the health and safety record at the bullet factory, after someone has bought a gun and gone on a shooting rampage I.e. a fairly irrelevant side issue.

There is an issue in seeing how an organization works without seeing the individuals. As I understand it the former CEO (or perhaps someone with a slightly different position) was supporting doing PACE. Once he (and his preferences when) support for PACE was not there and the team could see the obvious flaws.
.

 

[Adrian]

I wonder if the campaign to stop PACE originally was based on the poor trial design?
My memory is that the failings trial design did not really surface until 2015. Prior to that most of the complaints were about the diagnostic criteria for entry, which may not have suited PWME/CFS but were scientifically legitimate. Even the US committee that advised against GET in ?2015 seemed to base its decision on that.

I think @Tom Kindlon was raising issues very early on about trial design in terms of use of subjective measures and commented on the protocol etc and on other work.

 

[Jonathan Edwards]

I would describe chronic fatigue as a symptom so ME isn't a subset it is something where chronic fatigue is one of the symptoms.

But by Boolean algebra, if ME is the intersection of several symptoms, one of which is fatigue, then it will describe a subset of the set of people with fatigue.

 

[Jonathan Edwards]

I think @Tom Kindlon was raising issues very early on about trial design in terms of use of subjective measures and commented on the protocol etc and on other work.

I am sure that is right. Tom seems to be the person who saw the scientific problems long before anyone else. My thought was that this may not have fed through much to messages and campaigns run by organisations like MEA. The message I see from then is that PACE was not appropriate because it was known that ME was not psychological and that people found exertion intolerable.