What do the data show?
The authors appear to have attentively followed the original PACE protocol and presented their own analyses in a reasonably clear and transparent way. Sensible assumptions were made about the missing data and the data was analyzed in different ways to show that the results are robust. This reanalysis shows that if the authors of the initial study had followed their own protocol, they would have reached very different conclusions. The authors of this paper make appropriate use of their findings to raise concerns about the robustness of previously formulated claims on the efficacy of CBT and GET based on the PACE trial.
Additional unplanned analyses were performed on long-term data collected at least 2 years after treatment. These analyses were repeated after exclusion of patients who received additional treatment after the trial, suggesting that conclusions about the absence of long-term effects are reasonably robust and not the result of post-test interventions.
The Limits of this new analysis
There are points in the original PACE protocol that were either inconsistent or open to interpretation. The authors of this new paper seem to have chosen the most extreme analysis to put forward their point of view. For example by making adjustments for 6 comparisons where 3 or 5 comparisons are also described, and by focusing only on 52 week data.
The authors rely heavily on p-values and statistical significance thresholds when presenting results. This is a rather outdated approach and there is a lack of information on the extent and precision of treatment effects in most cases. The authors made few attempts to uncover the reasons for the protocol discrepancies in the PACE trial or the point where they were made. The evaluators could have been invited to comment.
No adjustment was made to the reanalysis of the patient's characteristics even if this had been planned in the initial trial. Sufficient data to do this were not available. The inclusion of appropriate demographic variables could improve the accuracy of the results.
Only 52-week data were analyzed yet data were collected at 12, 24 and 52 weeks. An analysis taking into account measurements over time could have led to more accurate results.
The new paper may give the impression that all or almost all of the evidence on CBT and GET comes from the PACE study, as this paper suggests that it seems unlikely that new research based on these treatments will yield more favorable results. In fact, CBT and exercise therapies have been studied in several other studies that have been reviewed in Cochrane reviews. The latest Cochrane Review includes 8 studies other than PACE and offers positive conclusions about some aspects of the effectiveness of exercise therapy.
Experts' opinion on this reevaluation
For Chris Ponting, professor of medical bioinformatics at the University of Edinburgh, the new analysis shows that the clinical trial does not reach a meaningful statistical result.
In 2011, the PACE group interpreted their randomized trial data as meaning that Cognitive Behavioral Therapy (CBT) and Graduated Therapy (GET) can be safely added to specialized medical care to moderately improve the outcomes of CFS.
Since then, Wilshire and her coauthors render a new analysis of PACE trial data. This reanalysis was necessary in part because the research group had revised its analysis from the plan published in its protocol. This revision meant that, in theory, some trial participants could be considered patients, but they were considered to be cured at the end of the trial even if their symptoms did not improve, or even worsened. Wilshire et al. provide proof that the effects of CBT and GET were very modest and not statistically reliable overall if we apply procedures very close to those specified in the originally published protocol. Their analysis also revealed that recovery rates according to the protocol definition were much lower than previously published and that CBT and GET did not lead to recovery.
Importantly, Wilshire et al. provide a plausible explanation even for these modest effects. Specifically, they argue that they are simply explained by the high expectations of participants in CBT and GET that their treatments would be effective (ie, a placebo effect). Expectations are exacerbated, they say, when participants are not blinded by their treatment and are assured that their treatment is effective as was the case for the PACE trial. The lack of significant gains in objective outcome measures, such as fitness, indicates that gains on self-report measures may not be reliable. This could largely explain the modest effects observed after one year and the disappearance of these effects thereafter. The authors also assert that the lack of substantial and long-lasting effects of CBT and GET observed from a trial the size of PACE implies that these therapies are unlikely to be widely effective.
So, according to Ponting, the results have little statistical significance as the moderate positive effect could be compared to placebo, even after a year.
