Caroline Struthers' correspondence and blog on the Cochrane Review: 'Exercise therapy for chronic fatigue syndrome, 2017 and 2019, Larun et al.

[Snow Leopard]

The problem of unblinded trials with subjective end points is not 'just an opinion'. It is the only sensible view and is embedded in all evidence based medicine. With due respect if Tovey's advisors do not share this view they are incompetent. Again with due respect if Tovey is not aware of this he is not qualified to act as an editor in an organisation like Cochrane. He seems to have acted fairly so far but in this instance he appears to be indicating that he has no grasp of reliable evidence. That makes Cochrane something of a basket case maybe.

The problem of subjective end points with no blinding is as basic as it gets. It is the reason we have blinded trials. No trial with this design of a drug would be taken seriously. Moreover the problem is much worse with therapist delivered treatments, not less bad, because of the role playing and manipulation of attitudes that is inherent in the human interaction.

We/you are assuming they have the same ideological basis as us/you. They don't believe that there is an underlying objective disease, they believe that subjective perception of pain/fatigue/whatever is the illness and that subjective reporting of improvements necessarily represents improvement in the illness, rather than reporting biases.

edit- I just noticed that @Barry made more or less the same point a few days ago...

 

[Hoopoe]

I'm pretty sure a skilled motivational speaker could put a room of people into the state of mind of feeling like a billionaire. Does that mean that these people will leave the room richer?

 

[Robert 1973]

We/you are assuming they have the same ideological basis as us/you. They don't believe that there is an underlying objective disease, they believe that subjective perception of pain/fatigue/whatever is the illness and that subjective reporting of improvements necessarily represents improvement in the illness, rather than reporting biases.

But, as @Jonathan Edwards pointed out in his submission to the Scottish Parliament, the value of PACE is that it provides evidence that the BPS theory is wrong:

For these, and other, reasons, these trials generally tell us nothing useful. However, the PACE trial at least may tell us more – that the theory behind the treatment is actually wrong. The theory was that the persistence of disability (measurable as inactivity) in ME is due to unhelpful beliefs about inability to improve. What PACE seems to show is that it may be possible to change patients’ beliefs, or at least to encourage reports of feeling better, but, crucially, the measures of disability did not change [1]. This suggests that even if unhelpful thoughts were present they were not the cause of the disability. Moreover, the long-term follow up data from PACE show no difference from control even in reported wellbeing in the CBT and GET groups.

————

1. It is now documented from trial committee minutes that detailed assessment of activity with actometers was abandoned because a previous study had already suggested no improvement would be seen – which would not have supported the preferred hypothesis.

http://www.parliament.scot/S5_PublicPetitionsCommittee/Submissions 2018/PE1690_F.pdf

 

[Barry]

Compared to voltmeters which have well-defined or straightforward to control errors, questionnaires are subject to a whole host of uncontrollable or difficult to control biases.

And indeed indeterminate biases.

 

[Snow Leopard]

I'm pretty sure a skilled motivational speaker could put a room of people into the state of mind of feeling like a billionaire. Does that mean that these people will leave the room richer?

People generally leave rooms of successful motivational speakers somewhat poorer.

 

[Jonathan Edwards]

We/you are assuming they have the same ideological basis as us/you. They don't believe that there is an underlying objective disease, they believe that subjective perception of pain/fatigue/whatever is the illness and that subjective reporting of improvements necessarily represents improvement in the illness, rather than reporting biases.

edit- I just noticed that @Barry made more or less the same point a few days ago...

Those beliefs are not really ideologically based. They are based in naive incompetence. As we have discussed before there is a contradiction in the premises. If subjective reports at the beginning do not actually reflect the persons activity potential then there is no reason to think they do at the end.

The issue is about the real life intrusion of human nature into clinical trials - which long ago was realised to be such that blinding is needed unless you have cast iron objective end points.

In other words the is no defence for Tovey in the fact that these people cannot even get their ideas coherent!

 

[rvallee]

Exactly. What you are recording is the output of an instrument, this cannot be assumed to be the same as the underlying phenomena.

How a patient answers on a symptom questionnaire is not the same as the underlying experience of that symptom. Compared to voltmeters which have well-defined or straightforward to control errors, questionnaires are subject to a whole host of uncontrollable or difficult to control biases.

Which is even a problem they noted themselves in the few trials that had both subjective and objective endpoints. They can't explain why patients self-report less fatigue but cannot do more because the truth is devastating to their case and so they just bury it, pretend it's an anomaly when in fact it is falsification.

It is often reported that fatigue is improved in their trials. It is not, it is merely the self-reporting of the perception following conversion therapy aimed at convincing the participants to believe themselves as not as sick as they are, further made irrelevant by reducing a complex illness down to a single hard-to-gauge symptom that is extremely common and whose meaning varies wildly from person to person.

A relevant comparison would be a trial for treatment to reduce excessive hunger where participants rate their hunger as less important but they objectively eat just as much. It's pure junk.

 

[Dolphin]

Which is even a problem they noted themselves in the few trials that had both subjective and objective endpoints. They can't explain why patients self-report less fatigue but cannot do more because the truth is devastating to their case and so they just bury it, pretend it's an anomaly when in fact it is falsification.

It is often reported that fatigue is improved in their trials. It is not, it is merely the self-reporting of the perception following conversion therapy aimed at convincing the participants to believe themselves as not as sick as they are, further made irrelevant by reducing a complex illness down to a single hard-to-gauge symptom that is extremely common and whose meaning varies wildly from person to person.

A relevant comparison would be a trial for treatment to reduce excessive hunger where participants rate their hunger as less important but they objectively eat just as much. It's pure junk.

This paper argues that there is a sort of hierarchy of subjectiveness of questionnaires with fatigue questionnaires being more subjective than physical functioning questionnaires which tend to mention specific activities.

https://journals.sagepub.com/doi/full/10.1177/1359105317707531

PACE-GATE: An alternative view on a study with a poor trial protocol


Bart Stouten
First Published May 12, 2017 Research Article
https://doi.org/10.1177/1359105317707531
Article information

Abstract
The controversies surrounding the effectiveness of cognitive behavioural therapy and graded exercise therapy for chronic fatigue syndrome are explained using Cohen’s d effect sizes rather than arbitrary thresholds for ‘success’. This article shows that the treatment effects vanish when switching to objective outcomes. The preference for subjective outcomes by the PACE trial team leads to false hope. This article provides a more realistic view, which will help patients and their doctors to evaluate the pros and cons.

Keywords chronic fatigue syndrome, cognitive behaviour therapy, effectiveness, graded exercise therapy, randomized controlled trial, treatment

 

[Barry]

This paper argues that there is a sort of hierarchy of subjectiveness of questionnaires with fatigue questionnaires being more subjective than physical functioning questionnaires which tend to mention specific activities.
View attachment 5553

Also shows that the very word "fatigue' is the most inappropriate description for the illness, and also no doubt why it is the most preferred naming by the BPS people.

 

[Sean]

And indeed indeterminate biases.

This is an important point. Some biases are known and quite specific. Some a lot less so.

In a voltmeter the biases are well understood and can be calculated – and hence compensated for – to a high degree of precision.

Self-report measures of fatigue? Not so much.

 

[TiredSam]

People generally leave rooms of successful motivational speakers somewhat poorer.

But the speaker leaves considerably richer, and will assume that everyone else does too.

 

[Mithriel]

I can't remember who said it, but it makes a very important point. The PACE authors insist that changes to the recovery criteria were OK because they were made before they saw any data. However, if you are measuring the height of boys in a school redefining tall as 5ft 8ins instead of 6ft you do not have to have seen any of the results to know that there will now be more boys in the category.

Similarly, if you make the recovery point lower on the scale more people will reach it. If most people will already recovered, you make a good result even better or, as happened, you make a successful trial from a failure.

As is usual with this group they say things that sound impressive but are meaningless.

 

[Esther12]

I can't remember who said it, but it makes a very important point. The PACE authors insist that changes to the recovery criteria were OK because they were made before they saw any data.

I don't think that they have clearly said this - although they've implied it.

This is what they said in their recovery paper:

We changed three of the thresholds for measuring recovery from our original protocol (White et al. 2007) before the analysis, as explained below.

The domains chosen and the criteria for recovery on each were defined before we undertook the analysis.

We changed some of the thresholds for measuring recovery from those of the original protocols (White et al. 2007); we made the changes before analysis and to more accurately reflect recovery.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3776285/

'Before the analysis' is different to 'before data was unblinded'.

In 2011(after the Lancet paper) patient charities made an FOI request for the results of outcomes specified in the trial protocol: https://www.meassociation.org.uk/2011/05/6171/

For the results for the recovery criteria, QMUL stated that these were "exempt under section 22 of the FOIA 2000 on the basis that the authors intend to publish these in the future."

https://www.meassociation.org.uk/wp-content/uploads/2011/06/FOI+from+Queen+Mary.pdf

If that were true, it would mean that the decision to not publish these results came after they had released their Lancet paper. There is no mention of approval for the recovery criteria they used in the 2013 'recovery' paper in the trial's TSC or TMG minutes.

 

[Barry]

'Before the analysis' is different to 'before data was unblinded'.

Quite. And given the trial had no blinding whatsoever, the researchers will have had strong indications how things were panning out quite early on. I mean, how likely is it really that they were not discussing along the way how things were going - their teams were collecting the data and were fully aware of how that information would have been trending. The data was unblinded before it even went into the database, it was already in the researchers' heads. I cannot believe they did not have good insights into how it might have been unravelling for them, and what would likely 'fix' their problem.

 

[Esther12]

I just thought I'd see how the protocol changes were described on the PACE FAQ: (there now seems to only be an annoying link for this: https://www.qmul.ac.uk/wolfson/research-projects/current-projects/projects/#faq )

For their primary outcomes they say the change was : "before any data was analysed".

For the recovery outcome they say the change was: "before the analysis".

I think that this is another example of White trying to be clever with his language.

Fuller quotes:

The two primary outcomes for the trial in the original trial protocol were the SF-36 physical function sub-scale and the Chalder fatigue questionnaire. These were not changed.

However, as a trial like PACE takes many years to complete, discussion about how best to analyse the findings continued after it began, but before any data was analysed. These discussions led to changes in how we analysed the two primary outcomes (fatigue and physical function). The two independent oversight committees approved the changes. The detailed analysis plan, including these changes, was published, and these changes and the reasons for them were also described in the main paper.

We also changed the thresholds for judging recovery on the two primary outcomes (fatigue and physical function) from the mean (average) for the population to the population mean plus a standard deviation from the mean, to better reflect the levels of fatigue and physical function found in the general population. This was informed by a recently published population study.

These changes were all made before the analysis occurred (i.e. they were pre-specified) and were fully described and explained in the paper itself. Furthermore, in the relevant paper we discussed the difficulties in measuring recovery, and showed how other ways of defining recovery could produce different results. The bottom line was that, however we defined recovery, significantly more patients had recovered after receiving CBT and GET than after other treatments.

 

[Dolphin]

The normal range analysis, which ended up being part of the revised recovery criteria, was described as post hoc in the 2011 Lancet paper.

 

[Dolphin]

Note that the recovery criteria are not part of the Larun et al Cochrane GET review.

 

[Esther12]

Note that the recovery criteria are not part of the Larun et al Cochrane GET review.

That's true, but these protocol deviations are still relevent to the Cochrane assessment of risk of bias, as Courtney pointed out:

As well as those outlined above, various other important outcomes in the trial were changed dramatically, such as the recovery analysis, which was reported in a separate publication [12]. Also, the pre‐defined 'clinically important difference' was dropped, and was replaced with a 'clinically useful difference' which had an entirely different definition. There were too many deviations from the protocol in the final analyses to list them all in detail here.

The Cochrane guidelines (8.14.2) state: "The assessment of risk of bias due to selective reporting of outcomes should be made for the study as a whole, rather than for each outcome. Although it may be clear for a particular study that some specific outcomes are subject to selective reporting while others are not, we recommend the study‐level approach because it is not practical to list all fully reported outcomes in the ‘Risk of bias’ table."

https://www.cochranelibrary.com/cds....pub7/detailed-comment/en?messageId=160462675

 

[MEMarge]

I have vague memories of Wesseley saying something incriminating, probably in one of his "Standing up for science/poor little me, all those militant Me activists are out to get me" speeches. He was referring to saying something like "well we had to change the criteria otherwise no-one would have got better" and then referred to this with a mixed metaphor: something like "I seem to have let the hare out of the bag...."

Does any of this resonate with anyone or am I just having weird dreams. I think @Nathalie Wright might have been at the relevant talk.

 

[large donner]

I have vague memories of Wesseley saying something incriminating, probably in one of his "Standing up for science/poor little me, all those militant Me activists are out to get me" speeches. He was referring to saying something like "well we had to change the criteria otherwise no-one would have got better" and then referred to this with a mixed metaphor: something like "I seem to have let the hare out of the bag...."

Does any of this resonate with anyone or am I just having weird dreams. I think @Nathalie Wright might have been at the relevant talk.

Wessely:
In essence though they decided they were using a overly harsh set of criteria that didn’t match what most people would consider recovery and were incongruent with previous work so they changed their minds – before a single piece of data had been looked at of course. Nothing at all wrong in that- happens in vast numbers of trials. The problem arises, as studies have shown, when these chnaged are not properly reported. PACE reported them properly. And indeed I happen to think the changes were right – the criteria they settled on gave results much more congruent with previous studies and indeed routine outcome measure studies of which there are many.

https://www.statnews.com/2016/09/21/chronic-fatigue-syndrome-pace-trial/comment-page-6/#comments