A new consensus? - ME/CFS skeptic blog

I heard the other day that doctors who want to be accepted into a college to train to be a specialist are often required to undertake some research. So, some medical research is undertaken by people who may or may not be very interested in research, and may or may not care great deal about finding a useful answer. They mostly need the study to fit into the 6 months that they have for the research and for it to tick the 'has done some research' box for the college. So, that means that there is often little interest in time-consuming consultation with relevant consumers.

That's all very nice for the doctor, but it may be bad luck if you have the condition that the, for example, aspiring orthopaedic surgeon just banged out a C-grade study on.

For sure, on the bad research not being confined to ME/CFS. I've seen some horrendous studies on schizophrenia in particular; I think people with schizophrenia have an even harder time than we do with stigma, and it's such a terribly difficult disease for the person and their supporters, with the result that there may not be much scrutiny.

For sure, it's something I've been thinking about lately. Any assessment of CBT for ME/CFS needs to look very closely at what the CBT was aiming to do. The phrase used recently, I think by the Kuut?, 'activating CBT' could be a useful term, that is CBT that aims to make the patient more active, as opposed to coping better with the losses and frustration, or better able to reduce activity to a level that doesn't cause PEM.

 

I've recently done a course on systematic reviews that had a short section on GRADE. Now, I may have things wrong, but I did not find GRADE to be a very bad system. In a systematic review at least, GRADE is applied to the collected evidence for an outcome, not to individual studies.

First off, you decide if a study is so egregiously flawed that the results must be rubbish e.g. it looked at a population that is mostly different to your target population, or there is overt evidence of fraud. In that case, you don't include it in the systematic review at all. I don't believe unblinded studies with only subjective outcomes fit in that category, because if all the patients were suddenly rising up from their beds and reporting 'cured' levels of SF-36, and 6 months later 90% were still in that 'healthy' range, then the study could well be indicating that the treatment is useful. If there were those sort of results in a decent sized unblinded study of people with well-defined ME/CFS, I'd be thinking hard about trying the treatment, even if the outcomes were subjective. So, I think the GET and CBT studies mostly qualify to be in reviews.

Then, my understanding is that an outcome from RCTs starts at 'high' certainty and is assessed against 5 criteria. Risk of bias is one of the five. For each criteria, you can downgrade the particular RCT outcome by one or two. So, I think it's clear that the CBT and GET outcomes drop by two categories to low certainty due to a high risk of bias. The other categories almost certainty add some further downgrades, for indirectness, for inconsistency (different results from different studies), for imprecision and for publication bias.
On indirectness, most BPS studies would be downgraded for ME/CFS outcomes due to poor selection criteria.
Outcomes could be downgraded for inconsistency if the few studies with objective criteria did not find an objective increase in activity, while the subjective reporting did, for example.
For imprecision, if there is a risk that important harms were not considered (as would be the case in ME/CFS studies that did not track activity for a long enough period when there are hints of substantial harm coming from non-trial sources), then the outcome would be downgraded.
Publication bias is where there is some evidence that studies that don't show a favourable effect of the intervention are not getting published. There are several ways to work that out, including looking at trial registers.

And then there's more. Having got the outcome, with it's 'Very Low' certainty rating, you can make some comment about the size of the benefit. Even if all the studies are consistently reporting a benefit, if the benefit is small, within the range that would often be seen with ineffective but hyped treatments in unblinded situations, then you can report that there is a very low certainty that the reported small benefit is real.

As with many tools, it's not so much the quality of the tool, as the quality of the person using it. Yes, to a certain extent, the GRADE outcome is subject to the prejudices of the person using it. What GRADE does is provide a framework for analysis and a requirement that assumptions that have been made are reported and made explicit. I think it's ok.

Here's a link to the handbook.
(And edited to add publication bias, which I had forgotten about.)

 

Just on that finding that pacing has no effect compared to usual care:
I agree that most people with ME/CFS learn fairly swiftly to reduce their activity in order to avoid post-exertional malaise. I think that within a year most people won't be trying to do the same amount of activity that they were doing before they became ill. They will be consciously reducing what they do, perhaps cutting out their weeknight social football game, or only having two showers a week, or changing to part-time work, or moving home to be looked after by Mum and Dad. So, I contend that maybe a year into the illness, most people have worked out how to pace in some way. Imperfectly, of course, but well enough that they are surviving.

That means that they are already pacing. Trying to then apply a treatment of "pacing" to people who may be years into their illness, probably often offered by therapists who really don't know what they are talking about, and who have got the message that pacing is some mistaken belief that patients have and/or that pacing should actually be GET-lite, - well - it's not surprising that the treatment shows no benefit.

 

Thanks for this, @ME/CFS Skeptic.

One point:

Don't know about the Norwegian trial, but in PACE the version they used was not pacing as generally understood and used by the patient community. It isn't a useful result in either direction.

I think we can. Any further indulgence in this kind of research is just a waste of funds and patients' lives. I think the question has been answered. It's a bust. Time to move on.

...then it should also be demonstrable via blinded and/or objective measures.

I find the whole notion that self-reported improvement in general activity capacity can exist without a clear result on more robust measures deeply problematic. All that leads to is the circularity of teaching to the test.

The whole history of ME/CFS makes it clear why that approach is just not safe.

 

I've moved my post discussing this to a new thread:
Research on pacing as treatment for ME/CFS. Discussion of how to do it.

 

Sure, you wouldn't want to stop collecting evidence if you just have unblinded trials with subjective evidence. I'm just making the point that if, for example you have multiple trials like that, with that being the only flaw, and they all were showing that the majority of people were reporting lasting and substantial benefits, then they would constitute evidence. It would be low or very low quality evidence, probably not strong enough evidence to support using the treatment outside of trials, but you wouldn't want to not acknowledge the finding. The question is 'is the flaw major enough to exclude the trials from a systematic review' - and I don't think it is, because we can later make some assessment about the reported benefits (or lack of them).

While @ME/CFS Skeptic's writing is as clear and brilliant as usual, I disagree that this is an accurate representation of how GRADE works. If assessors decide that an evaluation of an outcome in a trial really is fatally flawed, then the trial is not should not be included in the assessment of the evidence for the outcome. But, an unblinded trial with subjective outcomes is not necessarily fatally flawed. It can tell us something. It might tell us that there is no effect, even with all the biases stacked in favour of the intervention. It might tell us that an effect was reported, but it probably falls within the range of a placebo effect. Or it might tell us that there was an amazing result that exceeds the likely placebo response, and that this intervention shows real promise and should be investigated further, that it is worthwhile doing more expensive and difficult trials that are blinded and/or have objective outcomes.

I don't think GRADE is particularly rigid. The handbook itself says that it is just a transparent system for making assumptions explicit. GRADE does not make it impossible to do useful, rigorous systematic reviews. In fact, I think it is helpful.

 

As Simon says, I think it's still an important principle that there is not a reliance only on subjective outcomes in unblinded studies. When pain is bad, people can't keep functioning as normal, not on any regular basis. Secondary objective outcomes could include:
time spent being sedentary or lying down;
medication taken to control pain;
alterations in gait
results of cognitive tests

These things would obviously need to be tracked over a sufficiently long period of time. If it wasn't possible to solely rely on patient reported outcomes, I bet reasonable objective outcomes would be found.

 

Subjective self-report measures are particularly problematic when the treatment is aimed at changing cognition/perception. It is just ripe for all sorts of bias and confounding. Worst case scenario, indeed. It actually makes blinding, or additional less subjective measures more important, I think.

Need to distinguish between two different levels of causal knowledge: Casual relationships, and causal mechanisms.

It is possible to demonstrate a casual relationship (A causes B, at least probabilistically), without knowing the process by which that occurs.

Ideally, of course, you want to know both. But you have to be able to demonstrate at least the first to claim useful science-based knowledge.

Which is the problem with using unblinded subjective measure on their own. They don't allow us to say even that much. Causation has not been revealed and clarified. So we don't know if any effect is due to the treatment, or response bias, or...? All we can say is there is an effect, but not know its causal nature. Nor, hence, if it is of actual practical benefit.

Which leads to this sort of guff.