New Video - The PACE trial - Part 1: Moving the goalposts

I think you meant Adam...

 

It must depend surely on what importance each component has to the overall result, and how each contributes to it. The authors clearly presume there is a simple addition involved, else they could not be making such a statement. But given how important every component is, it feels to me that cannot be so, because if any one support falls then it all falls. Feels as if each component must have some multiplicative factor, possibly as well as an additive component. So if any one component drops to zero, then the whole (multiplicative aspect of the) result would drop to zero. Logically it's more like the contributions are in series, like a chain - any one link breaking and it is all broken.

In effect there must be a transfer function of some kind, where the individual components are the inputs to it, and outcome the output. It cannot, surely, be a simple additive transfer function, and the individual components not necessarily evenly weighted. Surely that transfer function should be pre-specified before a trial starts, or most certainly before awareness of outcomes is possible.

And any peer review should look at some "what if" test cases, to see if they make sense. The idea that someone could recover without any improvement in physical function (let alone if they get worse) should shout volumes that the transfer function has to be flawed.

I'm sure all you maths buffs will have a much better handle on this.

 

Psychology seems to assume that data is linear.

 

I always reckon I'm lucky to get the first letter right. Please accept my apologies Annie, and my congratulations to Alice.

(and no, it isn't the ME: I was this bad when I was teaching!)

All four criteria had to be satisfied. In set theory we would call this the intersection of the four groups: those members that satisfy all four criteria. As you correctly suggest, it is more akin to multiplication, but a better visualization is a sifting process. If all four sieves have large holes, a lot of stuff gets through, but it only needs one to be a fine sieve for very little to pass.

Another way of thinking about it is to think that each criteria subtracts from the pool of patients. So if we tighten up the first criteria and bring the percentage way down below 22%, further subtractions are pretty irrelevant.

It is obvious that originally they would have expected lots of people to pass each of the tests.

Perhaps I should go back to the data and see what effect this one basic change would have. I feel a video coming on. A short one. Does anyone have a link to one of the "rebuttals" that suggests that flaws with this one measure are not too important because there are three more criteria?

 

Thanks Graham, that's a great explanation.

As I was reading about sieves and patients my brain sort of cut out the middleman, and I found myself visualising it as patients falling through the sieves, with each sieve being sensitive to one of the criteria. Recovered patients have to make it through all four sieves into the "recovery bin" below. If patients are too 'big' (i.e. too unrecovered) on any criteria, they get stuck in that sieve. With PACE it's as if they found too many people were getting stuck in the Physical Function sieve, so they ... increased the size of the holes in that sieve, allowing them to drop through again.

Something seems to be full of holes.

 

Don't forget that they actually loosened all four sieves. It might be that they found that all four of them were too restrictive.

I think I would like to play around with the data and compare the severity of each sieve with each other, and with the original settings.

But the important argument is simply that changing the sf-36 target to 60 cannot be justified: it needs to be corrected. If so, the claim of 22% recovery is demolished. End of discussion. How strict the other sieves are is more of an intellectual but pointless challenge – it sounds right up my street.

I'm trying to think of a suitable analogy. How about advertising a clerical job? Applicants must have at least a grade C in Maths, at least a grade B in English, and to have had at least two years experience in a similar position? (Yes, I know the grading system has changed in the UK, but I'm old school?)

Dropping the requirement in Maths to a grade E or more makes that much easier to achieve, but the other two are still pretty restrictive, so the potential pool of those who pass all three won't increase that much.

If you want to increase the total pool of applicants, you have to make all three criteria much easier to attain. To my mind, that is what they did, asking for at least a grade E in Maths, and in English, and at least a month's experience in any office work. That is when changing the Maths requirement back to a C has such a big effect.

This might be a way of demonstrating that the other criteria were similarly weak.

I need a better analogy. Any thoughts?

 

Graham. I wish you had been my maths teacher. You have a wonderful talent for explaining things simply and coherently.

 

Table 1(b) of the recovery paper (Psychological Medicine (2013), 43, 2227–2235) shows that the transfer function used was the AND function - CFQ in range AND SF36 in range AND Oxford not met AND CGI 1 or 2. So a sieve, like Graham said.

 

Bearing in mind the grades were all self-reported anyway, and therefore liable to be pretty slack, it may be that as awareness began to dawn of how the outcomes were likely to pan out, maybe one in particular proved more 'stubborn'; outcomes perhaps more inclined toward the objective criterion it was purporting to represent - PF. (I don't know this! Just a thought). And if so then maybe spotted that messing around with the pass mark for this alone would make a big overall difference (except they had to mess with it to the point of absurdity). And then just fiddled a bit with the other criteria as well.

I refuse to believe they did not have strong awareness of how results were likely to pan out, and might therefore have undertaken a reverse engineering process to fix their issues.

 

Perfect! Any one dropping out drops the whole thing to Fail.

But wasn't something tweaked in this area, compared to the original? I seem to recall the way things were AND'ed introduced some slackness, compared to the original definition? It was subtle but there. @dave30th maybe reported it? Maybe in the reanalysis. Not got time to go hunting for it at the moment.

ETA: But of course the AND'ing is applied after the individual criterion pass/fail have been assessed, and effectively then become binary values in order to then AND them.

ETA2: But the AND'ing shows the invalidity of MS and Co. claiming that PF was only one of the criteria that had to be met, and therefore not a big deal to mess with. They all had to be met if the AND function is to be satisfied. But I still have a niggle there was something else ... ?

ETA3: (Sorry). When I say above "the way things were AND'ed", by that I mean that I think one of the AND clauses had a new OR introduced within it, with some weaker criteria on one side of the OR.

 

Thank you, but you do realize that after a year of being in my class your brain would have been warped forever.

 

Would it really have taken that long?

 

Does anyone have any suggestions of a totally different "medical" study where three or four criteria were used to determine recovery, or some similar concept?

 

Do you mean a real one that was previously done? Or a realistic hypothetical one?

 

That's pretty much the loophole. PACE wasn't medical research, does not even pretend to other than as its use in trying to erase a medical entity. It's graded on a curve even by the standards of psychological research but somehow also benefits from all the credibility of being medically valid because reasons, mostly greed with parts delusion of grandeur.

No one who promotes PACE thinks of it as medically relevant. It's psychological research showing "CFS" is a psychological problem related to attitude and behavior and the medical credibility is simply to humor us and give it more weight than it deserves because ME remains a valid medical entity. Everyone fluffing it is fully aware of this and simply plays along because it works and it validates their beliefs.

But since the lines between psychology and medicine are being erased it makes a lot of people quite happy in their quest to psychologize medicine. We're just stuck in the crossfire of an ideological transformation of medicine. One that will fail massively once the numbers and impact assessment are in and people realize it's financially (and morally) disastrous, but it will take some time for that realization to unfold, and more than a few fraudulent cherry-picked attempts at detailing how wonderful the emperor's butt flaps are.

 

A real one. I'd like to compare something that genuinely uses a number of criteria to measure success.

You might be interested to hear that I have been looking at the GET group, and eliminated all those patients without the full "recovery" criteria. That left 134 patients. The doctors diagnosed that 99 of them still had CFS at the end of the trial, but 39 of these diagnoses were overruled by the scores on the sf-36 or the CFQ (mostly the sf-36). On the one hand it is weird to think that the doctors themselves would have diagnosed a recovery rate of 35 out of 134 (27%), but even more weird to realize that the PACE researchers were happy that their two questionnaires were able to refute 40% of the doctors' diagnoses of CFS – and these were supposed to be specialists.

 

Do we know how the doctors made their assessments? Ongoing monitoring? Consultation? I'm guessing probably still based on subjective self-reporting?

 

Just been looking in some of the PACE literature, and although I realise this statement had been made, and was commented on somewhere not so long back, I'd not appreciated (or maybe I just forgot) it was actually stated in the 2013 Recovery paper:

Massively more likely related to the finding that the above quote is BS.

 

Further to the issue of 13% recovered at baseline on SF-36 PF with revised threshold, and MS and Co. supposedly rebutting this by stating it was only one of multiple recovery criteria.

Looking back at the 2007 PACE protocol:

Or in Boolean form, Recovery achieved if:

CFQ AND SF-36 PF AND CGI AND (Oxford OR CDC OR London) = True​

Basically, if any one of the AND'ed clauses is not true, then there is no recovery.

So when MS claims that lowering the SF-36 PF pass threshold from 85 to 60 was not a big deal, because it was only one of multiple criteria, that is itself rebutted by their own 2007 protocol! Of course it was a big deal - every single criterion was a big deal because they stated so in the protocol! Else they would not have said "all four"!

 

Which is why correcting just the sf-36 recovery target has such a big effect.