A general thread on the PACE trial!

Lucibee said:
An "improvement" of 2 points for fatigue could also indicate a worsening, depending on how the CFQ is interpreted (which I've discussed elsewhere).
Click to expand...

@Lucibee Table 3 in the supplementary material to Collin and Crawley’s 2017 paper demonstrates exactly the effect you suggest, where a decrease of 2 points on the Chalder Fatigue Questionnaire, used in PACE as indicating improvement in fatigue, can occur when the patient has deteriorated.

The patients who reported themselves “very much worse” (n=11) at 1 year follow-up after specialist treatment had a mean change in their Chalder Fatigue Questionnaire score of -2.27, with 95% CI of -4.77, 0.22.

The patients who reported themselves “much worse” (n=23) at 1 year follow-up after specialist treatment had a mean change in their CFQ score of -0.26, but the 95% CI was -2.81, 2.29.

Similarly, the 95% CI for those who reported themselves “a little worse” also includes -2:

The patients who reported themselves “a little worse” (n=51) at 1 year follow-up after specialist treatment had a mean change in their Chalder Fatigue Questionnaire score of -1.24, with 95% CI of -3.08, 0.61.

The patients who reported themselves “no change” (n=65) at 1 year follow-up after specialist treatment had a mean change in their CFQ score of -2.28, with 95% CI of -3.59, -0.96.

It is only when patients reported themselves better that we leave the -2 realm, with mean changes in their CFQ scores of:


A little better (n=157) -5.90

Much better (n=92) -11.0

Very much better (n=25) -14.9


I can’t find a way of linking to the supplementary material or copying the relevant row of the table – but it’s well worth a look.
The Collin and Crawley 2017 paper is here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5513420/

Please do check the original source before quoting these figures or interpretation as I have brain fog!

Someone with number skills could look at the FOI dataset from PACE and create a table showing how the PACE data compare to this. Mean change in Chalder Fatigue scores at 52 weeks according to patient-rated Clinical Global Impression scores. Any takers? I think we'd expect bigger changes, but the pattern/trend seems likely to be in the PACE data too.
 
Evergreen said:
Someone with number skills could look at the FOI dataset from PACE and create a table showing how the PACE data compare to this. Mean change in Chalder Fatigue scores at 52 weeks according to patient-rated Clinical Global Impression scores. Any takers? I think we'd expect bigger changes, but the pattern/trend seems likely to be in the PACE data too.
Click to expand...

I still don't think there is any value in looking at "change in CFQ score" because CFQ already measures change. But if you take what they've done, then their definition of improvement (decrease of 2 pts or more) is well and truly bust for the reasons you outline.

Because CFQ is such a useless measure, it is very difficult to draw any solid conclusions without knowing exactly how each participant interpreted it and used it. And of course, the investigators will have had their own interpretation, which will be different again.
 
Lucibee said:
Because CFQ is such a useless measure, it is very difficult to draw any solid conclusions without knowing exactly how each participant interpreted it and used it. And of course, the investigators will have had their own interpretation, which will be different again.
Click to expand...
Basically like presuming to build a solid foundation with jelly.
 
Lucibee said:
I still don't think there is any value in looking at "change in CFQ score" because CFQ already measures change. But if you take what they've done, then their definition of improvement (decrease of 2 pts or more) is well and truly bust for the reasons you outline.

Because CFQ is such a useless measure, it is very difficult to draw any solid conclusions without knowing exactly how each participant interpreted it and used it. And of course, the investigators will have had their own interpretation, which will be different again.
Click to expand...

I agree, and I've read your blogs on the CFQ. I think it's nice to have NHS data to demonstrate what happens when you try to use a tool that can't measure change to measure change. There are some who only get convinced when they see numbers, no matter how strong or incontrovertible the argument. I think most who read this literature never actually look at the scale itself, and just blindly accept it as an accurate measure of fatigue.

And as @Barry says, we're then building on jelly. And what we're building is a house of cards.
 
Thanks for taking the time to do that, @Lucibee .

Lucibee said:
Not sure it tells us very much
Click to expand...

I think that's 'cos it's wobbly, like all good jelly!

More seriously, I think it tells us a lot. If we assume that there is some worth in the Clinical Global Impression scale, then we'd expect a good outcome measure to have some semblance of a predictable, sensible relationship with it. We'd expect the CFQ scores to go down when the person says they improve on the CGI and go up when they say they get worse on the CGI. We'd expect some kind of a diagonal line in your chart above, instead of a splodge.

It seems that the CFQ scale cannot detect negative change in fatigue (meaning increased fatigue). It's skewed towards detecting, or creating an artefact of, positive change in fatigue (meaning reduction in fatigue). So if the desired result from a study is reduction in fatigue scores, and no harm aka increase in fatigue scores, then the CFQ is the way to go. You can't lose.

A change of -2 on the CFQ is clinically meaningless in the PACE data.
 
Last week, I wrote to my ex-boss (Richard Horton) about my concerns regarding The PACE trial, and in particular, the way the review process was handled.

Over the weekend, I got a response. They won't budge on this. As far as The Lancet is concerned, PACE trial is "science". So I suspect that any criticism of it is therefore treated as anti-science.

I don't know where to go with this now. But I can't stop. If I wasn't activated before, I am now.
 
I wonder what this guy would make of PACE:
https://en.wikipedia.org/wiki/John_Bohannon

"
Who's Afraid of Peer Review?
Main article: Who's Afraid of Peer Review?
In September 2013, he submitted a fake scientific article to a large number of fee-charging open-access publishers,[18] revealing that less than 40% were living up to their promise of rigorously peer-reviewing what is published.[19] This approach was criticized by some commentators as well as by some publishers of fee charging journals, who complained that his sting only targeted one type of open-access journal and no subscription-based journals, damaging the reputation of the open access movement.[20][21][22][23]

Misleading chocolate study


The study. Click for bigger version and original PDF file.
Publishing under the name Johannes Bohannon, he produced a deliberately bad study to see how the media would pick up their findings. He worked with a film-maker Peter Onneken who was making a film about junk science in the diet industry with fad diets becoming headline news despite terrible study design and almost no evidence.[24]"
 
Lucibee said:
Last week, I wrote to my ex-boss (Richard Horton) about my concerns regarding The PACE trial, and in particular, the way the review process was handled.

Over the weekend, I got a response. They won't budge on this. As far as The Lancet is concerned, PACE trial is "science". So I suspect that any criticism of it is therefore treated as anti-science.

I don't know where to go with this now. But I can't stop. If I wasn't activated before, I am now.
Click to expand...

Thank you for trying @Lucibee. I hope you will keep copies of that correspondence. Are you able to make it public here?

I have no idea where you can go next with this. Does the Lancet have an oversight board or owners who can be approached about their editor's refusal to consider valid scientific criticism of the papers he publishes?
 
Trish said:
Does the Lancet have an oversight board or owners who can be approached about their editor's refusal to consider valid scientific criticism of the papers he publishes?
Click to expand...

Others have already tried this route and got nowhere, I'm afraid. They simply don't accept it as valid scientific criticism. Once they've put you in the flat-earther, anti-vaxx, climate-denial, alien-conspiracy-theory, anti-science camp, it doesn't matter what you say or how valid it is. It's all treated as nonsense. Reputation is everything is science. If you have no reputation, nothing you say will be taken seriously.

Which is a shame. The best scientists I've encountered are those I've met on here.
 
Lucibee said:
Last week, I wrote to my ex-boss (Richard Horton) about my concerns regarding The PACE trial, and in particular, the way the review process was handled.

Over the weekend, I got a response. They won't budge on this. As far as The Lancet is concerned, PACE trial is "science". So I suspect that any criticism of it is therefore treated as anti-science.

I don't know where to go with this now. But I can't stop. If I wasn't activated before, I am now.
Click to expand...
As I said to @dave30th a while back, in so many words, when you hit a brick wall you cannot go through you then have to try and come at the problem from a different direction, rather than tackle head on. Maybe building on the political angle etc for more powerful allies, so there will come a point the Lancet etc simply cannot ignore all who criticise them.
 
Another thing I haven't mentioned (yet).

When I published my first blog on the PACE trial (2 years ago), Sonya Choudhury got in touch with me to ask if I would speak at the CMRC conference about it. I declined, because I felt that such an unscientific study had no place at a scientific conference. I think I also would have been thrown to the lions had I done so, knowing what I now know.
 
Lucibee said:
Over the weekend, I got a response. They won't budge on this. As far as The Lancet is concerned, PACE trial is "science". So I suspect that any criticism of it is therefore treated as anti-science.
Click to expand...

I suspect that is their attitude but in itself the attitude there is 'science' that tells truths and 'anti-science' that criticizes the truths is very unscientific. Science works by debate and analysis, does an experiment support a hypothesis, is the methodology strong enough to support conclusions, what about competing results?

It always concerns me how much science works by reputation rather than actual analysis. I suspect its getting worse if for no other reason than the volume of papers - too many academics have to publish too many rubbish papers to meet numbers rather than doing and properly reporting quality research.

In the end I think that it is important that the Lancet is on record of having been told what the problems are. They are taking a huge risk with their already questionable brand. How many bad publications can they stand before they become a joke and their reputation goes (no quality papers and readers who don't believe them). After all if they view reputation as important they need to know that their reputation can be harmed. Trust is hard to gain and easy to loose. However, companies only really get concerned when issues start to hit their profits or share prices. For the lancet profits will be linked to subscriber numbers (and university libraries) so unless they are loosing them then there will be little concern.
 
You must log in or register to reply here.
Back
Top Bottom