GETSET letters in The Lancet

It's a certainty if they subgroup the data like that! Those with low scores will tend to improve; those with high scores will tend to get worse (before any treatment effects). They don't seem to have selected on PF this time, but that will have happened to a certain extent anyway, by nature of the selection criteria for CFS.

They also said:

So it looks like they did have some kind of cut-off originally (at least in principle), to allow themselves room for improvement. In some ways, it's good they weren't selecting patients on the measure they were going to use at outcome. But they would have been using selection measures that were highly correlated with it.

But didn't they concede that by saying that there might be a ceiling effect? I think that's where I'm getting confused, because otherwise I don't understand what they haven't admitted to. To anyone who knows the PF scale, the improvement they reported is truly tiny.

I certainly agree that it would be useful to see the data. A nice set of histograms of the baseline and outcome measures in the 2 groups would make things much clearer.

 

They are not conceding anything by saying there is a ceiling effect in my view. 100 can be a ceiling. There could be a situation where people were at an average of 90 initially and you can't expect them to get that much better so there is a ceiling effect there. (Say a weight loss program where some people are close to a healthy weight, while others had more to lose). So that is what they seem to be suggesting: the group that started off high couldn't improve much as their scores were already pretty good.

However, when you actually look at the specifics, the subgroup with the initial higher scores either improved a tiny amount or actually decreased. And the final score averages around 54.9. We only know that from information about the other group, it is not information that is explicitly given in the paper. Their argument about a ceiling effect in this specific case could only be true if there is a very low ceiling. They haven't said that explicitly anywhere and I doubt they would concede it. Some people unfamiliar with the scale might think the ceiling they reached was normal functioning or something close to it.

Peter White thinks people can recover with GET and should be turned down for disability payments unless they have tried it. He hasn't said anything to suggest people could only get scores of 55-60 or so.

 

Ah - OK. There are two different types of ceiling effect. One could be in the scale itself - as you say, starting at 90 when 100 is the max. But there's also a treatment ceiling effect, whereby a treatment is acknowledged as only able to produce, say, a 10-point improvement.
I read it as a treatment ceiling, not a scale ceiling.

But it does seem that they've confused the two as well if they think that a score of 55 constitutes some kind of treatment ceiling.

ETA: But then that doesn't make sense if they say it is only relevant in those who had high scores at baseline. As you say, we need to know what those 'high' scores were.

ETASM: And you can't set a 'ceiling' at the mean.

 

If one crunches the numbers, the group with the lower initial scores (A) increased by an average of 16.9 points or more.
While the group with the higher initial scores (B) increased by a maximum of 2.7 and could even have decreased.

So a treatment ceiling (i.e. that they could only improve so much within the timeframe) doesn't justify the poor results for group B.

I asked for the scores so they would be put on record and people wouldn't need to trust my analysis. But I believe we have enough information to say that the ceiling effect doesn't justify the poor results for group B.

ETA: Only a small proportion of group B could have had high e.g. 85+ scores on completion given the average of around 54.9.

 

It's like an unfinished sentence - "but our finding that the effect size was greater in those with the worst baseline physical functioning suggests this might represent a ceiling effect ..." - that they deliberately avoid completing. Crucial to their results is what that ceiling effect might be.

 

Another reason is documentation and showing disagreement. The authors' responses to criticism are nearly always worthless in this field.

 

Claiming there could be a ceiling effect and not providing a histogram/visualisation or statistical test of the effect, leads to the claim lacking credibility. You'd lose marks doing this on an undergraduate project, it's simply not acceptable from professionals.

 

[my bold]

Therein lies the answer .

 

I think it looks more like regression to the mean. If they looked at the control group in the same way, they should see exactly the same, although without the 'treatment' effect the increases/decreases on each side would be more balanced (those below the mean would increase by about the same amount that those above the mean would decrease).

If only a few people were close to the top of the PF scale at baseline, I don't expect it would really make much difference, particularly as the overall mean values are so low.

 

Not so long as they don't state what ceiling effect they mean.

I suppose that if some participants had little physical function disability at baseline (as @Lucibee notes in post #41), so were close to good physical function anyway, then they would have started off close to a natural ceiling. Does it make sense to include such people into an assessment of PF improvement anyway? (genuine question). If all trial arms the same then maybe? If they meant this as a ceiling effect, then the authors might tacitly concede their trial design was questionable - and not wish to do so?

But as Tom says (if I understand right), if it's more likely the "ceiling effect" was simply that the treatment had run out of what little steam it had, surely they would be very chary of conceding that. Indeed that would seem like calling something a ceiling effect when in fact it's more of an outcome?

 

Note that this group had to have a baseline score of ≥45, which limits the proportion of high scores.

Here is what they said about possible ceiling effects in the main text

 

If anyone felt inclined, it would be interesting to get the baseline scores for the SF-36 physical function questionnaire using a freedom of information request.

These could be used to question their claim

The patients were divided up into two groups:

We know that the average final score for the lower group was 56.9. We also know that the average total final score was 55.7.
If we got the baseline scores, we could calculate the change scores for both groups. It looks like the higher initial group may even have decreased on average, it certainly didn't improve much given the jump for the other group from an average score of 40 or less.

The baseline scores of the higher initial group would probably be mainly 45-65, which would show they could easily have improved.

 

Are you sure?
https://www.space.com/20948-moon-property-ownership.html