Clinical and cost-effectiveness of the Lightning Process ... for paediatric chronic fatigue syndrome, 2018, Crawley et al (Smile Trial)

This is old but couldn't find it posted on the forum yet. Some interesting quotes in it.

Hawkes, N. (2017). Training for children with chronic fatigue works better than medical care alone, finds study. BMJ, j4372. doi:10.1136/bmj.j4372

sci-hub link: sci-hub.se/10.1136/bmj.j4372

Rapid response by Anton Mayer: https://www.bmj.com/content/358/bmj.j4372/rapid-responses

More about Hawkes:

https://www.s4me.info/threads/uk-he...uestion-february-2019.8045/page-5#post-142373

https://www.s4me.info/threads/trial-by-error-letters-to-fiona-godlee-and-nigel-hawkes.8233/
 
Could anyone check if the following paragraph is correct?

I would like to use it for a comment on the IQWiG report plan -- comment is due by Wednesday and there's a lot of work until then, so any help much appreciated.

That's my draft (translated from German) :

Edit -- go to:

MSEsperanza said:
I remember that bit and also some other issues.

Would this work:

"Explicitly mentioning the verifiability of seemingly objective end-points here seems important, as one large study of the benefits of an alternative medical therapy in children and adolescents (SMILE), for example, included school attendance as an endpoint. According to the study protocol, it was planned to verify the self-reported data by checking against school records in random samples, so that these could count as objective outcomes.

In the original trial paper, however, school attendance was self-reported by study participants alone, but this (among other) deviations from the study protocol where only added in a corrected version following repeated criticism."

Could the school absence study also be referenced? Sorry, I don't remember details about that one.

Edited to add amendments in bold.
Click to expand...

Explicitly mentioning the verifiability of self-reported 'objective' end-points here seems important, as one large study of the benefits of an alternative medical therapy in children and adolescents (SMILE), for example, used school attendance as an objective end point, but school attendance was self-reported by study participants alone and was not verified, even in random samples.

@dave30th @Lucibee @JohnTheJack @Jonathan Edwards @Sean @Snow Leopard @strategist @Esther12
@Robert 1973

Edited to add: I'm aware that there were many other issues with this trial, so that's only to check if the point about checking whether seemingly 'objective' outcomes are actually objective is one that can be made with respect to the SMILE trial.
 
Last edited:
Lucibee said:
The intention in the SMILE protocol was that they would check against school records - but the trial paper doesn't state that they did
Click to expand...

I remember that bit and also some other issues.

Would this work:

"Explicitly mentioning the verifiability of seemingly objective end-points here seems important, as one large study of the benefits of an alternative medical therapy in children and adolescents (SMILE), for example, included school attendance as an endpoint. According to the study protocol, it was planned to verify the self-reported data by checking against school records in random samples, so that these could count as objective outcomes.

In the original trial paper, however, school attendance was self-reported by study participants alone, but this (among other) deviations from the study protocol where only added in a corrected version following repeated criticism."

Could the school absence study also be referenced? Sorry, I don't remember details about that one.

Edited to add amendments in bold.

Edited again -- now completely re-phrased.
 
Last edited:
Had a look now at the corrected version -- Will have to reword the paragraph as the corrected paper now acknowledges:

"We did not have capacity to check school attendance using school records, but this could have provided an objective outcome."

And then they go on: "Further unpublished work suggests this is highly correlated with the self report measure we used."

What a bad correction to a bad paper that is -- and even the editorial note on the corrections seems again to be misleading.

Anyway, thanks @Lucibee that was already very helpful.

Any further help also appreciated (e.g. which measures did the school absence study use?)

Edit: Now edited the paragraph in the post above -- amendments in bold.
 
Last edited:
MSEsperanza said:
Any further help also appreciated (e.g. which measures did the school absence study use?)
Click to expand...

Is this the correct one? https://bmjopen.bmj.com/content/1/2/e000252
(Thread: https://www.s4me.info/threads/unide...school-based-clinics-2011-crawley-et-al.12187)

If this is the correct study, then "School attendance was recorded at the time of assessment using a single-item inventory." So, same - self-report.

Also from Methods: "The attendance officer in each school identified all children in years 7–11 (ages 11–16 years) who had missed ≥20% of school over a 6-week term."

"We used school attendance at 6 months, ascertained through follow-up questionnaires and by review of medical notes, as our primary outcome."
 
I now found one of @dave30th's very helpful replies that answers also my questions about the SMILE trial:

dave30th said:
Here's what happened with the school absence records: In both the feasibility trial and full trial protocols, they promised to vet self-reported school attendance with the actual records. Self-reported school attendance is like self-reported physical function--it is not an "objective" measure. So vetting it against actual records makes sense. The primary outcome initially was school attendance at 6 months. They also planned to measure school attendance at other times, including 12 months, but those were secondary measures.

When they published the paper, all the school attendance findings were secondary outcomes and solely self-reported. The paper made no mention of official school records. The self-reported six-month school attendance results, part of the original primary outcome (the rest of the original primary outcome would have been the official school records at six months), had null results. There were positive results at 12 months, so they touted that. But that was always a secondary result and in any event it was not vetted against official records. Or it was, and the records show otherwise, and they chose not to mention it. Either way, they promised official records and didn't provide them. And didn't mention the absence of these records in the paper. That's not allowed.

What Bishop got wrong was when she looked at it again was that she said, well, the school attendance was positive, so the outcome-swapping didn't make a difference anyway. Of course, she got two things wrong: 1) only the 12-month self-reported school attendance was positive; six month self-reported school attendance had null results, and school attendance at 6 months was the original primary outcome, not at 12 months. 2) They did not provide the official school record results. So even if Bishop had been right and the 12-month results were the original primary outcome, she would have been wrong because those results are not credible unless vetted by the actual records. The fact that they did not provide the actual records suggests they did not get them or they did and they didn't include them, for obvious reasons. I wrote Bishop back about this, but she ignored it.
Click to expand...
 
Lucibee said:
Just checking back. The intention in the SMILE protocol was that they would check against school records - but the trial paper doesn't state that they did - and the measure was downgraded to a secondary outcome measure anyway.
@Adrian will remember better than me probably...
Click to expand...

As far as I remember they intended to measure school attendence and talked about access to school records but they published results based on self-reported attendence with a claim that it correlated well with school records but no figures were given. School attendance can be quite dodgy in than many schools may have rooms where someone can rest and so they would be recorded as present even if not in all lessons.
 
Adrian said:
As far as I remember they intended to measure school attendence and talked about access to school records but they published results based on self-reported attendence with a claim that it correlated well with school records but no figures were given. School attendance can be quite dodgy in than many schools may have rooms where someone can rest and so they would be recorded as present even if not in all lessons.
Click to expand...
Would this not have been at the transition of GDPR?
Which may have made accessing school records a bit more difficult?
 
Adrian said:
As far as I remember they intended to measure school attendence and talked about access to school records but they published results based on self-reported attendence with a claim that it correlated well with school records but no figures were given. School attendance can be quite dodgy in than many schools may have rooms where someone can rest and so they would be recorded as present even if not in all lessons.
Click to expand...

In the protocols for both the feasibility and full trials (they wrote one for each), they promised to vet self-reported school attendance against official records. They didn't mention these official records one way or the other in the feasibility trial report and in the full trial report. The original primary outcome in the feasibility trial was school attendance at six months, measured both ways--self-report and school records. Physical function was a secondary outcome. So more than half the participants in the full trial were recruited when that was the primary outcome.

In the full trial, school attendance at six months had been relegated to being a secondary outcome. Of course, they still should have mentioned the school records, but they failed to. School attendance at six months had null results, but no one noticed. They had positive findings on another secondary outcome--school attendance at 12 months--so they did promote that as a benefit.

In the 3,000-word correction/clarification, it was stated that they didn't get the school records--I forget why exactly.
 
Sharing info on kids is more strictly controlled via GDPR legislation (2018 in Scotland)
Interests have to be legitimate and for general, non life threatening areas like this parental / childs' consent would be required .

In Scotland there was a legal challenge to GIRFEC ( get it right for every child which underpins education and social care provision ) re information sharing without consent as it conflicted with GDPR.

So it may boil down to how they handled the initial interaction ( given it involved attendance officers my guess is not well) - telling that letters / written info to parents etc were not provided despite being requested?

The school may also have had a part to play which they may not wish highlighted. They were facilitators.

My guess is that they did not have the legal consent required to obtain records.
It may not have been a necessity at feasibility when law was different and perhaps a bit more elastic in its definition and was therefore not considered at full trial stage.

Privilege tends to promote assumption.

Eta GIRFEC paragraph.
 
Thanks for replying again @dave30th -- I realize we had discussed this repeatedly before the correction was added (see e.g. post above )

dave30th said:
In the 3,000-word correction/clarification, it was stated that they didn't get the school records--I forget why exactly.
Click to expand...

"We did not have capacity to check school attendance using school records"

I wonder what people who aren't aware of the criticism and now read the editor's note and the corrected paper will make of the evidence.

I'll post other questions related to this and how to best refer to it in the comment for evidence reviews here.
 
Copying a comment I made back in 2017 on the Smile Trial over on Phoenix Rising - noting the missing school attendance data at 12 months, and the lack of clarity over how home tuition was quantified.
From Scary Vocal Critic's excellent comment:
The protocol that: “The primary outcome measure for the interventions will be school attendance/home tuition at 6 months.” It is worth noting that the new SMILE paper reported that there was no significant difference between groups for what was the trial’s primary outcome. There was a significant difference at 12 months, but by this point data on school attendance was missing for one third of the participants of the LP arm.
Click to expand...


I had not fully appreciated that the intention of the trial was to count both school attendance and home tuition.

Having supported my son through periods of both, I know that school attendance requires quite a different (and much better) level of health. Home tuition might consist of an hour of contact time with a teacher or parent, and then a few hours lying in bed working through questions. Or something quite different. But almost certainly not 6 hours a day sitting up being taught by a visiting teacher. Home tuition is very hard to define exactly or to quantify in terms of 'days of education'.

It seems possible, if school and home tuition are counted as the same thing, that a child might be attending school three days a week at the start of the trial and have moved to a relaxed 'home tuition' four days a week by the end of the trial and therefore count as improved on the measure of school attendance. They may even feel better now that they aren't using so much energy getting to school and sitting in class. But they still have ME.

I would like to know if 'home tuition' was counted as 'school attendance' in the SMILE paper and, if so, how it was defined. It would also be good to know what proportion of the 'school attendance' was actually 'home tuition', at the beginning and end of the trial.

Given that it seems that Crawley moved from a planned 'school attendance as reported by the school' to 'school attendance as reported by the parents/child', the inclusion of 'home tuition' as 'school attendance' muddies the water considerably. There is a lot more leeway for bias in how home tuition is quantified in self-reporting.

Also, the impact of 'home tuition' on the carer is much different to 'reliable school attendance'. Where home tuition is occurring, a parent is almost certainly at home, supporting the child. The impact on the cost of the disease to the family and society is much greater than if the child is able to reliably go to school. If home tuition continued to be counted as equivalent to school attendance, I wonder how that was handled in the cost benefit analysis.

Actually even 'school attendance that doesn't include home tuition' is quite a nuanced thing when it comes to cost benefit analyses of a treatment. If a child can reliably attend school three days a week, on the same days each week, then the carer can potentially work. But ME typically isn't like that. What is possible one week is not possible the next. This makes it very difficult for the carer to have a job that requires any sort of regularity in attendance or even productivity.

Even if the child was attending school for three days a week at the beginning of the trial and managed to attend 4 days a week for the particular two weeks the parents chose to report on at the end of the trial, that should not automatically be assumed to result in the carer being able to work and earn money for an extra day a week. It is quite possible that the child will need to be in bed all of the next week.

Clearly, for a sound cost benefit analysis of LP, we need to know if there was a stable improvement in the capacity of the child to attend school. That is going to require actual school records for at least a period of months, not two weeks. Preferably, due to the variability of school holidays and the cumulative effect of over-exertion, school attendance needs to be tracked for the year following treatment.
Click to expand...
 
Hutan said:
Copying a comment I made back in 2017 on the Smile Trial over on Phoenix Rising - noting the missing school attendance data at 12 months, and the lack of clarity over how home tuition was quantified.
Click to expand...
I was nearly done with high school when I got ill, and after having tried and failed two years to attend school, I became well enough by the third year that I could have 10 hours attendance/week in the autumn semester and 20 hours/week in the spring semester + home school for the rest. This solution allowed me to take courses that required labwork and/or had a lot of extra if one where to take it at home.

I've also done a lot of extra coursework as a university student, but then I opted for almost completely staying at home, mostly showing up for lectures if they were compulsory or on something I found very interesting. Sure my course plan looks impressive with 2-3x the normal courseload several terms, but I was still veyr ill and often I'd work while reclining in the bed at home.
 
It's been similar with my son, with a patchwork of rest, home learning, and part-time school from age 13. Some years he managed full course loads, but never full-time at school, and approaches were constantly changing with opportunities and health. It became much easier for him to have a full course load in university (during the pandemic) where he could watch lectures from home.

The complexity of describing education participation probably does make it unsuitable as an outcome, as Crawley et al found out. As others have said, the correct response to that awareness should have been the use of another, better objective outcome such as some form of automated activity monitoring, rather than making the primary outcome a subjective survey.
 
Lucibee said:
Just checking back. The intention in the SMILE protocol was that they would check against school records - but the trial paper doesn't state that they did - and the measure was downgraded to a secondary outcome measure anyway.
@Adrian will remember better than me probably...
Click to expand...
Given how schools work re: progrsss monitoring there should be predicted grades or attainment almost for each person who gets NE/CFS because they use this to measure ‘value added’ for schools by seeing whether a pupil who was eg really clever at start of year attained what they’d be reasonably expected to with good teaching, and do fir middle and lower down etc to total up a score.

no reason why - given my big bear if watching as the focus seems to be attendance at the expense of future recovery sbd health and ergo future grades bring worse than if say home learning or fewer subjects or years out had been possible - all of these different management possibilities couldn’t be compared based on actual ‘what grades should they have got vs what did they get in the end’ measure. Which would be more accurate as per how the system measures it.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom