SMILE patient cohorts

I had initially assumed that Goldin was referring to the fact that there was effectively a preselection bias, in so much as there may be, as discussed above, a bias in subject selection reflecting bias inherent in referral patterns to the Bath clinic. Also given there were exclusions based on willingness to participate, severity of MP, the subjective idea of not attending the research sessions because they were “too far” etc, the subject sample involves yet more potential imbalance. However this is potentially relevant to any research, though some steps could have been taken to minimise some factors, such as drawing on subjects from more than one source, not just the Bath clinic, which might also have overcome the presumed recruitment problems that presumably lead to the researchers feeling the need to combine the feasibility into the main trail, contrary to good research practice.

The current updated version of the SMILE paper says this about randomisation:

I am struggling to understand how the randomisation was done. The implication is that it was done by the Bristol RTC without reference to the researchers who found out the outcome for each subject by phone, but no explanation is given for the RTC’s methods. Further the above does suggest that the process had involved some attempt to match the two trial groups on age and gender, which potentially acted against randomisation. So it could be that Goldin was referring to the process allocating subjects between the two arms.

It is interesting the stress the Bristol researchers place on ‘randomisation’, for example the lead author of the newly revised Cochrane tool for assessing risk of bias (see S4ME thread on this https://www.s4me.info/threads/rob-2...in-randomised-trials-2019-sterne-et-al.11025/ ) was also an author for the SMILE paper. This tool seems to suggest that randomisation is sufficient to eliminate any bias, even magically eliminate bias occurring at other stages in the research process, such as that arising from non blinded trials relying on subjective outcomes. So it is perhaps odd that this write up reports randomisation as if it were another black box in their research process. (The actual LP intervention being the most significant unexplained black box.)

Given the protocol and the various versions of the final write up refer to it as a ‘randomised controlled trial’. I just want to repeat the comments elsewhere that it is not actually adequately controlled. Given one group is ‘Specialist Medical Care’ and the other group is ‘Specialist Medical Care + Lightning Process’ there are potentially many differences between the two treatment arms that are not controlled for, from differences in amount of time seeing people perceived to be health professions to amount of contact with other children with the same condition. In an ideal world there would be a third placebo arm where the children were given some for of group intervention distinct to the Lightning Process, though there are potentially ethical issues of this and practical difficulties due to the nature of the Lightning Process, both because of the secrecy as to what is involved and the ‘believer’ status of the practitioners. [added Though it would not be beyond the whit of researchers to devise an ‘art & drama’ group cutting up bits of paper to create patterns on the floor and acting out different ‘health’ roles.

All this in addition to the problems already raised that Specialist Medical Care is not a standardised Intervention consistent between individuals, so we do not know how comparable it was between the trial arms. Indeed we can not rule out, as @Trish says, that co-current involvement in the Lightning Process did not have a systematic modifying effect on the Specialist Medical Care creating systematic differences between this Care in the two trial arms.

So it is not reasonable to refer to this study as a ‘randomised controlled trial’ without significant caveats.]

 

Surely you either randomise or you don't. They didn't. So to advertise LP on the basis of SMILE being a RCT is blatantly untrue.

I would have thought that if people decided to not continue with their allocation, then their only other choice should be to drop out.

It's just another example of switching trial criteria and so introduce the potential for cherry picking. How can you legitimately 'unrandomise' some participants after you have done the randomisation? And then crow about RCTs being the gold standard. These folk seem to just play God with scientific methodology.

 

But then, you can't force people to receive the treatment they've been allocated, or to not pursue other treatments outside of the trial.

 

The patients that don't stick to their assigned treatment arm should in my opinion be counted as having dropped out.

I wonder if that would change any of the conclusions or moves the outcomes over the magical 0.05 threshold.

After all, 12 people assigned to LP either didn't get the intended dose of LP, so to speak. That's 23%.

It is absurd that these participants are apparently counted as having received LP when 9 didn't actually receive LP. Only 39 received the full dose of LP, but in the analysis they include 44 participants. Or maybe I'm misunderstanding but this seems to be a mess that's hard to interpret.

 

I guess they counted as having been randomised to receive free LP. I can see costs/benefits to either way of doing things and you can't impose perfect randomisation on people in a trial like this. Do we know if they prespecified how they'd analyse data for participants like these?

In addition to the problem of how to use data from those who did not follow through on the intervention they were randomised to, there was also the issue @dreampop pointed out of some of the figures seeming odd (though I may have misunderstood the point they were making).

 

Either patients that effectively self selected their treatment arm should be excluded from the analysis or they should be explicitly reported in the write up with a clear indication that this compromises the randomisation, not just left for readers to try to work out what happened.

It is interesting that it is taking so much analysis to work out what actually happened in this study, what does this tell us about the reporting. Is it indicating a careless write up, which ought not to be the case given this is a redraft following significant scrutiny? Is it a result of obfuscation? Or is it that the researchers have problems understanding the issues themselves?

[The title of the SMILE write up needs editing to say ‘a sort of randomised partially controlled trial with other gross methodological failings’.]

 

51 allocated to SMC+LP, but 9 only got SMC, and 3 got SMC+1 day of LP (which must be less than the full dose).

Only 39 received SMC+LP in the intended dose.
Only 42 received any LP at all.

Their statistical analysis for the SMC+LP group includes 44 participants. That's what is left after not including 7 participants for reasons other than not receiving LP.

Why are the researchers doing this? An uncharitable explanation is that this is an instance of p-hacking.

 

True, but I don't see your point. The point, surely, is that they don't pursue their favoured treatments as part of or during the trial; the trial results don't then get skewed by non-randomisation. I would also have thought it reasonable that if people wish to remain part of the trial, then a sign-up condition is they avoid any other treatments outside of the trial until they are finished with the trial. If they decide not to then that is fine, but by default they would be dropped from the trial.

ETA: X-posted with others saying much the same thing.

 

Bizarrely it is already inherent in this study that the active treatment condition participants are already engaged in other unspecified treatments as part of the Specialist Medical Care.

 

In general terms I think shifting people about like this would mostly be expected to dilute and difference between test and comparator, so prima facie, this may not be a big deal. The fact that the numbers in the group seem to keep jumping about may be more a concern in terms of sloppy analysis, although it is very easy to end up with one or two adrift.

Nevertheless, with an unblinded trial I have a suspicion that all the comforting aspects of intention to treat and risk being mostly of blunting are much less clear. If you switch treatments you may be even more likely to give the 'right' answer, just to show courtesy or even to make up for feeling guilty about screwing up their research.

 

The patients in the LP group who didn't actually end up doing LP were probably not random events. If the reason they didn't do LP was because "they were not ready" (didn't believe in the treatment), it could have plausibly inflated the efficacy of LP.

Any way it's nonsense to count them as having done LP in the statistical analysis plan when they didn't actually do LP (at least that seems to have happened).

 

The point of intention to treat analysis is that inflation is worse if you cut these people out because you then average just over the ones who liked it. If you use intention to treat then the ones who decided they didn't like LP are included and they drag the scores down rather than inflate.

We may be talking at cross purpose here - not sure.

 

Yes and this dragging down of scores can be avoided by simply not giving LP to people that don't believe in it, even if they were originally assigned to it and then counting them as having received LP (unless I'm misunderstanding that part).

Unless they switched these presumed LP refusers over to the SMC group, in which case the randomization is broken because some person is making decisions about which group a patient should be in after being randomly assigned to a treatment arm.

 

I think you may be misunderstanding. If they are counted as having received LP and they did not get brainwashed into saying they were better they will drag down the average reported improvement for LP - which is better than switching them to the other group and allowing the LP average to be based purely on brainwashed cases.

 

If disbelievers had been sent to do LP, they could plausibly drag down the average of the LP group because they might find it weird and inappropriate and form a negative opinion of the clinical trial. By not sending them to do LP, they could plausibly be inflating the average.

I now checked and as far as I can see, the published protocol says nothing about how the "assigned to SMC+LP but didn't do LP" subgroup in the SMC+LP arm will be analysed. That would give them the freedom to include or exclude as they want.

The protocol seems to confirm that these participants did not proceed to LP because they didn't believe in it:

I also noticed this:

Had the participants that did not receive LP or had only 1 session instead of the planned 3 been excluded from the statistical analysis, they would have had only 32 people. They would have lost 23.5% of the group.

 

I agree that that is possible. I guess that normally with intention to treat if someone drops out of a treatment they get a null score of some sort - like lost to follow up. People write long papers about al the possible biases that can come in with different ways of dealing with these. What would be a problem would be if the people who decided against LP were so relieved to get shot of it that they really enjoyed SMC and said they were marvellous.

So yes, in this unblinded scenario with cognitive manipulation as part of the treatment all sorts of rum things could happen.