Maybe this is what Goldin was referring to in her comment about the trial not being truly 'randomized'?
I'm too tired to think this through now but it did look like there was something odd with those figures and I hadn't seen people mention this before.
I had initially assumed that Goldin was referring to the fact that there was effectively a preselection bias, in so much as there may be, as discussed above, a bias in subject selection reflecting bias inherent in referral patterns to the Bath clinic. Also given there were exclusions based on willingness to participate, severity of MP, the subjective idea of not attending the research sessions because they were “too far” etc, the subject sample involves yet more potential imbalance. However this is potentially relevant to any research, though some steps could have been taken to minimise some factors, such as drawing on subjects from more than one source, not just the Bath clinic, which might also have overcome the presumed recruitment problems that presumably lead to the researchers feeling the need to combine the feasibility into the main trail, contrary to good research practice.
The current updated version of the SMILE paper says this about randomisation:
Randomisation and masking
Allocation to trial arms was in equal proportions using minimisation by age (
12–15/16–18 years) and gender, weighted towards minimising the imbalance in trial arms with probability 0.8. Allocation was concealed using a telephone-based interactive voice response system, created and maintained by the Bristol Randomised Trials Collaboration and accessed by the recruiting researcher. This was an open study: the randomised intervention was conveyed after obtaining consent, during the recruitment interview so that participants, parents, therapists and researchers were aware of treatment allocation. Data analyses were conducted using masked treatment codes.
I am struggling to understand how the randomisation was done. The implication is that it was done by the Bristol RTC without reference to the researchers who found out the outcome for each subject by phone, but no explanation is given for the RTC’s methods. Further the above does suggest that the process had involved some attempt to match the two trial groups on age and gender, which potentially acted against randomisation. So it could be that Goldin was referring to the process allocating subjects between the two arms.
It is interesting the stress the Bristol researchers place on ‘randomisation’, for example the lead author of the newly revised Cochrane tool for assessing risk of bias (see S4ME thread on this
https://www.s4me.info/threads/rob-2...in-randomised-trials-2019-sterne-et-al.11025/ ) was also an author for the SMILE paper. This tool seems to suggest that randomisation is sufficient to eliminate any bias, even magically eliminate bias occurring at other stages in the research process, such as that arising from non blinded trials relying on subjective outcomes. So it is perhaps odd that this write up reports randomisation as if it were another black box in their research process. (The actual LP intervention being the most significant unexplained black box.)
Given the protocol and the various versions of the final write up refer to it as a ‘randomised controlled trial’. I just want to repeat the comments elsewhere that it is not actually adequately controlled. Given one group is ‘Specialist Medical Care’ and the other group is ‘Specialist Medical Care + Lightning Process’ there are potentially many differences between the two treatment arms that are not controlled for, from differences in amount of time seeing people perceived to be health professions to amount of contact with other children with the same condition. In an ideal world there would be a third placebo arm where the children were given some for of group intervention distinct to the Lightning Process, though there are potentially ethical issues of this and practical difficulties due to the nature of the Lightning Process, both because of the secrecy as to what is involved and the ‘believer’ status of the practitioners. [added Though it would not be beyond the whit of researchers to devise an ‘art & drama’ group cutting up bits of paper to create patterns on the floor and acting out different ‘health’ roles.
All this in addition to the problems already raised that Specialist Medical Care is not a standardised Intervention consistent between individuals, so we do not know how comparable it was between the trial arms. Indeed we can not rule out, as
@Trish says, that co-current involvement in the Lightning Process did not have a systematic modifying effect on the Specialist Medical Care creating systematic differences between this Care in the two trial arms.
So it is not reasonable to refer to this study as a ‘randomised controlled trial’ without significant caveats.]
