Bias due to a lack of blinding: a discussion

[rvallee]

Excellent, Michiel.

Would it be worth writing a follow up piece that gives an example of a paper that demonstrates subjective and objective findings being different, and contrasting it with one (like PACE) that goes out of its way to focus on subjective measures and ignore or mishandle objective outcomes, and the influence this has. ie a demonstration of the ongoing use and impact of such bad research.

Or do you think that's been done to death already!

The Rituximab trial? Probably the best one. The paper doesn't directly contrast it but frankly this is something worthy of a paper. Especially in being a comparative paper it would be accepted by the current NICE request for papers.

For all the gloating by the PACE cheerleaders about it failing, it clearly showed the same thing as PACE did, yet somehow reaching opposite conclusions.

 

[ME/CFS Skeptic]

"it's winners" should be "its winners" - A gratifying error

Thank you.

 

[Simon M]

@Michiel Tack That was a truly brilliant blog: a beautiful assembly of strong evidence against the use of non-blinded trials with subjective outcomes, and a great story about Louis XVI.

Just one thing: please don't blog about biomedical research, else I'm out of a job.

 

[Barry]

For all the gloating by the PACE cheerleaders about it failing, it clearly showed the same thing as PACE did, yet somehow reaching opposite conclusions.

The key difference being true professionalism from Rituximab trial scientist, pursuance of the truth being their primary objective rather than self interest. There is no way back for the PACE authors now.

 

[Esther12]

Thanks for all this Michiel - I haven't been able to read things through yet but I'm looking forward to doing so and seeing what you've dug up.

 

[Cheshire]

Thanks @Michiel Tack this is a brilliant and clear sum up of this central question.

 

[Esther12]

I've not read this new paper from Hróbjartsson, but thought I'd post it here as it has different conclusions (with different methods):

Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study
BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.l6802 (Published 21 January 2020) Cite this as: BMJ 2020;368:l6802

Conclusion No evidence was found for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors. These results could reflect that blinding is less important than often believed or meta-epidemiological study limitations, such as residual confounding or imprecision. At this stage, replication of this study is suggested and blinding should remain a methodological safeguard in trials.

Open Access: https://www.bmj.com/content/368/bmj.l6802


edit: Two linked commentaries:

Blindsided: challenging the dogma of masking in clinical trials
BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.m229 (Published 21 January 2020) Cite this as: BMJ 2020;368:m229

1. Aaron M Drucker, assistant professor12,
2. An-Wen Chan, associate professor12

https://www.bmj.com/content/368/bmj.m229
https://sci-hub.tw/https://www.bmj.com/content/368/bmj.m229

Fool’s gold? Why blinded trials are not always best
BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.l6228 (Published 21 January 2020) Cite this as: BMJ 2020;368:l6228

1. Rohan Anand,, doctoral research student1,
2. John Norrie,, professor2,
3. Judy M Bradley,, professor1,
4. Danny F McAuley,, clinical professor1,
5. Mike Clarke,, professor3

https://www.bmj.com/content/368/bmj.l6228
https://sci-hub.tw/https://www.bmj.com/content/368/bmj.l6228

 

[Esther12]

I just had a quick read of this - the results for different analyses have really wide confidence intervals, but seem fairly consistently around zero (Actually, I don't really have any intuitive sense of how close to zero effect these analyses are - they seem much closer than the other figures I've seen though - https://www.bmj.com/content/bmj/368/bmj.l6802/F2.large.jpg.) It seems odd that none of the things assessed were significantly associated with more positive/negative results.

They do say:

We are unclear to what extent our results show that blinding is less important than previously believed, show the limitations of the meta-epidemiological approach (eg, residual confounding), or show a lack of precision in the comparisons made. Until our study has been replicated, and we have a clearer understanding of which types of trials are susceptible to bias associated with lack of blinding, we suggest that blinding remains an important methodological safeguard in trials in which it is feasible.

Seems like a surprising result, to say the least. Sometimes surprising results are righ. Seems like it would be a lot of work to go through the meta-analyses/studies and check their workings though.

 

[Snow Leopard]

Interpret with caution... Their meta-meta analysis is of trials included in Cochrane reviews only. I suspect there is a strong bias against unblinded trials that don't rely on strong objective outcomes in the Cochrane database.

There is large heterogeneity in the results, suggesting that lack of blinding can lead to bias (consistent with prior meta-analyses), but does not automatically lead to bias.

Note that they are comparing unblinded trials to blinded trials for a given form of outcome measure, for example patient rated outcome measures, they did not compare objective vs subjective within the same trial and did not separate mortality from other forms of objective outcome measures. Secondly, many of these trials did not actually bother to check whether blinding was actually maintained. When drugs have clear effects (including side effects), it is obvious as to whether you are taking it or not.

Review methods Blinding status was retrieved from trial publications and authors, and results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the increases in heterogeneity between trials, for non-blinded trials (or of unclear status) versus blinded trials.

The generalisability of our results could be affected by the sampling strategy inherent in a meta-epidemiological approach. Thus, inclusion of only meta-analyses containing both blinded and non-blinded trials excludes situations where all trials are blinded (as blinding is considered of paramount importance) or, conversely, areas where all trials tend to be non-blinded. Similarly, review authors might be more likely to include both blinded and non-blinded trials in a meta-analysis when there is no clear difference in effect estimates between the two.

So the effect may be as a result of biases of selection in the Cochrane meta-analyses.

These are all the mentions of objective outcomes:

Observer reported outcomes were subdivided into four outcomes: objective—all cause mortality, objective—other than total mortality (eg, automatised non-repeatable laboratory tests), subjective—pure observation (eg, assessment of radiographs), and subjective—interactive (eg, assessment of clinical status). Subjective observer reported outcomes were scored 1-3 according to the degree of subjectivity (that is, the extent to which determination of the outcome depended on the judgment of the observer, with 1 indicating a low degree of subjectivity).

We did not primarily focus on trials with objective outcomes, such as all cause mortality, because we did not suspect any marked effect of blinding in such trials. We conducted univariable analyses for each contrast in blinding status using all informative meta-analyses for that characteristic.

Yet no data is provided for this analysis.

Results of secondary analyses looking separately at the effect of blinding patients, healthcare providers, or outcome assessors across different types of outcomes are shown in appendix table 5. For example, an analysis based on observer reported outcomes classified as objective also showed little evidence of an effect of outcome assessor blinding status (ROR 0.94, 95% credible interval 0.61 to 1.26; meta-analyses with a hypothesis of benefit only).

A pre-planned repetition of the main analyses based only on trials scored as definitely yes versus trials scored as definitely no proved unfeasible due to insufficient numbers of meta-analyses (appendix table 5). A post hoc analysis indicated about 10% exaggeration of the odds ratio in trials without adequate concealment of the allocation sequence (table 3).

 

[Jonathan Edwards]

The agreed measures of quality of evidence for trials in general are:

1. Well matched controls
2. Randomisation
3. Blinding

This is a trial of trials which has no well matched controls, is unrandomised and unblinded.
It should not even make it into a meta-analysis.

I think the authors are either remarkably naïve or disingenuous.

 

[Snow Leopard]

I think the authors are either remarkably naïve or disingenuous.

Which study are you commenting on?

We're discussing the study that was published yesterday that was looking at risks of bias in Cochrane meta-analyses where blinded trials were compared to non-blinded for the same class of outcome measure.
https://www.bmj.com/content/368/bmj.l6802

 

[Jonathan Edwards]

Which study are you commenting on?

We're discussing the study that was published yesterday that was looking at risks of bias in Cochrane meta-analyses where blinded trials were compared to non-blinded for the same class of outcome measure.
https://www.bmj.com/content/368/bmj.l6802

Same as you @Snow Leopard. As you point out the controls are not well matched and moreover, the sampling of data is not randomised or blinded for content.

 

[Snow Leopard]

The agreed measures of quality of evidence for trials in general are:

1. Well matched controls
2. Randomisation
3. Blinding

This is a trial of trials which has no well matched controls, is unrandomised and unblinded.
It should not even make it into a meta-analysis.

I think the authors are either remarkably naïve or disingenuous.

How would you design such a meta-analysis comparing (randomised) unblinded to blinded trials without requiring a whole series of new trials being conducted?

 

[Jonathan Edwards]

How would you design such a meta-analysis comparing (randomised) unblinded to blinded trials without requiring a whole series of new trials being conducted?

No idea, but that does not mean it is any good doing it this way.

 

[ME/CFS Skeptic]

Thanks for posting this @Esther12

The editorial and accompanying opinion are quite frustrating. I'm afraid this will give researchers a free pass when it comes to risk of bias due to lack of blinding. I suspect that in the future researchers will just have to note somewhere in their discussion section that lack of blinding in their trial is not ideal but that more recent studies show that there is not really an effect of blinding and that "blinded trials are not always best", referencing one of these papers.

The opinion piece by Anand et al. has sentences like: "Minimising biases with blinding might weaken the ability to predict the future accurately, because blinding is unlikely to be used in routine practice." It seems that authors prefer pragmatic trials that do resemble routine clinical practice and to spend the money saved by ditching efforts to properly blind trial participants on more of these pragmatic trials with larger sample sizes. I think that would be a step backwards, a step away from using the scientific method in medicine.

When it comes to the actual paper by Moustgaard/Higgins/Sterne/Hrobjartsson itself, I agree with Jonathan that the problem lies with the methods used. It involves looking at meta-analyses that have both trials that used blinding and others that didn't to see if the effect sizes of the latter are inflated. That's a very rough method because the trials differed in selection, design and probably even the duration, dose and characteristics of the intervention.

I think researchers basically assumed that this was ok: if you take enough meta-analyses all possible confounders would somehow be balanced and a true effect would emerge. Previous meta-epidemiological studies (summarized in this AHRQ report) suggested this was the case. They were in line of expectations if you take them all together, but the results were always a bit messy and the effect size of bias was lower than what one would expect. It should be noted that the previous largest study of this kind, the BRANDO project found the largest effect for lack of double-blinding for subjective/mixed outcomes with a ROR of 0.78 (0.65 to 0.92). The effects for other sources of bias such as inadequate randomization or allocation concealment were lower. So I expect that the same methodology if repeated, could also result in studies that 'demonstrate' that randomization and allocation concealment' are fool's gold as well.

I suspect that the main problem with this approach is selection bias: the meta-analyses that have both trials with blinded and trials with unblinded participants are probably those were blinding is considered less of an issue. Areas of medicine where blinding is considered important probably won't have meta-analyses with trials that are unblinded because these are considered inadequate. So if we focus on comparable trials that already exist, some with and some without blinding, we're probably underestimating the effect of blinding overall. I think a better way would be to try and conduct studies with blinded and unblinded groups in the same trial so that other factors are controlled for.

 

[Jonathan Edwards]

"Minimising biases with blinding might weaken the ability to predict the future accurately, because blinding is unlikely to be used in routine practice."

Yes, this is a very bizarre statement that appears to show a complete lack of understanding of how information is being used. It is a bit like saying why use a weighing machine to weigh the sugar for your cake when there won't be any weighing machine involved when you actually mix the cake mix.

I fear that there is a group of people with a very clear agenda to push pragmatic trials as part of an empire building exercise. These people now seem to be in charge of the relevant committees. Politics has always been politics but the fashion for pretending that shoddy procedures are gold standard when we know they are not seems new.

An interesting possibility is that treatments that do not work are more likely to be tried in unblinded studies because the researchers realise they might well not work. And of course if unblinded studies show nothing they are very unlikely to get published, whereas blinded studies that show nothing may be seen as important. Treatments that actually work are probably more likely to be subjected to blinded studies because the preliminary data make it pretty certain that there will be an effect. - and they are also more likely to be based on sound rationale.

 

[ME/CFS Skeptic]

An interesting possibility is that treatments that do not work are more likely to be tried in unblinded studies because the researchers realise they might well not work. And of course if unblinded studies show nothing they are very unlikely to get published, whereas blinded studies that show nothing may be seen as important. Treatments that actually work are probably more likely to be subjected to blinded studies because the preliminary data make it pretty certain that there will be an effect. - and they are also more likely to be based on sound rationale.

Not sure how this would work. If the treatment isn't effective I would still expect unblinded studies to show exaggerated effect sizes compared to blinded studies that test the same intervention because of bias. If unblinded studies that show nothing are very unlikely to get published, in contrast to blinded studies, then I would also expect to see inflated effect sizes in the unblinded studies compared to blinded studies that test the same intervention, which is what Moustgaard et al. couldn't find.

I suspect the main problem is that intervention where both blinded and unblinded trials exist are also the ones where blinding is considered less important and so form a subset of medicine where the influence of blinding is less compared to for example interventions where all trials are blinded (because it's considered very important to do so).

 

[Snow Leopard]

The editorial and accompanying opinion are quite frustrating.

The accompanying editorials seem to be a bit more opinionated compared to the tentative conclusions by the authors of the study, namely that practise (blinding) should not change and their findings should not be considered generalisable until replicated using other trial databases.

Nevertheless, despite the title "Fool’s gold? Why blinded trials are not always best", the conclusion was:

We suggest that the key elements for clinical trials seeking to minimise bias when comparing the effects of interventions should be adequate randomisation, allocation concealment, use of objective outcomes, independent blinded adjudication of outcomes, and, when possible, blinded assessment of outcomes.

Which brings a key point, that just because a trial is designed to be double-blinded, does not mean it lacks bias. On the other hand, I think most people agree that trials (such as cancer treatments) where the outcome measure is avoiding mortality, should not be blinded.

The authors have started to be active on twitter and cited this paper they published last month:
Ten questions to consider when interpreting results of a meta‐epidemiological study—the MetaBLIND study as a case
https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1392

Abstract
Randomized clinical trials underpin evidence‐based clinical practice, but flaws in their conduct may lead to biased estimates of intervention effects and hence invalid treatment recommendations. The main approach to the empirical study of bias is to collate a number of meta‐analyses and, within each, compare the results of trials with and without a methodological characteristic such as blinding of participants and health professionals. Estimated within‐meta‐analysis differences are combined across meta‐analyses, leading to an estimate of mean bias. Such “meta‐epidemiological” studies are published in increasing numbers and have the potential to inform trial design, assessment of risk of bias, and reporting guidelines. However, their interpretation is complicated by issues of confounding, imprecision, and applicability. We developed a guide for interpreting meta‐epidemiological studies, illustrated using MetaBLIND, a large study on the impact of blinding. Applying generally accepted principles of research methodology to meta‐epidemiology, we framed 10 questions covering the main issues to consider when interpreting results of such studies, including risk of systematic error, risk of random error, issues related to heterogeneity, and theoretical plausibility. We suggest that readers of a meta‐epidemiological study reflect comprehensively on the research question posed in the study, whether an experimental intervention was unequivocally identified for all included trials, the risk of misclassification of the trial characteristic, and the risk of confounding, i.e the adequacy of any adjustment for the likely confounders. We hope that our guide to interpretation of results of meta‐epidemiological studies is helpful for readers of such studies.

 

[Barry]

It seems odd that none of the things assessed were significantly associated with more positive/negative results.

I don't think the ORs were necessarily insignificant, regarding the trial outcomes. It was the RORs that was apparently insignificant - the ORs of the blinded trials versus the ORs of the unblinded ones. Even if all the ORs had been 10 the ROR's would then have been 1. Unless I'm misunderstanding.

 

[Esther12]

I don't think the ORs were necessarily insignificant, regarding the trial outcomes. It was the RORs that was apparently insignificant - the ORs of the blinded trials versus the ORs of the unblinded ones. Even if all the ORs had been 10 the ROR's would then have been 1. Unless I'm misunderstanding.

I was having to learn what those stats meant as I was reading the paper so it's possible I misunderstood, was not clear, or have now misunderstood your post! (To confuse things further, I think that I may have linked to a different table to the on I intended.)

So I was talking about the likelihood of blinding being associated with exaggerated effects estimates in any of the different scenarios they were looking at. Is it right that none of those analyses found a significant result?

I really need to re-read this paper and think about what their figures really mean.