Bias due to a lack of blinding: a discussion

Michiel Tack said:
Sounds rather boring
Click to expand...

It's tactical... if giving the impression that you're on the fence makes sure you don't alienate readers who are firmly in either one camp or another. Her stance seems to be that "it's more complex than that", judging by the final paragraph.

It was the opening para that intrigued me though, because she said it was a screw up with randomisation, which it wasn't. It was defo the fault of an inability to blind the treatment allocation in PREDIMED that led to complaints by the participants. It was then the allocation itself that failed.

I still wonder how much the potential randomisation failure at the Edinburgh centre in PACE had on the outcome of the trial, and why no-one seems to care that it wasn't reported. I'd like to know what the treatment allocation was in the different centres, and whether failure at one centre helped the other centres predict which treatment was likely to be allocated next (there should have been an excess of GET places available at the other centres as a result). If there was any indication about participant preference prior to allocation, then that might have been a problem, and could lead to exaggerated responses. But I guess we'll never know.

eta: HB mentioned a paper on blinding by Kenneth Schulz and David Grimes from their very good Epidemiology series in The Lancet in 2002, which is worth a look.
 
Last edited:
I think I'm genuinely still on the fence about the new paper too. There are things that concern me about it, and the politics surrounding it and Sterne, but I don't feel like I have a good understanding of it yet.

edit: I'm on the fence on this paper - given the corruption we've seen around work like PACE, I'm not on the fence about the importance of blinding.
 
Last edited:
Finally received a response.

They said that the dataset will only be made available after a post-publication period of 1 year to allow time for follow-up studies, as stated in the main publication (under Data Sharing at the very end), and that they prefer not to share parts of the dataset at present.

Esther12 said:
A couple of new rapid responses about this too
Click to expand...
Most of these don't make much sense to me.
 
Michiel Tack said:
Finally received a response.

They said that the dataset will only be made available after a post-publication period of 1 year to allow time for follow-up studies, as stated in the main publication (under Data Sharing at the very end), and that they prefer not to share parts of the dataset at present.
Click to expand...

Thanks Michiel. That's annoying - the way they class trials as blinded or unblinded is a vital part of the paper so failing to let others check their assessment really makes it difficult to have an opinion on their findings. Given the way some people are talking about the importance of this review, making that information public would seem to be the responsible thing to do.

Maybe I should go back to that MS review. There's probably a way of working out what trials they assessed as blinded by looking at their data, once a list of likely blind/nonblind assessments was put together. It would be a pain though.
 
Esther12 said:
the way they class trials as blinded or unblinded is a vital part of the paper so failing to let others check their assessment really makes it difficult to have an opinion on their findings.
Click to expand...
I agree it's quite frustrating because I'm not asking about the raw data of an experiment. To me, it feels more like I'm asking about the methodology of their study rather than about a dataset.

It's a bit like someone does a systematic review and you ask them which studies they used and didn't use to form their conclusion. That's not really asking about a dataset, in my view. In a Cochrane review, for example, you have to list the studies selected and your risk of bias assessment. So I don't see why it should be different for this "meta-epidemiological study."

The BMJ study was accompanied by an editorial and a commentary paper and has received 7 rapid response. The authors have also published a separate paper on how to interpret meta-epidemiological studies with the MetaBLIND study as a case example. So clearly the results are already being discussed and will have an influence on trial design and interpretation of studies. If they aren't willing to release their dataset until they work out the follow-up studies, perhaps they should have waited with publishing their main results as well.

Esther12 said:
It would be a pain though.
Click to expand...
Yeah, and you would never be sure if your selection is the one used in the BMJ study. So I suspect we'll have to wait a whole year to find out more...
 
Michiel Tack said:
They said that the dataset will only be made available after a post-publication period of 1 year to allow time for follow-up studies, as stated in the main publication (under Data Sharing at the very end), and that they prefer not to share parts of the dataset at present.
Click to expand...

It sounds like they've misunderstood what you were asking for.
 
Lucibee said:
It sounds like they've misunderstood what you were asking for.
Click to expand...
I think they understand what I'm asking.

They are just treating the result of their systematic search of the literature as a dataset, like the raw data of an experiment - probably because they put in a lot of work to find and scan all these studies. I suspect their rationale for withholding is that they are concerned that other researchers might then use that dataset (their hard work) to publish papers and separate analyses on it.

I, however, see it as a part of the methodology that should be reported with the paper (for example in the supplementary material), much like a Cochrane review explains which studies it used and how these were assessed.
 
I guess it's possible that they googled your name and saw this thread? That could explain the delay in telling you they didn't want to share that information about their work.
 
Esther12 said:
I guess it's possible that they googled your name and saw this thread? That could explain the delay in telling you they didn't want to share that information about their work.
Click to expand...
Hmm, see no particular reason to think that. I suspect researchers are busy and sometimes forget to respond.
 
Adrian said:
It would also be interesting to know what he means by validated because I've read papers that claim to validate various 'scales' but the don't do anything I would consider reasonable validation. Sometimes validation is very hard as you have no way to measure ground truth to validate against (such as fatigue reports) so validations take the form of checks against other similar scales or just that you can pick a sick population from a healthy one. Sometimes test/retest is done but again this is hard in fluctuating diseases where there is no measure for ground truth. What I have never seen is any analysis which says scales are linear and hence improvement data can be aggregated - with most scales it is clearly not by looking at the questions. If I remember the CFQ had a number of significant principle components when the structure was analysed which seems to suggest its an aggregate measure rather than a scale in its own right (hence not good for measurements).
Click to expand...
I know I'm very, very late to the party here, but just wanted to comment on this.

There are a lot of different ways in which the word "validation" is used in the context of self-report instruments. They can mean any of these things:

Reliability: whether you get similar outcomes from testing different samples from a similar population, or the same sample at two timepoints (there are other types too, where you select out individudal items of the same kind and look at how consistent they are with one another). Technically this is not really a measure of validity, even though it is often referred to as such. Its just reliability. You need reliability before you can have validity but its not enough in itself.

Content validity: whether the items in your scale appear to sample: a) all of the aspects of your construct; and b) no aspects of other constructs that could contaminate your results. To take an example from the cognitive testing literature (just to illustrate the point), a test of working memory should include verbal as well as nonverbal test items, because we know these are different facets of the construct of working memory as its currently understood; at the same time, it shouldn't involve items that require knowledge that not all testees might posses (so, no questions the hinge on whether you have a good vocabulary). Content validity is usually assessed informally - you just show the reader what the items look like. So it isn't very strong.

Criterion validity (including concurrent and discriminant validity): whether two different tests designed to measure the same construct tend to rank people in the same way (concurrent validity), or whether they adequately discriminate between people categorised differently on a related test (discriminant validity) - for example, whether a depression scale effectively identifies between patients diagnosed with depression at interview. Again, this is not validity in my view, either, but another form of reliability. Usually, the scales are not truly independent - one is usually piggy-backed on the other - so its not very powerful. And even if the scales are truly independent, then its still really only a form of reliability.

None of these really assesses construct validity. To do that, you need to show that relationships between scores on your scale and other totally different measures are consistent with the theory on which the test was built. This is virtually never done in the land of self-report scales, its never even considered.
 
Just realized that we don't have a subforum on trial methdology, so I hope it's OK to leave this here:

Looking around what diverse experts say on the need for objective outcomes in open-label trials I repeatedly came across the Centre for Evidence Based Medicne (CEBM) at the University of Oxford: https://www.cebm.net/about-cebm/ and their 'Critical Appraisal Tools'.

Their 'critical appraisal' worksheet on RCTs I think doesn't say anything explicitly about open-label trials. It only states that:

"It is ideal if the study is ‘double-blinded’ – that is, both patients and investigators are unaware of treatment allocation. If the outcome is objective (eg death) then blinding is less critical. If the outcome is subjective (eg symptoms or function) then blinding of the outcome assessor is critical."

https://www.cebm.net/wp-content/uploads/2018/11/RCT.pdf

Apart from wondering about the example they chose for an objective outcome, I wonder why they omitted to explicitly mention open-label trials. But maybe I misunderstood.

The same center also produced a "response to systematic bias, wastage, error, and fraud in research underpinning patient care" -- paywalled & haven't read it, so just in case others find it interesting:

Heneghan Carl, Mahtani Kamal R, Goldacre Ben, Godlee Fiona, Macdonald Helen, Jarvies Duncan et al. Evidence based medicine manifesto for better healthcare: A response to systematic bias, wastage, error, and fraud in research underpinning patient care, BMJ 2017; 357 :j2973, https://www.bmj.com/content/357/bmj.j2973
 
MSEsperanza said:
Their 'critical appraisal' worksheet on RCTs I think doesn't say anything explicitly about open-label trials. It only states that:

"It is ideal if the study is ‘double-blinded’ – that is, both patients and investigators are unaware of treatment allocation. If the outcome is objective (eg death) then blinding is less critical. If the outcome is subjective (eg symptoms or function) then blinding of the outcome assessor is critical."
Click to expand...

I think this follows the approach we see in Wessely's textbook and in Jonathan Sterne's ROB2. What is egregious here is what is left out - any consideration the psychology of trials - the impact of knowing what is the 'test' treatment on the therapist and the patient. I am afraid I see this as deliberate.
 
It should be obvious that patients will always be the first line assessors of subjective outcomes. [edit - unless the outcome is substantially removed from patients' subjective experience]

Especially so to those heavily invested in the idea of 'illness beliefs' causing non-disease syndromes.

...

Of course if the syndrome is purely caused by subjective assessment, then maybe the cure can be effected purely through a different subjective assessment. How convenient!

I'm afraid I agree that this is a feature, not a bug.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom