Bias due to a lack of blinding: a discussion

[Snow Leopard]

"The fact that many symptoms can only be recorded subjectively is also not a disadvantage because ultimately only the patient can evaluate the success of his or her own treatment. If a person learns to rate his or her own symptoms as less severe or threatening, then this can be seen as a genuine relief, since here too it is the subjective patient perspective alone that counts. Overall, therefore, no need for change to the methods paper is seen on this point." (*)

How would they know when researchers HAVE NEVER BOTHERED TO ASK PATIENTS WHICH OUTCOME MEASURES ARE MOST IMPORTANT?

We are the patients, we are the ones telling them their methodology is inadequate. They have NO RIGHT to presume what we think is important.

(caps because I'm feeling angry about this point)

 

[MSEsperanza]

Thank you all for your comments on the IQWiG reply (posts starting at #180 )

Apologies for the confusion. I've tried to add some context and order (update: and now also fixed the links.)

1) The section of the IQWiG's methods paper on which the quoted comments were submitted are here. (English translation)

2) The mentioned comments on that section and the reply of the IQWiG are here. (English translation)

3) The German original of the mentioned comments on that section and the reply of the IQWiG are here.


I haven't read through all comments yet but look forward to.

I think it is a very relevant discussion.

The methods paper was published recently, but is is reviewed every 2-3 years.

Perhaps more relevant: the IQWiG's draft report/ "preliminary"report on ME/CFS will be open to consultation later in the year.

What do others think, would it be worthwhile to send a comment on their reply (and cc it to the team working on the ME/CFS report)?

 

[Jonathan Edwards]

I find it very difficult to engage with documents of this sort, written by people who do not understand the material they are supposed to be expert in. If I had to respond I would probably have to say:

Several statements in the response to comments indicate a fundamental lack of understanding of the psychology of delivery of non-pharmacological treatments and how that impacts on patient reports and trial methodology. All I can offer to clarify my reasons is to refer people to my written expert witness testimony to the UK NICE guideline committee for ME/CFS.
https://doctorswith.me/wp-content/uploads/NICE-2021-Expert-Testimony.pdf


(I am not sure whether or not this testimony is still directly accessible through NICE, as it should be.)

 

[Trish]

(I am not sure whether or not this testimony is still directly accessible through NICE, as it should be.)

Yes it is. Listed and linked on this page under Appendix 3 Expert testimonies
https://www.nice.org.uk/guidance/ng206/evidence

 

[Jonathan Edwards]

Yes it is. Listed and linked on this page under Appendix 3 Expert testimonies
https://www.nice.org.uk/guidance/ng206/evidence

That's good. But trying to Google it does not easily come up with the URL, despite putting in lots of relevant words.

 

[rvallee]

How would they know when researchers HAVE NEVER BOTHERED TO ASK PATIENTS WHICH OUTCOME MEASURES ARE MOST IMPORTANT?

Not quite true. They asked, it's just that the responses they get, symptom burden and disability, don't fit anywhere in their ideology, so they simply ignored it.

Every time I saw something like this, either directly asking or in more free form, it's almost always the same: recovery means no symptoms, the illness is defined by the presence of disabling symptoms, especially the pattern of increased disability following exertion. But since they reject the presence of actual symptoms, merely our perception of, and convinced themselves that PEM is actually the same as being sore and tired after exercise in deconditioned people, then it makes no sense for them since their ideology explicitly rejects what needs to be done.

Same principle behind describing what ME feels like. The most accurate word is sick, we feel sick, ill. That word is not accepted, somehow, so they ask for words that are less accurate, and they settled on one that sounds benign instead and made everything not just about fatigue, but somehow all derived from it, made fatigue a super-symptom, or something like that.

They did ask. They did hear the responses. They rejected reality and substituted their own.

 

[MSEsperanza]

If you learn to cope with a symptom you should still rate the symptom at the same level unless you are being asked questions about coping with it, surely.

I do think the BPS aim is for the patient to reframe their sensations, to truly believe that the fatigue or pain they feel is inconsequential, not only just to say that it is.

Not really able to comment at the moment but I think it's worthwhile to discuss further on these two points in particular.

Another point I'd like to add to the discussion is more general:

In their reply, does the IQWiG just not consider the possibility that there could be a discrepancy between objective and subjective measures in unblinded trials?

Or would they rate the subjective outcomes to be superior to the objective outcomes?

With either reasoning, how can they at the same time justify that in drug trials blinding is needed?

 

[MSEsperanza]

Moreover, my discussions with old colleagues about this when I presented the 'Horrors of PACE' some years back, and also chit-chat over the years, makes it clear to me that cynicism and hypocrisy spill over any half-coherent theorising. The shift of the head or overt wink is hard to suppress.

This is not encouraging.

Is this only a reaction you experienced when ME was involved, or was it a general attitude towards patients?

if the former, do you also remember colleagues outside the field of ME who reacted differently?

 

[MSEsperanza]

If I had to respond I would probably have to say:

Thanks, Jonathan.

Your expert testimony is already included in the submisson's references:

"The problem of bias due to non-blinding of study arms in non-drug intervention studies is mentioned here. Two ways in which these biases can be mitigated are mentioned. However, it does not explicitly call for a combination of the two possibilities. Thus, it is left open how to assess the evidence from studies that could not be blinded and only blinded data collection, but at the same time used exclusively subjective, self-reported, unexamined objective outcomes as primary endpoints.

"This does not adequately address a significant factor for bias in study results. [4], [1]"

[1] Edwards, J, The difficulties of conducting intervention trials for the treatment of myalgic encephalomyelitis/chronic fatigue syndrome: Expert testimony presented to the NICE guideline committee, 06.09.2019, https://www.nice.org.uk/guidance/GID-NG10091/documents/supporting-documentation-3

[4] Tack, M., Tuller, D, Struthers, C, Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome, Fatigue: Biomedicine, Health & Behavior, 2020, DOI: 10.1080/21641846.2020.1848262

And the commenter's proposed amendment:

Comment (point 1)

Proposed change:

Often blinding of the personnel performing the intervention will be impossible and blinding of the patients will be difficult or also impossible.

In this case, to still obtain meaningful results, it is necessary in such studies to use robust objective endpoints in addition to blinded endpoint assessment.

In general, when evaluating the design of randomized clinical trials as well as reviews of non-pharmacological interventions, it is important not only to take into account non-blinding and the resulting risk of bias due to both practitioners' and patients' expectations, but also to check the extent to which this bias is controlled by other methods, particularly the use of primary objective endpoints. If subjective measurements or self-reported data alone are used as primary endpoints in a non-blinded study, the evidence for the benefit of the intervention under investigation is judged to be unreliable.

I think in their reply, the IQWiG didn't really respond to this point of the comment. They did not include it in their 'appreciation', only the other two points (see their reply here ). So this is what perhaps could be criticized?

 

[Jonathan Edwards]

Is this only a reaction you experienced when ME was involved, or was it a general attitude towards patients?

It relates to application of the BPS approach. That is predominantly an issue for unexplained chronic pain and ME but it can spread out into all rheumatological practice for many colleagues.

 

[Jonathan Edwards]

Your expert testimony is already included in the submisson's references:

But nobody is likely to have read it.
My testimony is an attempt to explain the blinding/subjective issue to people who do not understand. I worried that it was redundant because everyone already understood, but in reality the problem is the many who do not understand never will. Fortunately, the committee included a number of people who did understand - so maybe it was redundant after all!

 

[MSEsperanza]

How would they know when researchers HAVE NEVER BOTHERED TO ASK PATIENTS WHICH OUTCOME MEASURES ARE MOST IMPORTANT?

Not quite true. They asked, it's just that the responses they get, symptom burden and disability, don't fit anywhere in their ideology, so they simply ignored it.

Another important point to discuss further I think.

 

[FMMM1]

To understand the background of the IQWiG's reply, I'll also post the second point of the comment which unfortunately might have facilitated confusion.

Will add original text in German in following posts.

From a submission to the IQWiG's draft of the Methods Paper:

Comment (point 2) on section 3.4 (Non-drug interventions)

Comment:

The actual testability of endpoints for objectivity should be made explicit. As a negative example, we refer to a large study on the benefit of alternative medical therapy in children and adolescents that used school attendance as an endpoint. However, the data on school attendance were self-reported by the study participants alone and were not verified, even in random samples.[22]

School attendance was eventually included as a secondary endpoint, rather than as a primary endpoint, as one of further deviations from the procedures specified in the study protocol. Only in a corrected version of the study article published in response to repeated criticism did it mention that data on school attendance were exclusively self-reported and not verified against objective data, again contrary to what was stated in the study protocol. [23] [24] [25]

Previously, experts invited by the Science Media Centre (London) to comment on this study had failed to recognize this issue and had erroneously referred to school attendance as objective endpoints in their comments.[26], [27]

Proposed change:

In addition, when using objective endpoints that measure an effect of symptoms, such as physical/cognitive activity, motor function, school attendance, ability to work, etc., care should be taken to ensure that these are actually collected objectively and not solely through self-reported data from the study participants.

Edit: Comment (point 1) and Comment (point 3) see this post.

What strikes me about this is a positive i.e. the availability of objective outcome criteria [EDIT - like data on school attendance] for unblinded studies - similarly, US Army studies (to support veterans) used geographical (movement) data from a phone --- another alterative objective outcome criteria.

 

[Subtropical Island]

That sounds interesting. Can you share a link?

This seems to fit with what I've observed in other people who are on the milder end of illness: they get used to feeling that way, build a life that is compatible with their limitations and then begin thinking that they're "almost normal", or at least not truly ill.

I only ran across it as a comment in a published bit I was reading for other reasons. It was just nice to see an example of medical researchers pointing out how assumptions about this sort of thing need to be addressed (so that quite eminently treatable problems, with huge QoL impacts, do have an opportunity to be treated). I appreciate these counter examples, gives me hope.
I think this is it: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1774131/ published back in 2004.

Clinical consequence
For a long time it was thought that the clinical symptoms of anaemia (such as fatigue, headache, dizziness, shortness of breath, or tachycardia) occurred only when the haemoglobin level dropped abruptly. It had been argued that patients would adapt to low haemoglobin levels if anaemia developed slowly. This has led to the concept of asymptomatic anaemia. In truth, asymptomatic reflects the fact that impairments in physical condition, quality of life, and cognitive function may be unrecognised by both patients and their doctors. This is a non-IBD specific dilemma. Nephrologists have learned this lesson in the recent past.3 They showed that the process of adaptation to chronic anaemia was in fact adaptation to lower quality of life and that this could be reversed.

 

[Sean]

We are the patients, we are the ones telling them their methodology is inadequate.

Bizarre, isn't it. Patients are demanding more robust methodology, and (some) clinicians are demanding weaker methodology.

 

[rvallee]

Bizarre, isn't it. Patients are demanding more robust methodology, and (some) clinicians are demanding weaker methodology.

I don't think that's fair. Some are demanding no methodology at all, we should just trust them. Many literally submitted that as a legitimate argument about the NICE guidelines, they don't need methodology, they know.

There's something like homeopathic thinking behind this, the less methodology the purer the placebo results must be. Or maybe they're just patient-hating charlatans, same difference.

 

[Mithriel]

Yes and it is strange that we antiscience patients are wanting research into mitochondrial function while the royal collages are defending the Lightning Process.

 

[Sean]

I don't think that's fair. Some are demanding no methodology at all, we should just trust them. Many literally submitted that as a legitimate argument about the NICE guidelines, they don't need methodology, they know.

Indeed.

 

[MSEsperanza]

My testimony is an attempt to explain the blinding/subjective issue to people who do not understand. I worried that it was redundant because everyone already understood, but in reality the problem is the many who do not understand never will. Fortunately, the committee included a number of people who did understand - so maybe it was redundant after all!

Don't think it was redundant.

Also, still hope I or anyone else will be able someday to a) find people who could (help) translate it into German, b) to get -- with the help of your testimony and the paper by Tack et al -- other patient communities / umbrella organizations interested in the issue and c) for the next update on the IQWiG methods handbook, suggest patient organizations coordinate their submissions and take that issue into account.

If I have been able to learn so many things from discussions on S4ME and still do, even if I am slow and it often takes me a while and many questions, I think others can, too.


Edit: I now remember someone made me aware a while ago that a German translation of your testimony already exists. I just thought it could do with some edits.

 

[MSEsperanza]

This paper posted on a thread of its own might be a useful addition to the discussion (not related to ME/CFS) ?

Bias was reduced in an open-label trial through the removal of subjective elements from the outcome definition

Objective: To determine whether modifying an outcome definition to remove subjective elements reduced bias in a trial that could not use blinded outcome assessment.

Study design and setting: Reanalysis of an open-label trial comparing a restrictive vs. liberal transfusion strategy for gastrointestinal bleeding. The usual definition of the primary outcome, further bleeding, allows subjective clinical symptoms to be used alone for diagnosis, whereas the definition used in the trial required more objective confirmation by endoscopy. We compared treatment effect estimates for these two definitions.

Results: Fewer subjective symptom-identified events were confirmed using more objective methods in the restrictive arm (18%) than in the liberal arm (56%), indicating differential assessment between arms. An analysis using all events (both subjective and more objective) led to an odds ratio of 0.83 (95% confidence interval [CI]: 0.50-1.37). When only events confirmed using more objective methods were included, the odds ratio was 0.50 (95% CI: 0.32-0.78). The ratio of the odds ratios was 1.66, indicating that including unconfirmed events in the definition biased the treatment effect upward by 66%.

Conclusion: Modifying the outcome definition to exclude subjective elements substantially reduced bias. This may be a useful strategy for reducing bias in trials that cannot blind outcome assessment.

https://www.jclinepi.com/article/S0895-4356(16)30141-X/fulltext

Free access: https://discovery.ucl.ac.uk/id/epri...tive elements from the outcome definition.pdf