When is lack of scientific integrity a reason for retracting a paper? A case study.(2020) Fiedorowicz et al. (about homeopathy for CFS)

Woolie

Woolie

Senior Member
When is lack of scientific integrity a reason for retracting a paper? A case study.
Abstract:
Objective: The present and past editors of this journal received a request to retract a paper reporting the results of a triple-blind randomized placebo-controlled trial. They expand on their decision not to retract this paper in spite of undisputable evidence of scientific misconduct on behalf of one of the investigators.
Click to expand...
Methods: The editors present an ethical reflection on the request to retract this randomized clinical trial with consideration of relevant guidelines from the Committee on Publication Ethics (COPE) and the International Committee of Medical Journal Editors (ICMJE) applied to the unique contextual issues of this case.
Results: In this case, scientific misconduct by a blinded provider of a homeopathy intervention attempted to undermine the study blind. As part of the study, the integrity of the study blind was assessed.Neither participants nor homeopaths were able to identify whether the participant was assigned to homeopathic medicine or placebo. Central to the decision not to retract the paper was the fact that the rigorous scientific design provided evidence that the outcome of the study was not affected by the misconduct. The misconduct itself was thought to be insufficient reason to retract the paper.
Conclusion: Retracting a paper of which the outcome is still valid was in itself considered unethical, as it takes away the opportunity to benefit from its results, rendering the whole study useless. In such cases, scientific misconduct is better handled through other professional channels.
Click to expand...

This editorial has just come out in the Journal of Psychosomatic Research. It is a discussion of issues that arose from this 2004 publication:

Weatherley-Jones et al. A randomised, controlled, triple-blind trial of the efficacy of homeopathic treatment for chronic fatigue syndrome. J Psychosom Res. 2004;56(2):189-97.
Click to expand...

This study is a triple-blinded randomised controlled trial of homeopathy for CFS. It might not come as a surprise to you that the primary outcomes of the study were all negative.

The current discussion is related to one of the co-authors, who stood up in a talk recently and said that she had worked out a cunning plan for discovering whether participants were in the homeopathy or the control group.

The current editorial explains why the journal did NOT retract the paper (because the results were all negative anyway, amongst other reasons).

The interesting bit is the issues they chose to comment on (emphasis mine):

While we cannot know for certain that her motivation was to discount the results of this study, what she said clearly seeks to undermine the credibility of a trial whose results challenged her firmly held but untested beliefs about the benefit of a treatment that she had high allegiance to
Click to expand...

Reporting on the integrity of the blind has merit and is especially valuable when dealing with subjective outcomes for which there is a greater risk of bias due to any unblinding.... Un-blinded assessors of subjective binary outcomes may exaggerate odds ratios by an average of 36% (13). Subjective outcomes are frequently used in studies that fall within this journal's scope, at the interface of psychology and medicine. We recommend assessing the integrity of the blind for any clinical trial, particularly those utilizing subjective outcomes akin to the primary outcomes of the Weatherley-Jones et al. study in question.

13. Hrobjartsson A, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012;344:e1119.
Click to expand...

None of us here will miss the irony that these requirements for good studies are simply waived for psychotherapy trials, without anyone even caring about the risk of bias. Silly homeopathy folks - don't they know that all they need to do to get around this is to combine their medicine with a wee chat. Hey presto - blinding problems solved!
 
Last edited:
Thanks for flagging this @Woolie. This is a nice ironic illustration of the core problem the GRADE enthusiasts and Cochrane reviewers seem to find hard to understand. Perhaps I could have cited this and Hrobjartsson in my testimony to NICE.

The citation of Hrobjartsson is interesting, though, for the way it gives the impression that you can readily put numbers on the problem. The first thing that is puzzling is that Hrob. measures the problem in terms of odds ratios. Looking at the paper this is because they decided to consider results in terms of binary good or bad outcomes. Clearly that would not apply to a result based on a comparison using a graded scale. The is no odds ratio for PACE as far asI can work out (unless you use 'recovery'). So the figure of 36% does not tell us much, if anything.

I rather suspect this is statisticians extracting the wrong number. What we want to know is the chance that using unblinded assessors leads to a clinically significant difference in outcomes. And of course that cannot be assessed from sampling the very few trials available that used both blinded and unblinded assessors. These turn out to be trials of rather esoteric things like retinopathy. There would be no way to extrapolate them to ME/CFS or anything else much.

So maybe all Hrobjartsson's studies (there are several) can tell us is that yes there is a problem. Which is fine but we knew that and I think there is also a serious question about whether the studies might have a serious bias of their own.

Hrob's studies have the very unusual feature of using both blinded and unblinded assessors and making use of this to assess the importance of blinding. This means that the trials were deliberately designed in a frame of mind that took blinding seriously and wanted to analyse its effects. And arguably the whole problem about trials using unblinded assessors and subjective outcomes is that they are set up by people who are not in this frame of mind but rather prefer to sweep the problem of blinding under the carpet. Being an unblinded assessor in a trial assessing the importance of blinding is going to be very different from being an unblinded assessor in a trial that ignores it.
 
Woolie said:
Silly homeopathy folks - don't they know that all they need to do to get around this is to combine their medicine with a wee chat. Hey presto - blinding problems solved!
Click to expand...

This is similar to how homeopathy is often delivered. Homeopaths believe that they can figure out the right homeopathic remedy by learning a lot of about the patient. They are an attentive listener who will take time for the patient. This is probably an important reason why homeopathy has survived.

It's not that different from psychosomatic medicine therapists searching for ways to connect the patient's medically unexplained symptoms to some psychosocial factors.
 
strategist said:
This is similar to how homeopathy is often delivered. Homeopaths believe that they can figure out the right homeopathic remedy by learning a lot of about the patient. They are an attentive listener who will take time for the patient. This is probably an important reason why homeopathy has survived.

It's not that different from psychosomatic medicine therapists searching for ways to connect the patient's medically unexplained symptoms to some psychosocial factors.
Click to expand...

It may not be that different from the Hawthorne effect https://en.wikipedia.org/wiki/Hawthorne_effect indeed @Jonathan Edwards advice that he showed empathy to his patients and tried to support them --- OK Jonathan was developing treatments that worked rather than just being interested!
 
Last edited:
Jonathan Edwards said:
The citation of Hrobjartsson is interesting, though, for the way it gives the impression that you can readily put numbers on the problem. The first thing that is puzzling is that Hrob. measures the problem in terms of odds ratios. Looking at the paper this is because they decided to consider results in terms of binary good or bad outcomes. Clearly that would not apply to a result based on a comparison using a graded scale. The is no odds ratio for PACE as far asI can work out (unless you use 'recovery'). So the figure of 36% does not tell us much, if anything.
Click to expand...
Actually the PACE trial's pre-registered primary outcomes were binary (whether fatigue was substantially reduced at trial end point, and also whether physical function substantially improved).

You're right that although ORs are used as estimates of effect sizes in studies using binary outcomes, they can't be directly compared to the kinds of effect size measures we use with continuous outcomes (like Cohen's d). I think the main point, though. about the Hrobartsson et al finding is that it suggests that outcomes can be very heavily inflated by bias resulting from the combination of non-blinding and subjective outcome measures - that is, not just by a bit, but by a lot.
 
Woolie said:
No, I think they dug through literature for trials that ran in two phases - blinded and nonblinded.
Click to expand...

That puzzles me because unblinded and blinded stages in drug development are widespread. If these are the very few that have unblinded and blinded phases within the same trial that still suggests to me that these people were taking blinding rather seriously and that people involved in the trial would be aware that they were in an unblinded or blinded phase. I can see that the original triallists may not have wanted to make a formal estimate of effect on odds ratio themselves but they must have taken blinding seriously to think it was worth repeating the same study with blinding after doing it unblinded (presumably to see if it had an effect)? That would be very different from the cases we are interested in where people firmly took the view that it was not important based on a textbook chapter that conveniently avoided spelling things out clearly! It still seems to me to be a very unrepresentative sample - but my concern about bias here is that Hrob is likely to have significantly underestimated the problem.

I agree that the value of the paper is in showing significant effects from unblinding. My scepticism is about trying to pull out numbers that then get quoted as indicators (largely because I suspect that the reality is much worse still).
 
Last edited:
The statement in the editorial about caution regarding subjective outcomes considering that Per Fink, Rosmalen, Sharpe, White etc are on the journal's advisory board. They don't seem to agree with the journal's beliefs about the bias inherent in combining subjective outcomes and non-blindedness.
 
Trial by Error by David Tuller: Psychosomatics Journal Linked to PACE Authors Highlights Bias from Subjective Outcome

Now the current and former editors of the Journal of Psychosomatic Research have made declarative statements about about blinding and subjective outcomes. So I’m curious: What is their opinion of the PACE trial? Do Fiedorowicz, Levenson and Leentjens know that this debacle was unblinded and relied on subjective outcomes, and that the investigators assessed their objective findings as irrelevant? Do they care about this contradiction between their own pronouncements on research integrity and the apparent belief by members of their advisory board that combining unblinded studies and subjective findings does not present a problem to proclaiming treatment success?
Click to expand...
 
@dave30th, its my impression that Jess Fiedorowicz is very much trying to clean up the dodgy science in this area and improve the quality of the research. I can't talk specifics in public, but he has been actively working to ensure BPS style papers are receiving rigorous reviews before publication. He's selecting reviewers from a range of perspectives (including those who've written against BPS-style approaches), and really listening to what they say.
 
You must log in or register to reply here.
Back
Top