1. Guest, the 'News in Brief' for the week beginning 7th June 2021 is here.
    Dismiss Notice
  2. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice
  3. Contribute to feedback on the CDC Evidence Review, for more details click here
    Dismiss Notice

When is lack of scientific integrity a reason for retracting a paper? A case study.(2020) Fiedorowicz et al. (about homeopathy for CFS)

Discussion in 'PsychoSocial ME/CFS Research' started by Woolie, Mar 3, 2021.

  1. Woolie

    Woolie Senior Member

    Messages:
    2,746
    Likes Received:
    18,724
    When is lack of scientific integrity a reason for retracting a paper? A case study.
    Abstract:

    This editorial has just come out in the Journal of Psychosomatic Research. It is a discussion of issues that arose from this 2004 publication:

    This study is a triple-blinded randomised controlled trial of homeopathy for CFS. It might not come as a surprise to you that the primary outcomes of the study were all negative.

    The current discussion is related to one of the co-authors, who stood up in a talk recently and said that she had worked out a cunning plan for discovering whether participants were in the homeopathy or the control group.

    The current editorial explains why the journal did NOT retract the paper (because the results were all negative anyway, amongst other reasons).

    The interesting bit is the issues they chose to comment on (emphasis mine):

    None of us here will miss the irony that these requirements for good studies are simply waived for psychotherapy trials, without anyone even caring about the risk of bias. Silly homeopathy folks - don't they know that all they need to do to get around this is to combine their medicine with a wee chat. Hey presto - blinding problems solved!
     
    Last edited: Mar 3, 2021
    Atle, spinoza577, Barry and 32 others like this.
  2. Wonko

    Wonko Senior Member (Voting Rights)

    Messages:
    4,939
    Likes Received:
    37,516
    Location:
    UK
    The correct answer is 'always'.

    IMO.
     
    Barry, JemPD, Invisible Woman and 8 others like this.
  3. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,099
    Likes Received:
    97,274
    But perhaps not if the lack of scientific integrity consisted of pretending to have lack of scientific integrity in order to have one's own paper retracted because the result caused a problem for one's practice income stream?
     
  4. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,099
    Likes Received:
    97,274
    Thanks for flagging this @Woolie. This is a nice ironic illustration of the core problem the GRADE enthusiasts and Cochrane reviewers seem to find hard to understand. Perhaps I could have cited this and Hrobjartsson in my testimony to NICE.

    The citation of Hrobjartsson is interesting, though, for the way it gives the impression that you can readily put numbers on the problem. The first thing that is puzzling is that Hrob. measures the problem in terms of odds ratios. Looking at the paper this is because they decided to consider results in terms of binary good or bad outcomes. Clearly that would not apply to a result based on a comparison using a graded scale. The is no odds ratio for PACE as far asI can work out (unless you use 'recovery'). So the figure of 36% does not tell us much, if anything.

    I rather suspect this is statisticians extracting the wrong number. What we want to know is the chance that using unblinded assessors leads to a clinically significant difference in outcomes. And of course that cannot be assessed from sampling the very few trials available that used both blinded and unblinded assessors. These turn out to be trials of rather esoteric things like retinopathy. There would be no way to extrapolate them to ME/CFS or anything else much.

    So maybe all Hrobjartsson's studies (there are several) can tell us is that yes there is a problem. Which is fine but we knew that and I think there is also a serious question about whether the studies might have a serious bias of their own.

    Hrob's studies have the very unusual feature of using both blinded and unblinded assessors and making use of this to assess the importance of blinding. This means that the trials were deliberately designed in a frame of mind that took blinding seriously and wanted to analyse its effects. And arguably the whole problem about trials using unblinded assessors and subjective outcomes is that they are set up by people who are not in this frame of mind but rather prefer to sweep the problem of blinding under the carpet. Being an unblinded assessor in a trial assessing the importance of blinding is going to be very different from being an unblinded assessor in a trial that ignores it.
     
  5. strategist

    strategist Senior Member (Voting Rights)

    Messages:
    3,669
    Likes Received:
    37,453
    This is similar to how homeopathy is often delivered. Homeopaths believe that they can figure out the right homeopathic remedy by learning a lot of about the patient. They are an attentive listener who will take time for the patient. This is probably an important reason why homeopathy has survived.

    It's not that different from psychosomatic medicine therapists searching for ways to connect the patient's medically unexplained symptoms to some psychosocial factors.
     
  6. FMMM1

    FMMM1 Senior Member (Voting Rights)

    Messages:
    938
    Likes Received:
    3,775
    It may not be that different from the Hawthorne effect https://en.wikipedia.org/wiki/Hawthorne_effect indeed @Jonathan Edwards advice that he showed empathy to his patients and tried to support them --- OK Jonathan was developing treatments that worked rather than just being interested!
     
    Last edited: Mar 6, 2021
  7. Woolie

    Woolie Senior Member

    Messages:
    2,746
    Likes Received:
    18,724
    Actually the PACE trial's pre-registered primary outcomes were binary (whether fatigue was substantially reduced at trial end point, and also whether physical function substantially improved).

    You're right that although ORs are used as estimates of effect sizes in studies using binary outcomes, they can't be directly compared to the kinds of effect size measures we use with continuous outcomes (like Cohen's d). I think the main point, though. about the Hrobartsson et al finding is that it suggests that outcomes can be very heavily inflated by bias resulting from the combination of non-blinding and subjective outcome measures - that is, not just by a bit, but by a lot.
     
  8. Woolie

    Woolie Senior Member

    Messages:
    2,746
    Likes Received:
    18,724
    No, I think they dug through literature for trials that ran in two phases - blinded and nonblinded.
     
  9. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,099
    Likes Received:
    97,274
    That puzzles me because unblinded and blinded stages in drug development are widespread. If these are the very few that have unblinded and blinded phases within the same trial that still suggests to me that these people were taking blinding rather seriously and that people involved in the trial would be aware that they were in an unblinded or blinded phase. I can see that the original triallists may not have wanted to make a formal estimate of effect on odds ratio themselves but they must have taken blinding seriously to think it was worth repeating the same study with blinding after doing it unblinded (presumably to see if it had an effect)? That would be very different from the cases we are interested in where people firmly took the view that it was not important based on a textbook chapter that conveniently avoided spelling things out clearly! It still seems to me to be a very unrepresentative sample - but my concern about bias here is that Hrob is likely to have significantly underestimated the problem.

    I agree that the value of the paper is in showing significant effects from unblinding. My scepticism is about trying to pull out numbers that then get quoted as indicators (largely because I suspect that the reality is much worse still).
     
    Last edited: Mar 5, 2021
    TrixieStix, FMMM1, Michelle and 3 others like this.
  10. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,326
    Likes Received:
    18,991
    I can only access the abstract--Elsevier and UC are in a dispute.
     
    Peter Trewhitt likes this.
  11. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,326
    Likes Received:
    18,991
    The statement in the editorial about caution regarding subjective outcomes considering that Per Fink, Rosmalen, Sharpe, White etc are on the journal's advisory board. They don't seem to agree with the journal's beliefs about the bias inherent in combining subjective outcomes and non-blindedness.
     
    Mithriel, Trish and Peter Trewhitt like this.
  12. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    6,127
    Likes Received:
    50,284
    Location:
    UK
  13. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    6,856
    Likes Received:
    46,571
    The link is broken because the "]or" has been inadvertently made part of the URL it is trying to go to. Works fine with that removed :).
     
    spinoza577 and FMMM1 like this.
  14. spinoza577

    spinoza577 Senior Member (Voting Rights)

    Messages:
    423
    Likes Received:
    1,103
    Location:
    neet meer för lang süük
    What a pity that it´s ony a case report!
     
    Peter Trewhitt likes this.
  15. Kalliope

    Kalliope Senior Member (Voting Rights)

    Messages:
    4,260
    Likes Received:
    43,299
    Location:
    Norway
    Trial by Error by David Tuller: Psychosomatics Journal Linked to PACE Authors Highlights Bias from Subjective Outcome

     
    Michelle, Woolie, Daisymay and 13 others like this.
  16. Woolie

    Woolie Senior Member

    Messages:
    2,746
    Likes Received:
    18,724
    @dave30th, its my impression that Jess Fiedorowicz is very much trying to clean up the dodgy science in this area and improve the quality of the research. I can't talk specifics in public, but he has been actively working to ensure BPS style papers are receiving rigorous reviews before publication. He's selecting reviewers from a range of perspectives (including those who've written against BPS-style approaches), and really listening to what they say.
     
  17. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,326
    Likes Received:
    18,991
    That would be good. Perhaps he could clean up his advisory board at the same time!
     
  18. Woolie

    Woolie Senior Member

    Messages:
    2,746
    Likes Received:
    18,724
    Maybe that's not within his power?
     
  19. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,326
    Likes Received:
    18,991
    I don't know. But I think I will send him a link to the blog post.
     
    MEMarge and NelliePledge like this.
  20. Grigor

    Grigor Senior Member (Voting Rights)

    Messages:
    317
    Likes Received:
    3,018

Share This Page