1. Sign our petition calling on Cochrane to withdraw their review of Exercise Therapy for CFS here.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 18th March 2024 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Outcome Reporting bias in Exercise Oncology trials (OREO): a cross-sectional study, 2021, Singh, Twomey et al

Discussion in 'Research methodology news and research' started by rvallee, Mar 15, 2021.

  1. Hutan

    Hutan Moderator Staff Member

    Messages:
    26,534
    Location:
    Aotearoa New Zealand
    Yes, regardless of the statistics, it's an approach that should be used a lot more when choosing which research proposals to allocate funds to.

    I recall one proposal to trial a mixed supplement for ME/CFS. Some of the evaluators thought that it would be a nice project as it might produce a treatment - and it would be blinded, so surely there would be no bias.

    The researchers had done a previous study on the same supplement mix for a different disease. There was a peer review that could be found online asking them to report that there was a null primary outcome in the abstract - they had neglected to do that in the pre-publication version. In the final version the null result was mentioned in passing, along with the widest array of excuses I had ever seen in a report; proposals for tweaking the mixture, suggestions that the length of study was not long enough. They even blamed their trial participants, suggesting that only the most treatment resistant patients turned up for trials and that substance abuse by the participants (remember this was a blinded, controlled study) had skewed results.

    The proposed trial had the same supplement mix, and the same trial length, just a different disease, and a similar hand wavey 'inflammation' sort of explanation. It was clear that even a null result in the proposed trial would not be assumed to be evidence against a particular component of the mix, or a different dosage or recipe, and there was a significant risk that even a null result would be spun as a success.

    It's easy to think of some pretty egregious research where the researchers just keep getting funded to produce more, often quite harmful and certainly wasteful, rubbish.

    If people tasked with the job of allocating funds to research took their job seriously (and most often I think these roles are done voluntarily, squeezed in between other 'more important' things - perhaps they should be paid to devote time to it), then they would research track records. They would get opinions from knowledgeable patient organisations. And then they would not give more funds to researchers who switch outcomes without declaring it, or don't report results honestly. If that happened more, then researchers would either adapt, or go and do something else. There needs to be consequences.
     
  2. ME/CFS Skeptic

    ME/CFS Skeptic Senior Member (Voting Rights)

    Messages:
    3,495
    Location:
    Belgium
    I think one could do the same with p-values: normally those just below 0.05 shouldn't be that common as seen in most papers, even if there was an effect.

    I doubt, however, that one can differentiate between data manipulation and reporting bias (or things like optional stopping).
     
  3. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    12,299
    Location:
    Canada
    And sometimes, just sometimes, the authors will literally plainly admit, using those very words, that they cheated to get the outcome they prefer and no one will do a damn thing about it. And frankly I think that's the biggest problem, that the self-correction has been effectively broken entirely where interests are opposed to reality. It's one thing to be able to push through something that shouldn't, but being caught and getting away with it is the main problem, the facts aren't even open to debate and yet they actually are subject to debate, a debate in which the main stakeholders are explicitly maligned and shut off.

    A big problem I'm seeing, in medicine in general but especially for EBM, is that stakeholders are often not involved, only some and too often biased ones. That can't work, it has to be a rule, an immutable standard, that stakeholders be involved according to their, well, stake, and its relative importance. And of course for patients it is literally our lives, while for the people pushing their nonsense it's all about career advancement and what can only be described as a jobs program for people who should never have been granted a medical license, they have no real stake in the matter and yet control the entire process. They don't even care about the outcomes, and that makes them systematically reckless.
     
    MEMarge, Peter Trewhitt and Hutan like this.
  4. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,274
    Location:
    London, UK
    My understanding is that the inverted funnel plot more or less does this but actually is more powerful because it makes use of theory of variance.

    The amusing thing about the telepathy studies was that there were fewer trials showing an obvious positive response than there should have been by chance. All the results came out exactly as only slightly positive as the study being replicated. I think you can probably show that this has to involve manipulation some sort.
     
  5. Arnie Pye

    Arnie Pye Senior Member (Voting Rights)

    Messages:
    6,048
    Location:
    UK
    It's very profitable?
     
    FMMM1, Michelle, Trish and 2 others like this.
  6. JohnTheJack

    JohnTheJack Moderator Staff Member

    Messages:
    4,350
    Thanks, Jo.

    It's been a tough 12 months, but I am slowly improving and am just dipping my toe in here and reading a few posts.

    I'll not be active for a while.

    Thanks all for the good wishes.

    (And please no replies or I'll get into trouble with the other mods for hijacking a thread. Hehe.)
     
  7. Joan Crawford

    Joan Crawford Senior Member (Voting Rights)

    Messages:
    547
    Location:
    Warton, Carnforth, Lancs, UK
    I don't know the answer to that, tbh.

    There has been much gained in terms of prestige, honours, knighthoods, professorships and so forth by those writing the WS type research material. And journals get to fill their pages.
     
    NelliePledge, Arnie Pye and Trish like this.

Share This Page