1. Guest, the 'News in Brief' for the week beginning 22nd February 2021 is here.
    Dismiss Notice
  2. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

First analysis of ‘pre-registered’ studies shows sharp rise in null findings

Discussion in 'Health News and Research unrelated to ME/CFS' started by Andy, Oct 25, 2018.

  1. Andy

    Andy Committee Member & Outreach

    Messages:
    12,092
    Likes Received:
    86,966
    Location:
    Hampshire, UK
    https://www.nature.com/articles/d41586-018-07118-1
     
    Manganus, TiredSam, Trish and 20 others like this.
  2. Snowdrop

    Snowdrop Senior Member (Voting Rights)

    Messages:
    1,687
    Likes Received:
    11,448
    So just for memory challenged among us: PACE protocols were pre-registered but not peer-reviewed?
     
    Inara, ladycatlover, MEMarge and 2 others like this.
  3. Peter Trewhitt

    Peter Trewhitt Senior Member (Voting Rights)

    Messages:
    1,331
    Likes Received:
    13,277
    Although PACE was pre-registered, they changed the design once the data was collected, so it was no longer as pre-specified, though the authors have claimed as they did not look at the data before the changes it still counts as a pre-specified study.

    Being pre-registered does not guarantee there are not subsequent problems, but it does mean people know what studies were planned so that it can be identified how many studies are just shelved because of null results.

    I seem to remember there was some element too of peer reviewing at the funding stage for PACE but may be getting confused. PACE was definitely peer reviewed by the Lancett prior to publication because the editor claimed that it was both fast tracked for publication because of its anticipated importance and simultaneously subjected to endless stages of peer reviewing, though it has not been revealed what that actually involved.

    There are so many stages that supposedly ensure quality control, but they just seem to increase the opportunities to get it things wrong.

    [edited for typos.]
     
    Last edited: Oct 25, 2018
  4. Woolie

    Woolie Senior Member

    Messages:
    2,649
    Likes Received:
    17,761
    Its quite confusing.

    Pre-registration means just that you made your protocol available publicly before you collected/analysed the data. You might have actually published the protocol in a journal, or you might have put it on your institutional website, or on one of those archive sites like PsycArXiv, or on ResearchGate or somewhere like that.

    Registered reports are a deal some scientific journals now offer. It means you submit your paper, with a full introduction an methods section, but without the results. Then the journal accepts or declines it based on what you say you're going to do. Then when you have your results, you have a guaranteed outlet to publish your findings.

    So PACE was pre-registered, but it wasn't submitted as a registered report.
     
  5. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    5,723
    Likes Received:
    45,973
    Location:
    Canada
    According to Sharpe (from a tweet) PACE was fast-tracked because it was pre-registered.

    Nevermind they deviated from protocol and switched outcomes to such a degree that this makes it deliberate fraud but that would be an inconvenient fact to acknowledge.
     
    andypants, Inara, Trish and 3 others like this.
  6. Woolie

    Woolie Senior Member

    Messages:
    2,649
    Likes Received:
    17,761
    PACE was pre-registered - but in my view, pre-registration doesn't count for anything unless you stick to the protocol you pre-registered. Which they didn't.

    And I don't think the fact of it had any effect on the Lancet. There's a lot of preregistered trials out there, why didn't they get fast-tracked at the Lancet? No. that's an error or a lie. Or both.

    PACE was fast-tracked because everybody in Lancet was behind it and thought it was SO important it needed to be out soon. I would be willing to bet that not a single one of them had read the trial protocol, so they certainly weren't going by that.
     
    andypants, Inara, rvallee and 7 others like this.
  7. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    3,934
    Likes Received:
    32,078
    I'm sure those null results will start getting rare once people have realised they can get away with Crawley's trick of 'pre-registering' half way through your trial, like she did with the SMILE trial.
     
  8. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    5,723
    Likes Received:
    45,973
    Location:
    Canada
    Can't remember who, I think it was an academic actually, but the reply to that tweet from Sharpe was precisely that they didn't follow the protocol anyway so how does pre-registration count here. No response, of course.

    And now of course it's public record he will have a hard time to explain. If there's ever any accountability anyway.
     

Share This Page