First analysis of ‘pre-registered’ studies shows sharp rise in null findings

https://www.nature.com/articles/d41586-018-07118-1

 

So just for memory challenged among us: PACE protocols were pre-registered but not peer-reviewed?

 

Although PACE was pre-registered, they changed the design once the data was collected, so it was no longer as pre-specified, though the authors have claimed as they did not look at the data before the changes it still counts as a pre-specified study.

Being pre-registered does not guarantee there are not subsequent problems, but it does mean people know what studies were planned so that it can be identified how many studies are just shelved because of null results.

I seem to remember there was some element too of peer reviewing at the funding stage for PACE but may be getting confused. PACE was definitely peer reviewed by the Lancett prior to publication because the editor claimed that it was both fast tracked for publication because of its anticipated importance and simultaneously subjected to endless stages of peer reviewing, though it has not been revealed what that actually involved.

There are so many stages that supposedly ensure quality control, but they just seem to increase the opportunities to get it things wrong.

[edited for typos.]

 

Its quite confusing.

Pre-registration means just that you made your protocol available publicly before you collected/analysed the data. You might have actually published the protocol in a journal, or you might have put it on your institutional website, or on one of those archive sites like PsycArXiv, or on ResearchGate or somewhere like that.

Registered reports are a deal some scientific journals now offer. It means you submit your paper, with a full introduction an methods section, but without the results. Then the journal accepts or declines it based on what you say you're going to do. Then when you have your results, you have a guaranteed outlet to publish your findings.

So PACE was pre-registered, but it wasn't submitted as a registered report.

 

According to Sharpe (from a tweet) PACE was fast-tracked because it was pre-registered.

Nevermind they deviated from protocol and switched outcomes to such a degree that this makes it deliberate fraud but that would be an inconvenient fact to acknowledge.

 

PACE was pre-registered - but in my view, pre-registration doesn't count for anything unless you stick to the protocol you pre-registered. Which they didn't.

And I don't think the fact of it had any effect on the Lancet. There's a lot of preregistered trials out there, why didn't they get fast-tracked at the Lancet? No. that's an error or a lie. Or both.

PACE was fast-tracked because everybody in Lancet was behind it and thought it was SO important it needed to be out soon. I would be willing to bet that not a single one of them had read the trial protocol, so they certainly weren't going by that.

 

I'm sure those null results will start getting rare once people have realised they can get away with Crawley's trick of 'pre-registering' half way through your trial, like she did with the SMILE trial.

 

Can't remember who, I think it was an academic actually, but the reply to that tweet from Sharpe was precisely that they didn't follow the protocol anyway so how does pre-registration count here. No response, of course.

And now of course it's public record he will have a hard time to explain. If there's ever any accountability anyway.