NEJM Evidence: A New Look at P Values for Randomised Clinical Trials, Zwet et al, 2023

Abstract

BACKGROUND
We have examined the primary efficacy results of 23,551 randomized clinical trials from the Cochrane Database of Systematic Reviews.

METHODS
We estimate that the great majority of trials have much lower statistical power for actual effects than the 80 or 90% for the stated effect sizes. Consequently, “statistically significant” estimates tend to seriously overestimate actual treatment effects, “nonsignificant” results often correspond to important effects, and efforts to replicate often fail to achieve “significance” and may even appear to contradict initial results. To address these issues, we reinterpret the P value in terms of a reference population of studies that are, or could have been, in the Cochrane Database.

RESULTS
This leads to an empirical guide for the interpretation of an observed P value from a “typical” clinical trial in terms of the degree of overestimation of the reported effect, the probability of the effect’s sign being wrong, and the predictive power of the trial.

CONCLUSIONS
Such an interpretation provides additional insight about the effect under study and can guard medical researchers against naive interpretations of the P value and overoptimistic effect sizes. Because many research fields suffer from low power, our results are also relevant outside the medical domain. (Funded by the U.S. Office of Naval Research.)

https://evidence.nejm.org/doi/abs/10.1056/EVIDoa2300003

 

Oh but that's Cochrane's entire business model and the bread-and-butter of evidence-based medicine.

Who knew that industrializing a GIGO model would lead to a lot of garbage? Everyone? Yeah, literally everyone. Well, except the medical profession, I guess. Everyone else, though? Yeah, duh.

 

This is not surprising given the publish-or-perish culture of academia, medical/bio science community in particular. Theo Baker did an expose on this: Opinion | The Story of the Student Journalist and the Stanford President - The New York Times (nytimes.com)