One statistical analysis must not rule them all, 2022, Wagenmakers et al

One statistical analysis must not rule them all, 2022, Wagenmakers, Sarafoglou, Aczel, Nature, https://www.nature.com/articles/d41586-022-01332-8

"Any single analysis hides an iceberg of uncertainty. Multi-team analysis can reveal it.

A typical journal article contains the results of only one analysis pipeline, by one set of analysts. Even in the best of circumstances, there is reason to think that judicious alternative analyses would yield different outcomes.

For example, in 2020, the UK Scientific Pandemic Influenza Group on Modelling asked nine teams to calculate the reproduction number R for COVID-19 infections1. The teams chose from an abundance of data (deaths, hospital admissions, testing rates) and modelling approaches. Despite the clarity of the question, the variability of the estimates across teams was considerable (see ‘Nine teams, nine estimates’).

On 8 October 2020, the most optimistic estimate suggested that every 100 people with COVID-19 would infect 115 others, but perhaps as few as 96, the latter figure implying that the pandemic might actually be retreating. By contrast, the most pessimistic estimate had 100 people with COVID-19 infecting 166 others, with an upper bound of 182, indicating a rapid spread. Although the consensus was that the trajectory of disease spread was cause for concern, the uncertainty across the nine teams was considerably larger than the uncertainty within any one team. It informed future work as the pandemic continued."

More at link: https://www.nature.com/articles/d41586-022-01332-8

 

Here is what the authors propose: