1. Guest, the 'News in Brief' for the week beginning 23rd January 2023 is here.
    Dismiss Notice
  2. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Large P-Values Cannot be Explained by Power Analysis, André, 2022

Discussion in 'Research methodology news and research' started by cassava7, Aug 19, 2022.

  1. cassava7

    cassava7 Senior Member (Voting Rights)

    Messages:
    792
    Excerpt:

    Could the p-values in a paper be large because the researchers have conducted a careful power analysis?

    The argument goes like this:

    1. Researchers don’t want to be wasteful with their resources, so they do their best to predict the effect size that their experiment will yield.
    2. Once they know this effect size, they perform a power analysis to determine how many observations they should collect.
    3. Because they run exactly the number of participants they need (not more, not less) to detect a significant effect, they will tend to find p-values that are close to .05.
    Another flavor of this argument is “small p-values means that researchers were wasteful, collected too many observations, and made their effect too significant”.

    Are there any merits to this argument? Could large p-values be explained by thoughtful researchers collecting just enough participants to detect a carefully-estimated effect size? Let’s dive in!

    […]

    Conclusion

    Researchers cannot “aim” for p = .05, not even with a careful, perfectly accurate, power analysis.

    In fact, the opposite is true: If their power analysis is accurate (i.e., gives them a good chance to detect their effect), they should get p-values that are much smaller, rather than close, to .05.

    The most likely explanation for a set of multiple p-values hugging .05 is low statistical power, and/or p-hacking.

    Blog post: https://quentinandre.net/post/large-pvalues-and-power-analysis/

    https://twitter.com/user/status/1560395816283516928
     
    Last edited: Aug 19, 2022
    Lisa108, Sean, Hutan and 6 others like this.

Share This Page