Manipulating the Alpha Level Cannot Cure Significance Testing, Trafimow et al, 2018

Discussion in 'Research methodology news and research' started by Indigophoton, Jun 7, 2018.

  1. Indigophoton

    Indigophoton Senior Member (Voting Rights)

    Messages:
    849
    Location:
    UK
    A long list of authors making this plea,
    http://sci-hub.tw/https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00699/full
     
    Hutan, Sean, alktipping and 2 others like this.
  2. BurnA

    BurnA Senior Member (Voting Rights)

    Messages:
    410
    Reducing the p value means being more certain the results are not by chance.

    I don't think anyone would advocate "boiling it down" to a binary decision based on p value alone, but maybe some people do this.

    Also, I would have thought it goes without saying that inference should be based on strength of experimental design, but again maybe this is novel to some.

    I don't see why having a lower p value is deleterious for the progress of science.
    Interesting that the signatories are mostly from schools of psychology.
    Maybe they should top worrying about p values and concentrate on robust study design?
     
  3. Hip

    Hip Senior Member (Voting Rights)

    Messages:
    726
    Isn't it because a lower p value means that you will have to have larger cohorts in your study, which then will cost more and require more work. Given how funding is limited, a lower p value will mean there will be less studies performed.

    With a p value of 0.05, the positive results in one in twenty studies will be due random chance. So 1 in 20 results will be duff in this way.

    However, one could argue that we can live with that, because initial small scale studies are performed in a cheap and cheerful manner, just to see if there might be any effect. If there is an effect, then larger studies (with lower p values) are usually planned to confirm or refute that effect.
     
  4. Trish

    Trish Moderator Staff Member

    Messages:
    54,723
    Location:
    UK
    That would be OK if that were true, but even large scale studies seem to use 0.05 routinely, especially all those dubious psychological studies that not only use 0.05, but compound the problem by searching all their data for anything that drops below 0.05 and claim they have discovered real effects.
     
    BurnA likes this.
  5. Hip

    Hip Senior Member (Voting Rights)

    Messages:
    726
    I can't say that I've really looked into what p values are used in studies; but isn't the point of a large scale study to try to ensure that the positive results from previous smaller studies were not due to random chance? So large scale studies would need to use lower p values, otherwise there is no point in performing them, I would have thought.
     
    Last edited: Jun 10, 2018

Share This Page