Manipulating the Alpha Level Cannot Cure Significance Testing, Trafimow et al, 2018

A long list of authors making this plea,

http://sci-hub.tw/https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00699/full

 

Reducing the p value means being more certain the results are not by chance.

I don't think anyone would advocate "boiling it down" to a binary decision based on p value alone, but maybe some people do this.

Also, I would have thought it goes without saying that inference should be based on strength of experimental design, but again maybe this is novel to some.

I don't see why having a lower p value is deleterious for the progress of science.
Interesting that the signatories are mostly from schools of psychology.
Maybe they should top worrying about p values and concentrate on robust study design?

 

Isn't it because a lower p value means that you will have to have larger cohorts in your study, which then will cost more and require more work. Given how funding is limited, a lower p value will mean there will be less studies performed.

With a p value of 0.05, the positive results in one in twenty studies will be due random chance. So 1 in 20 results will be duff in this way.

However, one could argue that we can live with that, because initial small scale studies are performed in a cheap and cheerful manner, just to see if there might be any effect. If there is an effect, then larger studies (with lower p values) are usually planned to confirm or refute that effect.

 

I can't say that I've really looked into what p values are used in studies; but isn't the point of a large scale study to try to ensure that the positive results from previous smaller studies were not due to random chance? So large scale studies would need to use lower p values, otherwise there is no point in performing them, I would have thought.