Manipulating the Alpha Level Cannot Cure Significance Testing, Trafimow et al, 2018

[Indigophoton]

A long list of authors making this plea,

Abstract
We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science.

Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers.

Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications.

To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.

http://sci-hub.tw/https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00699/full

 

[BurnA]

Reducing the p value means being more certain the results are not by chance.

I don't think anyone would advocate "boiling it down" to a binary decision based on p value alone, but maybe some people do this.

Also, I would have thought it goes without saying that inference should be based on strength of experimental design, but again maybe this is novel to some.

I don't see why having a lower p value is deleterious for the progress of science.
Interesting that the signatories are mostly from schools of psychology.
Maybe they should top worrying about p values and concentrate on robust study design?

 

[Hip]

I don't see why having a lower p value is deleterious for the progress of science.

Isn't it because a lower p value means that you will have to have larger cohorts in your study, which then will cost more and require more work. Given how funding is limited, a lower p value will mean there will be less studies performed.

With a p value of 0.05, the positive results in one in twenty studies will be due random chance. So 1 in 20 results will be duff in this way.

However, one could argue that we can live with that, because initial small scale studies are performed in a cheap and cheerful manner, just to see if there might be any effect. If there is an effect, then larger studies (with lower p values) are usually planned to confirm or refute that effect.

 

[Trish]

However, one could argue that we can live with that, because initial small scale studies are performed in a cheap and cheerful manner, just to see if there might be any effect. If there is an effect, then larger studies (with lower p values) are usually planned to confirm or refute that effect.

That would be OK if that were true, but even large scale studies seem to use 0.05 routinely, especially all those dubious psychological studies that not only use 0.05, but compound the problem by searching all their data for anything that drops below 0.05 and claim they have discovered real effects.

 

[Hip]

That would be OK if that were true, but even large scale studies seem to use 0.05 routinely, especially all those dubious psychological studies that not only use 0.05, but compound the problem by searching all their data for anything that drops below 0.05 and claim they have discovered real effects.

I can't say that I've really looked into what p values are used in studies; but isn't the point of a large scale study to try to ensure that the positive results from previous smaller studies were not due to random chance? So large scale studies would need to use lower p values, otherwise there is no point in performing them, I would have thought.