P-value (p)

How to read it

OpenStax’s Introductory Statistics defines the p-value as “the probability that, if the null hypothesis is true, the results from another randomly selected sample will be as extreme or more extreme as the results obtained from the given sample”. The smaller the p-value, the more unlikely the outcome and the stronger the evidence against the null hypothesis H₀. In OpenStax’s procedure the p-value is compared with a significance level, α, set beforehand: if α is greater than the p-value, H₀ is rejected and the result called significant. OpenStax gives a memory aid for it: “If the p-value is low, the null must go.”

Also searched as: p, p < 0.05, p value, p value meaning, p-value symbol.

What p-value is significant?

By the convention R. A. Fisher set out in 1925, one below 0.05. In Statistical Methods for Research Workers he wrote that the value for which P = .05, “or 1 in 20”, is 1.96 or nearly 2 standard deviations, and “it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not.” NIST’s Engineering Statistics Handbook calls the choice of α “somewhat arbitrary” and gives 0.1, 0.05 and 0.01 as the values commonly used. It also separates statistical from practical significance: with a large enough sample, a test can reject the null hypothesis for a difference too small to matter in engineering terms.

What a p-value doesn’t measure

The American Statistical Association’s statement of March 2016 set out six principles. Two of them: p-values “do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone”, and a p-value “does not measure the size of an effect or the importance of a result”. The statement also said that scientific conclusions “should not be based only on whether a p-value passes a specific threshold”.

The argument over p < 0.05

Statisticians disagree about what should happen to the threshold. In 2019 an editorial in The American Statistician by Ronald Wasserstein, the ASA’s executive director, with Allen Schirm and Nicole Lazar, went further than the 2016 statement: “it is time to stop using the term “statistically significant” entirely”, and variants such as “p < 0.05” should not survive either. Researchers, they wrote, should be free to treat p = 0.051 and p = 0.049 as not categorically different. The ASA’s president then set up a task force, over concerns that the editorial might be mistaken for official ASA policy. Its statement, published in The Annals of Applied Statistics in 2021, says that p-values and significance testing, “properly applied and interpreted, are important tools that should not be abandoned”.

Look-alikes

The p of a p-value is a different p from the population proportion p and the sample proportion p̂, though they appear in the same chapters: one of OpenStax’s tests is H₀: p ≤ 0.066, about a proportion, and is decided with a p-value.