A p-value is one of the most quoted numbers in science and one of the most consistently misread. It does not tell you the probability that your hypothesis is true, it does not tell you the probability that your result happened by chance, and a small one does not mean the effect is large. This guide covers what the number actually is, how to compute it from five different test statistics, and the five misreadings that account for most of the confusion.
A p-value is the probability of observing data at least as extreme as your sample, assuming the null hypothesis is true. It is a conditional probability about data, not a probability about hypotheses. Formally, for a test statistic S with observed value x, the two-tailed p-value is 2 × min{ Pr(S ≤ x | H₀), Pr(S ≥ x | H₀) }, or equivalently the area under the null distribution in both tails beyond the observed value.
To compute it you need three things: the test statistic, its distribution under the null hypothesis, and whether the alternative is one-tailed or two-tailed. Five statistics cover almost every case — Z (standard normal), t (Student's t, requires degrees of freedom), F (two degrees of freedom, right tail only), χ² (chi-square, right tail only), and r (correlation, converted to a t statistic with n − 2 degrees of freedom).
What a p-value is, exactly
The definition is one sentence long and every word in it matters. The American Statistical Association's 2016 statement puts it this way: a p-value indicates how incompatible the data are with a specified statistical model — specifically, the null hypothesis.
It is a conditional probability. The conditioning is the crucial part. You assume the null hypothesis is true, and then you ask: how surprising would data like mine be? A small p-value means "quite surprising." It does not mean "the null is false," because a surprising outcome is still possible under a true null.
Three properties follow directly from that definition and are worth stating explicitly, because each one is routinely violated in interpretation.
- It is a property of the data given the model. It says nothing about the probability that the model is correct.
P(data | H₀)andP(H₀ | data)are different quantities, and confusing them is the inverse-probability fallacy. - It is uniformly distributed under a true null. If the null is exactly true, p-values are uniform on [0, 1] — so 5% of studies will produce p < 0.05 by construction, not because anything interesting happened.
- It depends on the alternative hypothesis. "At least as extreme" has no meaning until you specify which direction counts as extreme. That is why one-tailed and two-tailed tests produce different p-values from the same data.
A p-value is the probability of obtaining a test statistic at least as extreme as the observed one, calculated under the assumption that the null hypothesis is true. It is a conditional probability about the data, not a probability that the null hypothesis is true, and it is not the probability that the result occurred by chance.
The null distribution and the tail area
Every p-value calculation has the same three-step structure, regardless of which statistic is involved. Once you see the pattern, the five formulas below become variations on one procedure.
- Compute the test statistic from your data. This is a single number that summarises how far the data sit from what the null hypothesis predicts.
- Choose the distribution that statistic would follow if the null were true. Z uses the standard normal, t uses Student's t with a specific degrees of freedom, F uses the F distribution with two degrees of freedom, χ² uses chi-square, and r is converted to a t statistic first.
- Find the probability of a value at least as extreme. That probability is the area under the density curve beyond the observed value, in the direction or directions specified by the alternative hypothesis.
| Step | What you need | Where it comes from | Common error |
|---|---|---|---|
| 1. Statistic | Z, t, F, χ² or r | The test output or a hand calculation | Using the wrong statistic for the design |
| 2. Distribution | The null distribution | Determined by the statistic and, for t/F/χ², the degrees of freedom | Omitting or miscounting degrees of freedom |
| 3. Tail area | One-tailed or two-tailed | The alternative hypothesis, stated before looking at the data | Choosing the tail after seeing the result |
Steps 2 and 3 are where most calculator errors occur. The statistic is usually produced by software; the degrees of freedom and the tail choice are usually entered by hand, and both change the answer.
The shape of the null distribution matters more than it might seem. The standard normal is symmetric, so a two-tailed p-value is exactly twice the one-tailed value. The t distribution is also symmetric but has heavier tails, which is why the same numeric statistic produces a larger p-value from a t distribution with small degrees of freedom than from a normal. The F and chi-square distributions are not symmetric — both are bounded below at zero and have a long right tail — so their p-values are always right-tail areas and the one-tailed/two-tailed distinction does not apply in the same way.
Z score and the normal distribution
A Z score expresses how many standard deviations an observation sits from the mean of a standard normal distribution, which has mean 0 and standard deviation 1. It is the appropriate statistic when the population standard deviation is known, or when the sample is large enough that the sample standard deviation is a reliable substitute.
Φ is the cumulative distribution function of the standard normal. Because the distribution is symmetric, the sign of Z does not affect the two-tailed p-value — a Z of −2.5 and a Z of +2.5 produce the same result. This is worth knowing because some software reports negative Z values for left-tailed tests and users occasionally assume the p-value must be negative too. It cannot be: p-values are probabilities and lie between 0 and 1.
| Z | One-tailed p | Two-tailed p | Conventional reading |
|---|---|---|---|
| 1.645 | 0.050 | 0.100 | Boundary at α = 0.05, one-tailed only |
| 1.960 | 0.025 | 0.050 | The familiar 0.05 two-tailed threshold |
| 2.326 | 0.010 | 0.020 | — |
| 2.576 | 0.005 | 0.010 | The familiar 0.01 two-tailed threshold |
| 3.291 | 0.0005 | 0.001 | The familiar 0.001 two-tailed threshold |
The first row is the source of a common confusion. A one-tailed test at α = 0.05 uses a critical Z of 1.645, while a two-tailed test at the same α uses 1.960. Switching from two-tailed to one-tailed without saying so makes a result look more significant than it is.
Z = 2.10, two-tailed
Z = 2.10
The upper-tail area beyond 2.10 is 0.01786. Doubling it for the two-tailed test gives 0.03572. Since 0.0357 < 0.05, the result is conventionally described as statistically significant at the 5% level.
Answer: p = 0.0357
T score and degrees of freedom
A t statistic has the same structure as a Z statistic but replaces the known population standard deviation with the sample standard deviation. That substitution introduces extra uncertainty, which is why the t distribution has heavier tails than the normal — and why the same numeric value produces a larger p-value from a t distribution.
The degrees of freedom are not a formality. They determine the exact shape of the null distribution, and getting them wrong changes the p-value. For a one-sample t-test, df = n − 1. For an independent two-sample test, df depends on whether equal variances are assumed; the Welch correction produces a fractional df. For a paired test, df = n − 1 where n is the number of pairs.
| |t| | df = 3 | df = 10 | df = 30 | df = ∞ (Z) |
|---|---|---|---|---|
| 2.0 | 0.1393 | 0.0734 | 0.0546 | 0.0455 |
| 2.5 | 0.0876 | 0.0313 | 0.0182 | 0.0124 |
| 3.0 | 0.0577 | 0.0132 | 0.0055 | 0.0027 |
A t of 2.0 is significant at the 5% level with 30 degrees of freedom (p = 0.0546, marginally not significant) but clearly significant with a large sample. With 3 degrees of freedom the same t is not close to significant. The degrees of freedom are not a technicality; they are the difference between a result and a non-result.
t = 2.45, df = 18, two-tailed
t = 2.45 · df = 18
The upper-tail area beyond 2.45 with 18 degrees of freedom is 0.0123. Doubling gives 0.0246. Since 0.0246 < 0.05, the result is significant at the 5% level, but not at the 1% level.
Answer: p = 0.0246
F statistic and ANOVA
The F statistic is a ratio of two variances, and it is used whenever you are comparing more than two group means — the analysis of variance, or ANOVA. Unlike Z and t, it has two degrees of freedom: one for the numerator and one for the denominator, and their order matters.
Two properties distinguish the F test from the others. First, the F distribution is not symmetric — it is bounded below at zero and has a long right tail — so the p-value is always a right-tail area. A large F means the between-group variance is large relative to the within-group variance, which is evidence against the null of equal means. Second, the order of the degrees of freedom matters. Entering df₁ and df₂ in the wrong order produces a different p-value, and some calculators do not label the fields clearly enough to prevent that mistake.
| F | df₁ = 2, df₂ = 27 | df₁ = 3, df₂ = 20 | df₁ = 1, df₂ = 10 |
|---|---|---|---|
| 1.0 | 0.380 | 0.410 | 0.341 |
| 2.0 | 0.155 | 0.146 | 0.185 |
| 3.0 | 0.063 | 0.050 | 0.109 |
| 4.0 | 0.031 | 0.022 | 0.074 |
The same F value produces different p-values depending on the degrees of freedom. An F of 3.0 is significant with df₁ = 3, df₂ = 20 (p = 0.050) but not with df₁ = 2, df₂ = 27 (p = 0.063) or with df₁ = 1, df₂ = 10 (p = 0.109).
Reversing the two degrees of freedom. On a one-way ANOVA with k = 4 groups and N = 40 observations, the correct entries are df₁ = 3 (numerator) and df₂ = 36 (denominator). Entering 36 and 3 instead produces a p-value that is not just slightly different — it can be an order of magnitude off. Always check which df the calculator labels as numerator and which as denominator.
Chi-square and categorical data
The chi-square statistic is used when the data are counts in categories rather than means or variances. It compares the observed counts in each cell with the counts expected under the null hypothesis of independence or goodness of fit.
Like the F distribution, chi-square is asymmetric and bounded below at zero, so its p-value is always a right-tail area. The degrees of freedom depend on the design: for a goodness-of-fit test with k categories, df = k − 1; for a contingency table with r rows and c columns, df = (r − 1)(c − 1).
The statistic is the sum of squared deviations between observed and expected counts, each divided by the expected count. A large χ² means the observed counts differ substantially from what independence would predict, which is evidence against the null.
| χ² | df = 1 | df = 2 | df = 5 | df = 10 |
|---|---|---|---|---|
| 3.84 | 0.050 | 0.147 | 0.571 | 0.952 |
| 5.99 | 0.014 | 0.050 | 0.307 | 0.817 |
| 9.49 | 0.002 | 0.009 | 0.091 | 0.486 |
| 15.51 | < 0.001 | < 0.001 | 0.008 | 0.116 |
The critical value for significance depends entirely on the degrees of freedom. χ² = 3.84 is the 5% cutoff with df = 1, but it is nowhere near significant with df = 5 (p = 0.571). Entering the wrong df turns a significant result into a non-significant one and vice versa.
χ² = 6.25, df = 2
χ² = 6.25 · df = 2
The right-tail area beyond 6.25 with 2 degrees of freedom is 0.0439. Since 0.0439 < 0.05, the result is significant at the 5% level.
Answer: p = 0.0439
Correlation coefficient r
The correlation coefficient r measures the strength and direction of a linear relationship between two variables. Testing whether it differs from zero requires converting it to a t statistic, because r itself does not follow a distribution with a known closed form under the null.
Two details matter. The degrees of freedom are n − 2, not n − 1, because two parameters have been estimated (the slope and the intercept of the regression line). And the sign of r is irrelevant for the two-tailed p-value, because the t statistic is squared in the tail calculation — r = 0.4 and r = −0.4 produce the same p-value.
The practical consequence is that the same r can be significant or not depending on the sample size. An r of 0.30 with n = 10 is not significant (p ≈ 0.40); the same r with n = 100 is highly significant (p ≈ 0.002). This is the clearest illustration of why a p-value is not an effect size.
| r | n = 10 (df = 8) | n = 30 (df = 28) | n = 100 (df = 98) | n = 500 (df = 498) |
|---|---|---|---|---|
| 0.20 | 0.579 | 0.289 | 0.046 | < 0.001 |
| 0.30 | 0.400 | 0.107 | 0.002 | < 0.001 |
| 0.50 | 0.141 | 0.005 | < 0.001 | < 0.001 |
The same correlation of 0.20 is nowhere near significant with 10 observations and highly significant with 500. The relationship is identical; only the sample size changed. A p-value that ignores the effect size will always mislead when the sample is large.
r = 0.45, n = 25
r = 0.45 · n = 25 · df = 23
t = 0.45 × √23 / √(1 − 0.2025) = 0.45 × 4.796 / 0.893 = 2.417. The two-tailed area from t with 23 df beyond 2.417 is 0.0239.
Answer: p = 0.0239
Which statistic should I use?
The choice of statistic is determined by the design of the study, not by preference. The decision tree below covers the five cases that account for almost all p-value calculations.
| Situation | Statistic | Degrees of freedom | Tail |
|---|---|---|---|
| One mean, σ known or n large | Z | none | One or two |
| One mean, σ estimated | t | n − 1 | One or two |
| Two independent means | t | n₁ + n₂ − 2 (or Welch) | One or two |
| Paired means | t | n − 1 pairs | One or two |
| Three or more means | F | k − 1 and N − k | Right only |
| Counts, goodness of fit | χ² | k − 1 | Right only |
| Counts, independence | χ² | (r − 1)(c − 1) | Right only |
| Correlation | r → t | n − 2 | One or two |
Z is used when the population standard deviation is known or the sample is large; t when it is estimated and the sample is small; F for comparing three or more group means in an ANOVA, with two degrees of freedom and a right-tail p-value; chi-square for categorical counts, also right-tailed; and r is converted to a t statistic with n − 2 degrees of freedom before its p-value is computed.
One-tailed versus two-tailed
The tail choice is not a technical detail. It determines whether "extreme" means "far in either direction" or "far in one specified direction," and it changes the p-value by a factor of two for symmetric distributions.
| Alternative hypothesis | Tail | p-value formula | When to use |
|---|---|---|---|
| μ ≠ μ₀ | Two-tailed | 2 × min{left tail, right tail} | When a difference in either direction matters |
| μ > μ₀ | Right-tailed | right tail area | When only an increase would be meaningful |
| μ < μ₀ | Left-tailed | left tail area | When only a decrease would be meaningful |
The two-tailed test is the default in most fields because a difference in the unexpected direction is still a difference worth knowing about. A one-tailed test doubles the power to detect an effect in the specified direction but gives zero power to detect one in the opposite direction — which is why it must be specified before the data are seen, not after.
For the F and chi-square distributions the distinction collapses, because those distributions are asymmetric and bounded at zero. Their p-values are always right-tail areas, and "two-tailed" is not a meaningful option.
If you look at the result first and then decide whether to run a one-tailed or two-tailed test, you have doubled the effective Type I error rate. A two-tailed p of 0.08 becomes a one-tailed p of 0.04 if you pick the direction that matches your data — but the decision to pick that direction was informed by the data, so the 0.04 is not a valid probability. This is one of the questionable research practices behind the replication crisis described in §12.
What p-values cannot tell you
Five statements about p-values are false but widely believed. Each one is a different version of the same error: treating a conditional probability about data as if it were a probability about hypotheses or about the size of an effect.
| False statement | Why it is wrong | What the p-value actually says |
|---|---|---|
| "p = 0.03 means there is a 3% chance the null is true." | It is P(data | H₀), not P(H₀ | data) | The data are somewhat unlikely if the null holds |
| "p = 0.03 means there is a 3% chance the result was due to chance." | Chance is not a model; the null is | Only about incompatibility with a specified model |
| "A small p means a large effect." | p depends on effect size and sample size | Nothing about magnitude without an effect-size estimate |
| "A large p proves the null is true." | Absence of evidence is not evidence of absence | The data are not incompatible with the null, which may still be false |
| "p < 0.05 means the result is important." | Statistical significance is not practical significance | Only that the result is unlikely under the null |
The first two are the most persistent. They arise because P(A | B) and P(B | A) are easily confused, and the p-value is a particularly tempting case of that confusion. If 1% of tested hypotheses are true and you test 1,000 hypotheses, you expect about 10 true effects and 990 nulls. At α = 0.05 you will reject about 50 of the nulls by chance and about 8 of the true effects (assuming 80% power). So of the 58 "significant" results, roughly 50 — about 86% — are false positives even though every individual p-value is below 0.05.
That calculation is the false discovery rate, and it is the number that actually matters when many tests are run. A p-value of 0.03 does not mean a 3% chance of being wrong; in a low-prevalence setting it can mean a much higher chance.
A p-value is not the probability that the null hypothesis is true, not the probability that the result occurred by chance, and not a measure of effect size. A small p-value indicates incompatibility between the data and the null model; it does not establish that the effect is large, important, or even real.
Sample size and practical significance
The p-value is a function of two quantities: the size of the effect and the size of the sample. A large enough sample will make any non-zero effect statistically significant, and a small enough sample will make any effect non-significant. This is not a flaw in the p-value; it is a property of how it is constructed. But it means that a p-value in isolation cannot tell you whether an effect matters.
A 0.2 standard-deviation difference is small — about the difference in height between two people separated by one inch. With 20 observations it produces p ≈ 0.65, which is indistinguishable from noise. With 6,400 observations it produces p ≈ 0.006, which is comfortably significant. The effect did not change; the sample did.
The converse is also true. A study with 15 participants that reports p = 0.06 has not shown that an effect is absent. It has shown that the study was not precise enough to detect one of that size. The confidence interval is the honest way to report this: it shows the range of effect sizes compatible with the data, rather than a binary significant/not-significant verdict.
An effect size (Cohen's d, odds ratio, risk difference, or a regression coefficient), a confidence interval for that effect, and the sample size. A p-value answers a yes/no question about compatibility with a model; the effect size and interval answer the question a reader actually has, which is "how big is this and how precisely do we know it."
The ASA statement and the replication problem
In March 2016 the American Statistical Association issued a statement on p-values that is unusual for its directness. It set out six principles, several of which exist specifically to correct the misreadings described in §10.
| ASA principle | What it says | Which misconception it targets |
|---|---|---|
| 1 | P-values can indicate how incompatible the data are with a specified statistical model | That a p-value measures the probability of the hypothesis |
| 2 | P-values do not measure the probability that the studied hypothesis is true, or that the data were produced by random chance alone | The inverse-probability fallacy and the "due to chance" phrasing |
| 3 | Scientific conclusions should not be based only on whether a p-value passes a specific threshold | Dichotomania — treating 0.049 and 0.051 as different in kind |
| 4 | Proper inference requires full reporting and transparency | Selective reporting and p-hacking |
| 5 | A p-value does not measure the size of an effect or the importance of a result | Confusing statistical with practical significance |
| 6 | By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis | Treating a single number as a verdict |
The six principles are quoted or paraphrased from the ASA statement (Wasserstein & Lazar, 2016). They are not a rejection of p-values — the statement explicitly calls the p-value a useful statistical measure — but a correction of how it is used and reported.
The replication crisis gave those principles urgency. Large-scale replication projects have found that a substantial fraction of published significant results do not replicate. One frequently cited figure is that only about 24% of psychology studies reporting p ≈ 0.05 could be confirmed by further experiments, although that number depends heavily on the subfield and the replication criteria.
Three mechanisms drive the gap, and all three are connected to p-values:
- Publication bias. Significant results are more likely to be published, so the literature over-represents them.
- P-hacking. Trying multiple analyses and reporting the one that crosses 0.05 inflates the false positive rate. If you test two conditions after each new observation, the false positive rate can reach 22% despite a nominal 5% threshold.
- Low power. Underpowered studies that do find a significant result tend to overestimate the effect size, a phenomenon known as the winner's curse.
The proposed remedies are not "stop using p-values." They are to report effect sizes and confidence intervals, to pre-register analyses, to correct for multiple comparisons, and to stop treating 0.05 as a bright line between truth and falsehood.
0.05 is a convention, not a law of nature. It was popularised by Fisher as a convenient threshold, not as a universal decision rule. A p of 0.049 and a p of 0.051 are not meaningfully different, yet one is routinely published as a finding and the other as a null result. The ASA statement's third principle exists precisely to discourage that dichotomy.
Common mistakes and their cost
Each error below changes the p-value in a predictable direction, which makes it recognisable in a result that looks wrong.
| Mistake | What it does | Direction | Approximate cost |
|---|---|---|---|
| Using the wrong degrees of freedom | Changes the null distribution | Either direction, often large | 0.01 → 0.10 or worse |
| Reversing F df₁ and df₂ | Uses the wrong F distribution | Often too small | Can be an order of magnitude |
| Choosing one-tailed after seeing the data | Halves the p-value artificially | Too small | p → p/2 |
| Entering a negative t or Z as positive | No effect on two-tailed p | None | 0 |
| Using n instead of n − 2 for r | Too many degrees of freedom | Too small | Modest, larger at small n |
| Using a one-sample df for a two-sample test | Wrong df | Usually too small | Can flip significance |
| Reporting p = 0.000 | Impossible; p is never exactly zero | Misleading precision | Report p < 0.001 |
| Treating p > 0.05 as proof of no effect | Confuses non-significance with equivalence | False negative claim | Substantial |
The first three are the ones that change a conclusion. The others affect precision or interpretation but rarely flip a result from significant to non-significant on their own.
(1) Confirm which statistic the test actually produced. (2) Confirm the degrees of freedom against the design — n − 1 for one sample, n₁ + n₂ − 2 for two, k − 1 and N − k for ANOVA, (r − 1)(c − 1) for a contingency table, n − 2 for correlation. (3) Confirm the tail choice was specified before the data were seen. (4) Check that the reported p is a probability between 0 and 1 — if it is outside that range, the statistic or the df is wrong.
Critical values at a glance
The statistic value that corresponds to p = 0.05 and p = 0.01 for the common distributions and degrees of freedom.
| Distribution | df | Critical value for p = 0.05 | Critical value for p = 0.01 |
|---|---|---|---|
| Z (two-tailed) | — | 1.960 | 2.576 |
| t (two-tailed) | 5 | 2.571 | 4.032 |
| t (two-tailed) | 10 | 2.228 | 3.169 |
| t (two-tailed) | 30 | 2.042 | 2.750 |
| t (two-tailed) | 100 | 1.984 | 2.626 |
| χ² | 1 | 3.841 | 6.635 |
| χ² | 2 | 5.991 | 9.210 |
| χ² | 5 | 11.070 | 15.086 |
| F (df₁ = 1, df₂ = 10) | 1, 10 | 4.965 | 10.044 |
| F (df₁ = 3, df₂ = 20) | 3, 20 | 3.098 | 4.938 |
| F (df₁ = 2, df₂ = 27) | 2, 27 | 3.354 | 5.488 |
A statistic beyond the 0.05 critical value corresponds to p < 0.05; beyond the 0.01 value, p < 0.01. For F and chi-square, the critical value is always in the right tail.
Questions people actually search for
Twelve answers, each self-contained. Tap to expand.
What is a p-value in simple terms?
A p-value answers: if the null hypothesis were true, how likely is it that I would see data at least as extreme as mine? A small p-value means the data are unlikely under the null, which is evidence against it. It is not the probability that the null is true, and it is not the probability that the result happened by chance.
What is the difference between a Z score and a T score?
Both measure how far a statistic is from its hypothesised value in standard-deviation units. A Z score uses the known population standard deviation and the standard normal distribution. A t score uses the sample standard deviation, which introduces extra uncertainty, so it follows Student's t distribution with degrees of freedom. The same numeric value produces a larger p-value from a t distribution when the sample is small.
What is an F statistic and why does it have two degrees of freedom?
An F statistic is a ratio of two variances — typically between-group variance divided by within-group variance in an ANOVA. Because it is a ratio of two independently estimated quantities, it needs a degrees of freedom for the numerator and one for the denominator. Their order matters: reversing df₁ and df₂ produces a different, wrong p-value. The p-value is always a right-tail area, because F is bounded below at zero.
When should I use chi-square instead of t or F?
Use chi-square when your data are counts in categories rather than means or variances — for example, a goodness-of-fit test comparing observed counts to expected counts, or a contingency table testing whether two categorical variables are independent. Use t for comparing means and F for comparing three or more means.
How do I get a p-value from a correlation coefficient r?
Convert r to a t statistic: t = r × √(n − 2) / √(1 − r²), with n − 2 degrees of freedom. Then find the two-tailed area from the t distribution. The same r can be non-significant with a small sample and highly significant with a large one — an r of 0.30 has p ≈ 0.40 at n = 10 and p ≈ 0.002 at n = 100.
Should I use a one-tailed or two-tailed test?
Use two-tailed when a difference in either direction would be meaningful, which is the default in most fields. Use one-tailed only when you have a directional hypothesis specified before seeing the data and an effect in the opposite direction would be uninteresting or impossible. Choosing the tail after looking at the result is a form of p-hacking and inflates the false positive rate.
Why is p = 0.05 so common, and is it special?
0.05 is a convention, popularised by R.A. Fisher as a convenient threshold, not a law of nature. A p of 0.049 and a p of 0.051 are not meaningfully different, yet one is often published as a finding and the other as a null result. The ASA's 2016 statement explicitly warns against basing conclusions only on whether a p-value passes a specific threshold.
What does p = 0.000 mean?
It is impossible for a p-value to be exactly zero. Software that reports p = 0.000 is rounding a very small number, and the correct report is p < 0.001. Reporting "p = 0.000" overstates the precision and is one of the most common reporting errors in the literature.
Does a small p-value mean the effect is large?
No. A p-value depends on both the effect size and the sample size. A very small effect can produce a very small p-value if the sample is large enough. Conversely, a large effect in a small sample may produce a non-significant p-value. To judge magnitude, report an effect size (Cohen's d, odds ratio, risk difference) and a confidence interval, not a p-value alone.
Does a large p-value prove the null hypothesis is true?
No. A large p-value means the data are compatible with the null, not that the null is correct. Absence of evidence is not evidence of absence. The correct interpretation is that the study did not find a statistically significant difference — which may reflect a true null, low power, or a small sample. A confidence interval shows the range of effects compatible with the data and is the honest way to report a non-significant result.
What does the ASA say about p-values?
The American Statistical Association's 2016 statement set out six principles, including that p-values can indicate incompatibility with a model, that they do not measure the probability that a hypothesis is true or that data arose by chance, that conclusions should not rest only on a threshold, and that a p-value does not measure effect size or importance. It calls the p-value useful but commonly misused and misinterpreted.
Why do some studies fail to replicate if their p-value was below 0.05?
Because a p-value below 0.05 does not guarantee a true effect. Publication bias, p-hacking, low statistical power and small sample sizes all inflate the proportion of significant results that are false positives. Large-scale replication projects have found that a substantial fraction of published significant findings do not replicate, and one widely cited figure is that only about 24% of psychology studies reporting p ≈ 0.05 could be confirmed.
Compute a p-value from any of the five statistics
The calculator accepts Z, t, F, r and chi-square, shows the tail area for the distribution you select, and makes the degrees of freedom explicit so you can check them against your design.
Open the P value calculator