P-Value Calculator

Enter a z-score and this returns all five p-values that go with it, each with the region of the normal curve it measures shaded in. Leave the z-score blank and fill any one of the p-value boxes instead, and it works backwards to the z-score. The calculation runs both directions because reports quote p-values while tables and software quote z.

What a p-value actually is

A p-value is the probability of getting a result at least as extreme as the one you observed, assuming the null hypothesis is true. That last clause is the whole definition and it is the part that gets dropped.

The null hypothesis is the boring default: no difference between the groups, no relationship between the variables, the coin is fair. You start by granting it, then ask how surprising your data would be in a world where it holds. A small p-value means the data would be surprising in that world, which counts as evidence against it. A large p-value means the data is unremarkable there, so you have learned nothing that contradicts it.

So p = 0.03 says: if there were genuinely no effect, results this striking or more so would turn up about 3% of the time. It does not say there is a 3% chance the null is true, and it does not say there is a 97% chance your hypothesis is right. Those are different quantities entirely, and getting them mixed up is the most common error in applied statistics.

What a p-value is not

• Not the probability that the null hypothesis is true. The p-value is calculated by assuming the null; it cannot then measure how likely the null is. Answering that question needs prior probabilities and Bayes' theorem.
• Not the probability your result was a fluke. Same confusion, worded differently.
• Not a measure of how big the effect is. A trivial difference measured on a huge sample produces a tiny p-value. p tells you about evidence against the null, never about size or importance.
• Not a measure of how well the study was run. Bias, poor measurement and a bad sample all survive the arithmetic untouched, and a p-value computed on rubbish is rubbish with a decimal point.
• Not a decision on its own. A result at p = 0.049 and one at p = 0.051 are practically identical, and treating them as opposites is a habit rather than a finding.

From a measurement to a z-score

Before a p-value exists there is a test statistic. For a mean, the z-score counts how many standard errors your sample sits from the value the null claims:

$$z=\frac{\overline{x}-\mu }{\sigma /\sqrt{n}}$$

The denominator is the standard error, not the standard deviation, and dividing by the square root of n is why bigger samples produce bigger z-scores for the same difference. Quadrupling the sample halves the standard error. That is the whole mechanism behind large studies finding statistical significance in effects too small to care about.

Once you have z, the p-value is an area under the standard normal curve. For the usual two-sided test:

$$p=2(1-\Phi (\left|z\right|))$$

Φ is the cumulative normal function, giving the area to the left of a point. The calculator above evaluates it and inverts it, so z = 2 returns 0.0455 and entering 0.0455 in the two-tails box returns z = 2.

One tail or two

A two-tailed test asks whether the result differs from the null in either direction. A one-tailed test asks about one direction only, and it puts the whole rejection region in that tail, which makes it easier to reach significance. At z = 2 the two-tailed p is 0.0455 and the one-tailed p is 0.02275, exactly half.

That halving is why the choice has to be made before you look at the data. Deciding on a one-tailed test after seeing which way the result fell is not a test, it is a way of turning a non-significant result into a significant one. Two-tailed is the default for a reason, and one-tailed needs a justification that would have held before the data existed.

Where 0.05 came from

Ronald Fisher proposed 0.05 in Statistical Methods for Research Workers in 1925 as a convenient line, roughly one in twenty, and said plainly it was a matter of convention. It was never derived from anything. A century later it is treated in many fields as the boundary between a result and a non-result.

Other fields picked their own. Particle physics requires five sigma before announcing a discovery, which is a one-sided p of about 2.9 × 10⁻⁷, or roughly one in 1.7 million on a two-sided reading. Genome-wide association studies use around 5 × 10⁻⁸ because they run millions of tests at once. The threshold follows the cost of being wrong and the number of chances you gave yourself to be wrong.

Two ways to be wrong

A Type I error is rejecting a null hypothesis that was true, a false positive. Your significance level α is the rate you accept for it, so testing at 0.05 means accepting a 5% false positive rate on true nulls.

A Type II error is failing to reject a null that was false, a false negative, and its rate is β. Power is 1 − β, the chance of detecting a real effect when there is one. Power depends on the size of the effect, the noise in the measurement, and the sample size, and 80% is a common target. An underpowered study that returns p = 0.4 has not shown there is no effect; it has shown it was never in a position to find one.

The two trade off. Lowering α to 0.01 cuts false positives and raises false negatives on the same data. There is no threshold that avoids both, only a choice about which error costs you more.

Multiple comparisons

Run one test at α = 0.05 and the false positive rate is 5%. Run twenty independent tests on data with no real effects and the chance that at least one comes back significant is 1 − 0.95²⁰, which is about 64%. Run a hundred and it is 99.4%.

This is why testing many outcomes and reporting the one that worked is not analysis. The Bonferroni correction divides α by the number of tests, so twenty tests are each judged at 0.0025; it is conservative but easy to defend. False discovery rate procedures such as Benjamini-Hochberg are less strict and more usual in fields that run thousands of tests. Whichever you use, the count that matters is how many tests you ran, not how many you published.

What the profession now says

In 2016 the American Statistical Association issued a formal statement on p-values, the first policy statement of its kind in its history, setting out six principles: p-values can indicate incompatibility with a model, they do not measure the probability that the hypothesis is true, decisions should not be based on whether p crosses a threshold, proper inference requires full reporting, p does not measure effect size, and on its own it is a poor measure of evidence.

In 2019 a comment in Nature by Valentin Amrhein, Sander Greenland and Blake McShane, signed by more than 800 researchers, called for abandoning the notion of statistical significance altogether, arguing that dichotomising results at a threshold discards information and produces false certainty in both directions.

Neither says p-values are useless. Both say a p-value is one summary of one aspect of the data, and reporting it alone as a verdict is the problem. Report the effect size with a confidence interval, say how many tests you ran, and give the p-value as one number among several.

Reading your result

Match the row to the question you asked. P(x>Z) is the right-tail p for a one-sided test in the positive direction, P(x<Z) the left-tail equivalent, and P(x<−Z or x>Z) is the two-tailed p that most software reports by default. P(−Z<x<Z) is the confidence level that goes with that z, so z = 1.96 gives 0.95.

Then treat the number as one piece of evidence. If it is near your threshold in either direction, the honest reading is that the study was inconclusive, not that something was proved or disproved. Our Probability Calculator covers the normal areas these figures come from, and the Statistics Calculator gives the mean and standard deviation a z-score is built from.

One caveat on the method: this calculator uses the normal distribution. That is right when the population standard deviation is known or the sample is large. With a small sample and an estimated standard deviation, a t-distribution with n − 1 degrees of freedom is the correct reference, and it gives slightly larger p-values.