What Is Statistical Significance?
Statistical significance tells you whether an observed difference between two proportions is bigger than chance alone would readily produce. The calculator runs a two-proportion z-test using the pooled proportion and reports the p-value and the statistical confidence (1 − p).
It also gives the absolute difference, the relative uplift of B over A, a confidence interval for the difference, Cohen's h as an effect size and the observed power at this effect. Significance alone does not say whether a difference is large enough to matter, so read the interval and effect size too.
Below you will find the formulas, a variables table, a worked example and answers to common questions such as what p-value to require and why stopping a test early distorts results.
How to Use the Statistical Significance Calculator
- Enter visitors and conversions for the control (A) and the variant (B).
- Choose a two-tailed test unless you fixed a one-sided hypothesis before the experiment started.
- Pick α. 0.05 is the standard (95% confidence).
- Read the verdict, p-value, uplift and confidence interval. Treat a non-significant result as “not enough evidence”, not as proof of no effect.
Statistical Significance Formula
Pooled proportion
z statistic
Confidence interval for the difference
Relative uplift
Statistical Significance Variables Explained
| Symbol | Meaning | Notes |
|---|---|---|
| nA, nB | Visitors in each version | |
| xA, xB | Conversions in each version | |
| p̂A, p̂B | Conversion rates | x ÷ n |
| p̂ | Pooled proportion | (xA + xB) ÷ (nA + nB) |
| z | Test statistic | Difference in rates ÷ standard error |
Statistical Significance Worked Example
With the example values loaded in the calculator above, the steps are:
- Rates: A = 250/5000 = 0.05; B = 300/5000 = 0.06.
- Pooled rate p̂ = (250 + 300) ÷ (5000 + 5000) = 0.055; standard error = 0.00455961.
- z = (p₂ − p₁) ÷ SE = 0.01 ÷ 0.00455961 = 2.193.
- p-value (two) = 0.028295; less than α = 0.05.
Result: Result is statistically significant = Yes; p-value = 0.028295; z statistic = 2.193.
How to Read an A/B Test Result
Version A converted 250 of 5,000 visitors (5%) and version B converted 300 of 5,000 (6%). The difference of 1 point is 20% in relative terms, and the p-value is 0.0283 (z = 2.19), so the result is statistically significant at the 5% level.
Look at the interval, not only the p-value: the 95% interval for the difference runs from 0.11 to 1.89 points, so the true gain could be very small. The observed power of 34.2% is just another way of restating the p-value and is not a reason to trust or distrust the result. Decide the sample size in advance, and avoid stopping the test the moment it crosses 0.05, because that inflates false positives.
What “statistically significant” does and does not mean
A significant result at α = 0.05 means that, if there were truly no difference between the versions, a gap this large would occur less than 5% of the time. It does not mean there is a 95% chance that B is better, and it says nothing about whether the difference is large enough to matter commercially.
Look at the confidence interval for the size of the effect. A significant uplift of 0.1 percentage points may not justify a change; a non-significant uplift with a wide interval may simply be an underpowered test.
| Result | Interpretation |
|---|---|
| Significant, large uplift | Strong case for shipping the variant |
| Significant, tiny uplift | Real but may not be worth the cost |
| Not significant, wide interval | Inconclusive; collect more data |
| Not significant, narrow interval around 0 | Good evidence of no meaningful difference |
One-tailed or two-tailed?
A two-tailed test detects a difference in either direction and is the safe default. A one-tailed test has more power for the direction you chose but cannot detect a harmful effect. Use it only when the direction was fixed in advance and an effect in the other direction would lead to the same action as no effect.
How much data do you need?
The required sample size depends on your baseline conversion rate, the smallest uplift you care about, α and the power you want (80% is typical). Smaller effects need far more traffic. Use the sample size calculator before launching an experiment.
Where Statistical Significance Is Used
A/B tests
Check whether a difference in conversion rates is statistically significant.
Uplift
Read the absolute and relative difference between variants.
Interval
See the 95% interval for the difference.
Choosing tails
Run a one-sided or two-sided test.
Common Statistical Significance Mistakes to Avoid
- Peeking: stopping the test the first time p dips below 0.05 inflates false positives dramatically. Fix the sample size in advance or use sequential methods.
- Multiple variants or metrics: each extra comparison raises the chance of a false win; adjust α (for example divide by the number of comparisons).
- Unequal traffic split or novelty effects can mislead; run the test across full weekly cycles.
- Too little data: small samples can only detect big effects. Check power and plan sample size first.
- Counting the same user more than once breaks independence.
- Decide the sample size and run time before starting.
- Use α = 0.05 and 80% power as a minimum standard.
- Report the interval for the difference, not only the p-value.
- This test assumes independent visitors and reasonably large counts (at least about 10 conversions and non-conversions per group).
Statistical Significance Terms and Definitions
- Uplift
- Relative improvement of the variant over the control.
- Power
- Probability of detecting a real effect of a given size.
- Type I error
- Declaring a difference when none exists (false positive).
Statistical Significance FAQs
How do I know if my A/B test is statistically significant?
Compare the p-value with your α. If p is below α, usually 0.05, the difference is statistically significant.
What is a good significance level?
0.05 is the convention. Use 0.01 when a false positive is costly and 0.10 for early exploratory tests.
What does 95% statistical confidence mean in A/B testing?
Many tools define it as 1 − p. Strictly, it means the p-value is below 0.05; it is not the probability that the variant is better.
Can I stop my test as soon as it is significant?
Not safely. Repeatedly checking inflates the false-positive rate. Decide the sample size in advance or use a sequential testing method.
What if my result is not significant?
It means the data do not give enough evidence of a difference. Check the confidence interval and power; you may need more data.
What sample size do I need?
It depends on the baseline rate and the smallest uplift worth detecting. Smaller effects need much more data. Use a sample size calculator.
Is this test valid for small numbers of conversions?
The z-test is an approximation that needs roughly 10 or more conversions and non-conversions in each group. For very small counts, use Fisher’s exact test.
Is statistical significance the same as a meaningful improvement?
No. With enough traffic a tiny uplift can be significant. Look at the confidence interval and the effect size to judge whether the difference matters for your goals.
Does statistical significance mean the result matters?
Not by itself. Check the size of the lift and its confidence interval. Here the 1 point difference has a 95% interval of 0.11 to 1.89 points, which is wide.
What is observed power?
The chance of detecting the observed difference at this sample size. The example shows 34.2%, below the usual 80% target, so the test was underpowered for such a small effect.