// tools · a/b test significance calculator

A/B test significance calculator

Enter visitors and conversions for both variants to see whether the difference is likely real or just noise.

Did version B really do better, or is it noise?

Variant A (control)

Visitors

Conversions

Variant B

Visitors

Conversions

Confidence-

Conversion rate A: -
Conversion rate B: -
Change from A to B: -

Two-sided z-test for two proportions. Decide the sample size before you start and don't stop the test the moment it looks significant.

Talk to us about testing this properly →

// how it works

How it's calculated

Conversion rates
Conversions divided by visitors for each variant, and the relative change from A to B.
Confidence
A two-sided z-test for two proportions. 95% confidence means a difference this large would show up by chance about 1 time in 20 if the variants performed the same.
Sample size warning
Below about 100 conversions per variant, results swing a lot, so the calculator flags them as a rough signal.

// reference

How many visitors does an A/B test need?

Visitors per variant to detect a relative lift at 95% confidence and 80% power.

Current conversion rate	5% lift	10% lift	20% lift	30% lift
1%	637,008	163,095	42,693	19,827
2%	315,205	80,682	21,109	9,798
3%	207,938	53,211	13,914	6,455
5%	122,124	31,234	8,158	3,780
10%	57,763	14,751	3,841	1,774

Multiply by the number of variants, control included, for the total.

Common questions

What does statistical significance mean in an A/B test?

It means the difference between the variants is unlikely to be random. At 95% confidence, a difference this large would show up by chance about 1 time in 20 if the variants performed the same. It does not tell you how big the real effect is. The likely range does.

What sample size do I need for an A/B test?

It depends on your current conversion rate and the smallest lift you care about. Low conversion rates and small lifts need far more visitors. Use Plan a test above, or the table on this page, and decide before the test starts.

What confidence level should I use?

95% is the usual standard. For changes that are expensive to roll back, wait for more. For small, easily reversed tweaks, a lower bar can be reasonable, as long as you decide it before the test starts.

How long should a test run?

Decide the sample size up front and run at least one or two full weeks, so every weekday is covered. Stopping as soon as the result looks significant inflates false positives.

What if the result is not significant?

Treat it as no proven difference, not as proof the variants are equal. Keep the original or test a bolder change. Running the test longer than planned just to reach significance inflates false positives.

The result is significant. Is B definitely better?

Not definitely. It means the difference is unlikely to be pure chance. Check that tracking worked for both variants, that nothing else changed during the test, and that the lift holds on the metric you actually care about, like revenue.

// more tools

A/B test significance calculator

How it's calculated

Conversion rates

Confidence

Sample size warning

How many visitors does an A/B test need?

Common questions

Want a hand with what you found?