How to Calculate A/B Test Sample Size Before You Launch

How do you calculate a/b test sample size?

Three inputs decide a/b test sample size before a single visitor sees the test: the baseline conversion rate, the minimum detectable effect and the statistical power required. Plug those three into the sample size formula along with a chosen significance level, and the formula returns the number of visitors each variant needs before the test can be read. Calculate that number before launch, not after the test has been running for a week and someone wants an early answer, and use it to decide what is a good conversion rate target for the test before comparing the result to any outside benchmark.

What is the baseline conversion rate?

The baseline conversion rate is the current rate of the metric being tested, measured on the page before any change ships, using the same conversion rate formula the business already tracks. A checkout page converting at 5% today is the baseline for a test on that same page, and the formula defines every other input relative to that starting number. Pull the baseline from at least two to four weeks of typical traffic rather than a single unusual day.

What is minimum detectable effect?

Minimum detectable effect is the smallest change in conversion rate the test is built to catch, stated as the difference between the baseline and the target rate. Choosing a 1 percentage point minimum detectable effect on a 5% baseline means the test is designed to reliably detect a move to 6%, and it will not reliably catch a smaller move like 5.3%. A smaller minimum detectable effect needs a larger sample, since the formula has to separate a smaller gap from random noise.

What does statistical power mean in an A/B test?

Statistical power is the probability the test detects a real effect of the size chosen, when that effect actually exists. A power of 80%, the common default, means the test misses a real effect one time in five, so raising power to 90% catches more true effects but requires a larger sample to do it.

What does significance level mean in an A/B test?

Significance level is the probability of declaring a winner when there is no real difference between the variants, commonly set to 5%, which corresponds to a 95% significance level. Lowering that false positive rate to 1% makes the test more conservative and, like raising power, increases the sample size needed to reach a verdict, the same tradeoff that shapes a/b testing on live traffic more broadly.

What is the sample size formula?

The sample size formula combines the baseline rate, the target rate and the Z-scores for the chosen power and significance level into one calculation per variant.

n = (Z(1-alpha/2) + Z(1-beta))^2 x [p1(1-p1) + p2(1-p2)] / (p2 - p1)^2

Work it on a checkout test. The baseline conversion rate p1 is 5% (0.05), the target p2 is 6% (0.06) after a 1 percentage point minimum detectable effect, significance is 95% (Z = 1.96) and power is 80% (Z = 0.84).

Input	Symbol	Value
Baseline conversion rate	p1	5%
Target conversion rate	p2	6%
Minimum detectable effect	p2 minus p1	1 percentage point
Significance level	alpha	5% (Z = 1.96)
Statistical power	beta	80% (Z = 0.84)
Sample size per variant	n	8,146 visitors

(1.96 + 0.84)^2 = 7.84, and [0.05 x 0.95 + 0.06 x 0.94] = 0.1039. Dividing 7.84 x 0.1039 by (0.01)^2 gives 8,145.76, rounded up to 8,146 visitors per variant, or about 16,292 total across both variants. Halving the minimum detectable effect to 0.5 percentage points roughly quadruples that number, since the denominator squares the effect size.

Two ways to read an in-progress test

Run to 8,146 per variant

• One read, at the number the formula set
• The 95% significance level holds as built
• A verdict counts as a decision

Peek before the number

• Checked daily for two weeks: fourteen chances for noise to cross the threshold
• Actual false positive rate ends up well above 5%
• A favorable swing is informal, not a verdict

Why does peeking at results early break the sample size calculation?

Peeking at results before the calculated sample size is reached breaks the test because each early look is a fresh chance for random noise to cross the significance threshold. A test checked daily for two weeks gives random variation fourteen separate opportunities to produce a false positive, so the actual false positive rate ends up well above the 5% the test was built to hold, even though each single check used a valid 95% threshold. Fix the sample size before launch, run the test to that number without checking for a verdict along the way, and use funnel and goal data to confirm the daily traffic rate needed to reach it on schedule.

Sizing a test before it launches

1

Measure the baseline. Two to four weeks of typical traffic on the page being tested.

2

Set the minimum detectable effect. The smallest change worth catching, as a percentage point difference.

3

Fix power and significance. 80% power and 95% significance are the common defaults.

4

Run to the number, not the calendar. Stop reading results as a verdict before the sample size is reached.

Frequently Asked Questions

What sample size do I need for an A/B test?

The sample size depends on your baseline conversion rate, the minimum detectable effect and the power and significance level chosen, with no fixed number that applies to every test. A test moving a 5% baseline by 1 percentage point at 80% power and 95% significance needs about 8,146 visitors per variant.

What is a good minimum detectable effect?

A good minimum detectable effect is the smallest change that would still be worth shipping, not the smallest change that is theoretically possible to measure. Setting it too small forces an unreasonably large sample size; setting it too large lets real, smaller improvements pass through the test undetected.

Why do smaller effects need bigger sample sizes?

Smaller effects need bigger sample sizes because the sample size formula squares the minimum detectable effect in the denominator, so halving the effect roughly quadruples the required sample. A test built to catch a 0.5 percentage point move needs about four times the visitors of one built to catch a 1 percentage point move.

What happens if I stop an A/B test early?

Stopping an A/B test early, based on a favorable-looking result before the calculated sample size is reached, inflates the actual false positive rate above the significance level the test was designed to hold. The reported 95% confidence no longer reflects the real odds of a false winner once the team checks results and acts on them repeatedly.

Is 80% power always the right choice?

80% power is a common default, not a requirement, and it means the test misses one in five real effects of the size chosen. Raising power to 90% catches more true effects at the cost of a larger required sample size.

Can I use a smaller sample size if I just want a directional signal?

Running below the calculated sample size for a directional read is possible, but the result carries a materially higher chance of being noise, not a smaller version of the same confidence. Treat any read taken before the target sample size as informal, and do not use it to make a permanent decision.

How long does it take to reach the sample size an A/B test needs?

The formula returns a visitor count, not a number of days, so the timeline depends on how much traffic the tested page already gets. A checkout page needing 8,146 visitors per variant reaches that number faster on a high-traffic page than a low-traffic one. Checking funnel and goal data before launch shows whether the current daily traffic rate can hit the target on a reasonable schedule.

Why measure the baseline over two to four weeks instead of a single day?

A single day can be unusually high or low for reasons that have nothing to do with the test, like a weekend dip or a marketing push. Two to four weeks of typical traffic smooths those swings out and gives the formula a baseline conversion rate that actually represents the page. Every other input in the formula, including the target rate and the minimum detectable effect, is defined relative to that starting number.

Does the sample size formula assume each variant gets equal traffic?

The formula calculates n as the number of visitors needed per variant, and the checkout example applies that same 8,146 to both the control and the variant, for 16,292 visitors total. That means the calculation assumes an even split of traffic between the two versions being compared. A test that sends unequal traffic to each variant will not reach both sample sizes at the same time.

Can the baseline conversion rate come from a different page than the one being tested?

No, the baseline has to come from the exact page the test will run on, measured on the metric that page is being tested for. A checkout page converting at 5% is the baseline for a test on that same checkout page, not a rate borrowed from a different page or a different metric. Using a rate from elsewhere breaks the relationship the formula relies on between the baseline and the target rate.