A/B test calculator
Check if your test won, plan the sample size and how many weeks it needs, and see what the winner is worth in revenue. Significance, Bayesian chance to win, sample ratio check and revenue tests in one place. Free, no sign-up.
Made by Anatolii Ulitovskyi 
Fact-checked by RH Fardin Updated Your next test idea
Test an AI agent that answers visitors' questions on your pricing page as your next variation.
CRMsoftware.pro answers questions on your website, WhatsApp, Telegram and email in seconds, day and night, and turns each conversation into a contact and a deal you can measure.
- AI agent trained on your website, documents and policies
- Hands off to a teammate with the full context
- Every chat becomes a contact and a deal in your pipeline
Free plan, no credit card. Paid plans from $19/month, billed yearly.
The method
How A/B test significance works
The calculator runs a two-proportion z-test: is the difference between A and B bigger than random noise would usually produce?
The standard error comes from the pooled conversion rate of both groups. The z-score turns into a p-value: the chance of seeing a difference this large if A and B were really the same. A p-value below 0.05 is significant at 95% confidence. The confidence interval for the difference uses the unpooled standard error.
5.0% vs 5.8%
z = 2.50, p = 0.012
10,000 visitors each, 500 vs 580 conversions. B is up 16%, significant at 95%. The lift is likely between 3.5% and 28.5%.
Detect a 10% lift
31,234 visitors per version
At a 5% conversion rate, 95% confidence and 80% power. With 20,000 visitors a week, run it 4 full weeks.
Ship a 10% winner
+100 conversions = $8,000 a month
50,000 visitors a month at 2% and $80 per conversion. That is $96,000 a year.
Before you launch
Sample size and minimum detectable effect
The smaller the lift you want to detect, the more visitors you need. Halving the lift roughly quadruples the sample.
The minimum detectable effect (MDE) is the smallest lift your test can reliably find. Plan my test shows the visitors you need for your MDE and, in the table, the smallest lift you can detect if you run for 1 to 8 weeks. If the lift you hope for is smaller than what your traffic can detect, test a bolder change or a page with more traffic.
Testing more than two versions splits the traffic and needs a stricter threshold. The calculator applies the Bonferroni correction to the confidence level, so each comparison with the control stays honest.
Timing
How long should an A/B test run?
Until it reaches the planned sample size, and for whole weeks.
Cover every weekday
Buyers behave differently on Monday and Saturday. Round the duration up to full weeks so each version sees the same mix.
Decide the sample first
Checking results every day and stopping at the first significant moment inflates false wins. Set the sample size in advance and read the result once.
Watch for pollution
Very long tests pick up cookie loss, seasonality and other campaigns. If you need months, test a bigger change instead.
Another view
Bayesian chance to win
Next to the p-value, the result shows the probability that B is better than A, and how much you risk losing if you ship B and it is actually worse.
The calculator uses a Beta(1, 1) prior for each version and computes the exact chance that B's true rate is above A's. In the example, B has a 99.4% chance to be better. Many teams find this easier to explain than a p-value, but it answers a different question, so use the same threshold every time.
Data quality
Sample ratio mismatch
If you split traffic 50/50 but one version got noticeably more visitors, something in the setup is broken and the result cannot be trusted.
The calculator runs a chi-square test on the visitor counts against your planned split. A p-value below 0.01 flags a mismatch. Common causes are redirects that lose visitors, bots, tracking that fires only on one version, and caching. Fix the cause and restart the test rather than trusting the numbers.
Why it matters
What a winning test is worth
A lift in conversion rate becomes money only through your traffic and order value.
Revenue impact also shows what each week of delay costs once you have a winner, and the effect at the low end of a range if you enter a cautious lift. For revenue per visitor or average order value, use Average value: it runs Welch's t-test, which does not assume both versions have the same spread.
Spreadsheets
A/B test formulas for Excel and Google Sheets
A1 = visitors A, B1 = conversions A, C1 = visitors B, D1 = conversions B.
Lift of B over A
z-score
Two-sided p-value (E1 = z-score)
Visitors per version (F1 = baseline rate, G1 = rate you hope for)
FAQ
A/B testing questions
How do I know if my A/B test is statistically significant?
Enter visitors and conversions for both versions. If the p-value is below 1 minus your confidence level, for example below 0.05 at 95%, the difference is significant. With 500 of 10,000 against 580 of 10,000, p = 0.012, so B wins at 95% confidence.
How many visitors do I need for an A/B test?
It depends on your conversion rate and the smallest lift you care about. At a 5% conversion rate, finding a 10% relative lift with 95% confidence and 80% power needs about 31,234 visitors per version.
How long should I run an A/B test?
Until each version reaches the planned sample size, rounded up to full weeks. Divide the total visitors needed by your weekly traffic, and do not stop early because the result looks good.
What is a good confidence level?
95% is the common default. Use 99% for changes that are costly to reverse and 90% for quick, low-risk tests. Decide before you start.
What is statistical power?
Power is the chance your test detects a lift of at least your MDE when it really exists. 80% is standard; 90% needs about a third more visitors.
Bayesian or frequentist: which should I use?
Both use the same data. The p-value controls how often you call a false winner; the Bayesian chance to win tells you how likely B is better. Pick one rule for decisions and apply it consistently.
What is a sample ratio mismatch?
A visitor split that differs from the plan by more than chance allows, for example 52/48 on large traffic with a 50/50 setup. It usually means a tracking or redirect bug, and the test should be fixed and rerun.
Can I test revenue instead of conversion rate?
Yes. Use Average value with the mean and standard deviation of revenue per visitor for each version. It runs Welch's t-test and gives the difference with a confidence interval.
References
Sources
- Sample size for conversion tests: Evan Miller
- Diagnosing sample ratio mismatch in online experiments, KDD 2019: Fabijan et al., ACM
- Welch's t-test: Wikipedia
More free sales tools
Turn more visitors into conversations, and conversations into deals.
CRMsoftware.pro answers your visitors with AI, captures every lead and shows which conversations close. Free plan, no card.