A/B Test Significance Calculator

Is the difference between two conversion rates real? Significance, confidence interval and the sample size you still need. The result updates as you type so you can compare scenarios quickly.

The method

How it is worked out

z = (p_B − p_A) ÷ √(p(1−p)(1/n_A + 1/n_B)) with pooled p; significant when |z| exceeds the critical value for your confidence
p-value
Chance of seeing a gap this large if there were no real difference
confidence interval
Range the true difference likely falls in

Keep in mind

What it leaves out

  • Multiple variantsTesting several at once needs a correction.
  • NoveltyEarly lifts fade; run full weeks.
  • Revenue per visitorConversion rate ignores order value.

Use the calculator to compare scenarios rather than as a guaranteed forecast.

About the A/B Test Significance Calculator

Test whether the difference between two conversion rates is real: significance, p-value, confidence interval and the sample still needed.

How it works and an example to tryShow less

How to use the result

A two-proportion z-test with pooled variance; significance when |z| exceeds the critical value; the interval uses unpooled standard error; sample size for the observed effect at 80% power.

Example to try

12,000 visitors each: A converts 4.5%, B 5.0%, an 11% lift, p = 0.07 — not significant at 95%. About 15,600 per variant needed.

Put it to work

Try it on a real situation

Load an example

What moves the result most

    Worked out from your figures above. Tap a row to try it.

    Who uses it, and when

    Product, growth and marketing teams running experiments.

    Mistakes worth avoiding

    • Stopping the test when it first looks good.
    • Testing several variants without a correction.
    • Ignoring order value.

    Questions

    What does the A/B Test Significance Calculator do?

    Is the difference between two conversion rates real? Significance, confidence interval and the sample size you still need.

    How should I use the result?

    A two-proportion z-test with pooled variance; significance when |z| exceeds the critical value; the interval uses unpooled standard error; sample size for the observed effect at 80% power.

    Does TechBullion store the information I enter?

    The core calculation or transformation runs in your browser. Normal website security and analytics may still record page-level events, but the tool does not need to create a public record from your inputs.

    What mistakes do people make with an a/b test significance calculator?

    Stopping the test when it first looks good. Testing several variants without a correction. Ignoring order value.

    Is the result guaranteed?

    No. Use the result as a practical check or planning aid and verify important decisions with the relevant primary source, provider or professional advice.

    Part of the guide · Campaign returns · 14 tools

    Marketing and website tools

    Return on ad spend, A/B test significance, email ROI, plus free checks for site speed, SSL, redirects, broken links, security headers, SEO basics, SERP previews and AI search readiness.

    Also under “Campaign returns”

    Open the guide: all 14 tools

    Executive Insights

    Leaders on this topic