Skip to main content
E-Commerce & Conversions

Why Most A/B Tests Lose, and What Actually Moves Conversion

The average A/B test barely moves the needle, and most B2B sites don't have the traffic to test much at all. Here's what the data says, why a system beats a pile of tests, and how to run the few tests worth running.

PK
Co-founder and CTO, Chykalophia · Updated · 16 min read
Two page variants, A and B, whose test results come out level

A/B testing has a marketing problem: it's sold as a slot machine that prints wins. The reality is less flattering and more useful. Most tests barely move the number, many never reach significance, and the occasional huge win is impossible to spot from any single result. Once you accept that, you can stop gambling on tests and start building the thing that compounds.

The math nobody puts on the slide

The case studies tend to leave this part out. When experimentation analyst Georgi Georgiev reviewed 115 published A/B tests, the 85 that survived his quality checks averaged a lift of 3.77 percent, and he judged about 70 percent of the tests underpowered for the effects they could realistically detect (Georgi Georgiev, Analytics-Toolkit, 2018). And that sample skews optimistic, because it's drawn from tests people chose to publish. The flops mostly never get written up.

3.77%average lift in a review of published A/B tests
Across the 85 of 115 published tests that passed quality checks. About 70% of the tests were judged underpowered, and the sample skews optimistic since it counts only tests people chose to publish.Georgi Georgiev, Analytics-Toolkit, 2018

Two things follow from that. First, the typical win is small, nowhere near the doubling you see in the ads. Second, and worse, a lot of the "wins" teams celebrate are noise dressed up as signal, because the test was called before it had the power to know anything. If you've ever shipped a "winner" that mysteriously stopped winning in production, you're in good company, and this is usually why.

But when a test wins, it can really win

So why does anyone bother? Because the payoff curve is lopsided. The classic example is Bing: a change to how ad headlines displayed, rated so low-priority it sat in the backlog for more than six months, increased revenue by 12 percent, which worked out to more than 100 million dollars a year in the US alone (Harvard Business Review, Kohavi and Thomke, 2017).

$100M+a year from a single 12% ad-headline test at Bing
The change was rated so low-priority it sat in the backlog for months before anyone shipped it.Harvard Business Review, Kohavi and Thomke, 2017

That's the real shape of testing. In the same article, Kohavi and Thomke report that only about 10% to 20% of experiments at Google and Bing generate positive results, and that at Microsoft as a whole one-third prove effective, one-third are neutral and one-third are negative (Harvard Business Review, Kohavi and Thomke, 2017). Most experiments do a little or nothing, and once in a while one of them is worth a fortune. The problem is you can't tell in advance which is which, and you can't tell from a single test after the fact either. The value comes from a machine that runs enough of the right tests, measured properly, over time.

A pile of individual tests is a pile of weak, noisy signals.

Why "just run more tests" is the wrong fix

The intuitive response is to test more. Run everything, all the time, and let the winners pile up. That doesn't work, for two reasons.

Most sites don't have the traffic. Reaching significance on a modest lift can take tens of thousands of visitors per variation. A manufacturer's RFQ page or a B2B firm's contact page might get a few thousand visits a month, so you'd wait a quarter or more to learn one small thing, and by then the season, the offer or the market has changed underneath you. Testing volume is a luxury good, and most B2B sites can't afford it on the pages that matter.

And volume without direction is just faster noise. If you test button colors and headline fonts because they're easy, you'll spend your limited traffic learning that button colors rarely matter. Test ideas are cheap. The traffic to resolve them is scarce, so where you spend it matters a lot.

The headroom is real, which is the good news

None of this means conversion work doesn't pay. The spread between pages is wide. Across more than 41,000 landing pages, the median converted 6.6 percent of visitors, and medians by industry ran from 3.8 to 12.3 percent (Unbounce, 2024). That spread isn't luck, and it's rarely one heroic test. It comes from a lot of correct decisions about who the page is for, what it promises, and how little friction stands between interest and action.

The businesses at the top of that range are rarely the ones who ran the most tests. They are the ones who got the fundamentals right, tested the few things worth testing, and trusted the results.

What actually works: a system, not a pile of tests

It works better to stop treating testing as the strategy and treat it as one instrument inside a system. That system has four parts, and testing is only the third.

Start with research, not opinions

Before you test anything, find where the page is leaking: analytics for where people drop, session recordings for where they struggle, and a handful of real conversations with buyers. Most high-value fixes aren't close calls that need a test. They're obvious once you look, so ship them. We wrote up where to start in why your website gets traffic but no leads and the ten elements of a high-converting page. If you're a manufacturer, the eight checks in what manufacturing buyers check on your website are a ready-made research list.

How to do it

  1. In GA4, open Explore and build a Funnel exploration from landing to form-submit, then note the single step with the highest abandonment rate.
  2. Filter Microsoft Clarity recordings and heatmaps to that step, watch 10 to 15 sessions where people bailed, and tag the recurring friction (dead clicks, rage clicks, a field that errors).
  3. Run three to five short calls with recent buyers using an open-ended interview guide, and ask what almost stopped them from moving forward.
  4. Ship the unambiguous fixes straight to production without a test (a broken field, a confusing label, a missing answer to a common objection).
See a worked example

Here's a made-up but typical case. On a manufacturer's RFQ page, a GA4 funnel exploration might show most visitors reaching the form but very few submitting, and Clarity recordings might trace the gap to one broken field, such as a drawing upload that rejects common CAD file types without saying why. You fix that field directly. No test needed.

Tools to use
Steal our AI prompt

I'm doing pre-test research on a B2B [describe the page, for example a demo-request landing page] to find where it leaks before I run any A/B test. Here is my GA4 funnel data by step: [paste step names, users, and abandonment rates]. Here are notes from [number] Microsoft Clarity recordings of users who dropped: [paste observations such as dead clicks, rage clicks, form errors]. Here are quotes from [number] recent buyer calls: [paste quotes]. Help me (1) rank the likely leak points by impact and confidence, (2) separate "obvious fixes to ship now without a test" from "genuine hypotheses worth testing later," and (3) for each obvious fix, write the specific change to make. Do not invent numbers or causes I did not give you, and flag anywhere the data is too thin to draw a conclusion.

Prioritize by evidence, then spend your traffic carefully

Rank each candidate by how likely it is to matter and how much traffic it'll need to reach significance. Test the few big, plausible bets. Ship the obvious fixes without a test. Ignore the trivia, like button-color tests that will only teach you that button colors rarely matter.

How to do it

  1. Score every candidate on likely impact, strength of evidence, and ease using a framework like ICE or PXL, so ideas backed by analytics, session replays, and research outrank pure opinions.
  2. For each top-ranked idea, enter your baseline conversion rate and the smallest lift worth catching into Evan Miller's sample-size calculator to see how much traffic the test needs.
  3. Run that required sample against your real monthly traffic in VWO's duration calculator. If a test would take close to a year to resolve, treat it as not testable right now.
  4. Commit traffic only to the few big, plausible bets, ship obvious usability fixes without testing, and drop trivia like button colors that can never reach significance at your volume.
See a worked example

Take a hypothetical quote-request page converting near 3% on about 1,000 visitors a month. A button-color tweak you hope nudges conversions by a couple of percent would need over a million visitors per variant at 95% confidence and 80% power, so it gets dropped. Even a bold headline-and-offer rework aiming for a 25% lift needs roughly 9,000 visitors per variant, about a year and a half of traffic. At that volume, you're better off shipping the rework on the strength of your research and watching the quote requests than waiting for a formal test.

Tools to use
Steal our AI prompt

Act as a CRO analyst helping me prioritize A/B tests for a low-traffic B2B site. Context: my primary conversion goal is [describe, for example demo requests], the current conversion rate on the target page is [paste %], monthly traffic to that page is [paste number], and here are my candidate test ideas with the evidence behind each: [paste ideas plus any supporting analytics, session-replay, or research notes]. Score each idea for likely impact, strength of evidence, and implementation ease, then rank them. For the top ideas, estimate the sample size and rough number of weeks each would need to reach significance at my traffic given a realistic minimum detectable effect, and flag any that are not testable at my volume. Finally sort the list into three buckets: big bets worth testing now, obvious fixes I should just ship without a test, and trivia to ignore. Do not invent statistics; if you need a number I did not provide, tell me exactly what to measure.

Measure like you mean it

Decide the sample size and duration before you start, and hold to it. Watch guardrail metrics so a "win" on one number isn't wrecking another, like a checkout tweak that lifts add-to-cart but sinks completed orders. A result you called early is a guess with a chart attached.

How to do it

  1. Before launch, put your baseline conversion rate and the smallest lift worth shipping into Evan Miller's sample-size calculator, then divide the required per-variant sample by daily eligible traffic and round up to whole weeks so weekday and weekend cycles stay balanced.
  2. Write the fixed end date, the one primary metric, and one to three guardrail metrics (for example completed orders, revenue per session, refund rate, page load time) into the test doc before you look at any data, and commit in writing to not stopping early.
  3. Once traffic is flowing, run a sample-ratio-mismatch check to confirm the split matches your target. If it flags, treat the test as invalid and fix the cause instead of reading the result.
  4. At the pre-set end date, read the primary metric and every guardrail together, and only call it a win if the primary moved and no guardrail degraded.
See a worked example

A page converting around 4% needs roughly 10,000 visitors per variant to reliably detect a lift to about 4.8% at 95% confidence and 80% power, which at about 1,700 eligible visitors a day is close to two full weeks. So when clicks on the "request a quote" button jump on day 4, you keep running to the pre-set end date, because the guardrail that pays the bills, completed quote requests, is still flat.

Tools to use
Steal our AI prompt

I'm planning an A/B test and want to lock the measurement plan before I start. My baseline conversion rate is [paste baseline %], the smallest lift worth shipping is [paste minimum detectable effect], and I get about [paste daily or weekly eligible traffic]. Using 95% confidence and 80% power, walk me through the minimum sample size per variant and how many full weeks that implies, rounding up to whole weeks. Then help me pick one to three guardrail metrics this change could quietly harm (for example completed orders, revenue per session, refund rate, page load time, or [describe my funnel]), and draft a short pre-registration note stating the fixed end date, the primary metric, the guardrails, and a commitment not to stop early. Do not invent numbers; if you need an input from me, ask.

Compound every result into the next one

Every test, win or lose, teaches you something about your buyer, so write it down. That growing understanding is the real asset, and a one-off test can't give it to you. It's why your tenth test is smarter than your first.

How to do it

  1. Create one central log in Airtable or Notion with fixed fields for hypothesis, variant, primary result, win or lose, and a required plain-language "what this taught us about the buyer" entry.
  2. After every test, win or lose, write the buyer insight in words, not just the number, and tag it by page, buyer motivation, and the objection it touches.
  3. Before scoping the next test, search the log for related tags and cite at least one prior learning inside the new hypothesis.
  4. Review the whole log once a quarter to spot repeating buyer patterns across tests, and promote the recurring ones into your prioritization rules.
See a worked example

For example, picture a manufacturer that logs every change and test in one Airtable base with a required "what we learned about the buyer" field. After a losing test of a shorter quote form, they note that buyers kept phoning to ask about lead times instead. Combined with two earlier notes about lead-time questions, the pattern suggests buyers were deciding on delivery certainty rather than form length, so the next change puts typical lead times on the page instead of chasing another form tweak.

Tools to use
Steal our AI prompt

I run A/B tests on our B2B website and I keep a running log of what each test teaches me about our buyer. Here is the test I just finished: [paste hypothesis, variant, primary metric result, and whether it won or lost]. Here is what I already know about our buyer from earlier tests: [paste 2 to 3 prior learnings]. Help me write a concise log entry titled "what this taught us about the buyer": state the buyer insight in plain language rather than restating the metric, say whether it confirms or contradicts my prior learnings, tag it by the buyer motivation or objection it touches, and propose one follow-up hypothesis that builds directly on this insight. Do not invent numbers, sources, or claims beyond what I pasted; if something is unclear, ask me before filling it in.

That's the difference between the Conversion System and buying a testing tool. The tool runs the experiment. The system decides what's worth experimenting on, keeps you from fooling yourself, and turns each result into the next hypothesis. We measure it in booked calls and pipeline instead of a dashboard of inconclusive tests, and it's backed by a 2x conversion guarantee.

If you do run a test: the basics

When a decision is big enough and the page has enough traffic, run the test properly. The steps are simple, and skipping them is what turns tests into noise. Our docs guide to A/B testing for small businesses goes through each step in more detail.

  1. Start from evidence, and change one thing. Use your analytics and session recordings to pick one element on a high-traffic page: the headline, the offer, the call-to-action placement, or the form. Write the hypothesis down in one line: "Changing X to Y will increase [metric] because [reason]."
  2. Make two versions, three at most. Make them different in angle, beyond cosmetic tweaks. "Get a quote in 24 hours" versus "Talk to an engineer about your part" is a real test. "Submit" versus "Submit now" is not, and modest traffic will never detect the difference.
  3. Size it before you build it. Put your baseline rate and the smallest lift worth catching into a sample-size calculator. If the answer is more traffic than you'll see in a few months, don't run the test.
  4. Split fairly and don't peek. Send the same audience to each version over the same period, evenly split, and let it run to the sample size you calculated, in whole weeks.
  5. Check the result before you call it. Put the visitor and conversion counts into a significance calculator. If nothing clears 95 percent, keep the control and write down what you learned.

Good first tests on a B2B site tend to be the big ones: the headline and offer, where the main call to action sits (a heatmap that shows most visitors never scroll to it is a strong reason to test placement), and how much a quote form asks for. Button colors and title case versus sentence case rarely deserve the traffic.

Tools: Evan Miller's sample-size calculator to size the test, Microsoft Clarity to find the element worth testing, and the A/B Test Guide calculator to check significance.

The honest version

If you take one thing from the data, let it be this: testing is a way to reduce uncertainty on decisions that are close. It can't stand in for a strategy. For a high-traffic store, a disciplined testing program is one of the best investments you can make. For a lower-traffic B2B site or a manufacturer, the fastest gains come from research, sound fundamentals, and conversations with buyers, with testing reserved for the handful of decisions big enough to justify the wait.

Either way, the work is the same: build the system that makes good decisions repeatable, and let the tests serve it instead of hoping a pile of tests adds up to a strategy on its own.

See exactly where your site is losing conversions

Paste your URL and get an instant conversion score. No form to fill out, no sales call.

The Next Draft

Strategies for businesses building what's next.

Occasional notes on turning B2B websites into pipeline. Unsubscribe any time.

Join the conversation