Sample Ratio Mismatch (SRM): The Silent A/B Test Killer

The short version: sample ratio mismatch (SRM) is when the real split of users across your A/B test variants differs from the split you designed by more than chance can plausibly explain. You asked for 50/50, you got 50,214 users in control and 49,001 in treatment, and that gap is too large to be luck. It's a symptom, and the disease is usually a broken pipe somewhere between your randomizer and your data warehouse. When it shows up, the honest response is to stop analyzing and start debugging, because the experiment's headline result is no longer trustworthy.

I've watched teams ship a "winning" variant off an SRM'd test, roll it out, and then wonder why the lift never materialized in production. The lift was never real. Randomization is the single assumption that lets you say "the button color caused the conversion change" instead of "the two groups were just different people." Break randomization and every metric downstream inherits the bias. No amount of statistical significance on your primary metric buys that back.

This is more common than most people assume. Microsoft Research, in the 2019 KDD paper that most practitioners point to, reported that roughly 6% of experiments at Microsoft show an SRM, drawn from a review of more than 10,000 experiments across Microsoft, Booking.com, Outreach.io, and Online Dialog. Six percent sounds small until you're the one running a hundred tests a quarter.

How to actually check for it

The check is a chi-squared goodness-of-fit test on the raw counts. You take the number of users you observed in each group and compare it against the number you'd expect under your intended split. The test statistic is:

X² = Σ (O − E)² / E

Where O is the observed count for a group and E is the expected count. For a 50/50 test, the expected count per group is just the total divided by two. One degree of freedom for a two-group test.

Let's run the numbers on that example split above. Total users: 99,215. Expected per group under a fair 50/50: 49,607.5.

Variant	Observed (O)	Expected (E)	(O − E)² / E
Control	50,214	49,607.5	7.42
Treatment	49,001	49,607.5	7.42
X²			14.83

A chi-squared statistic of 14.83 on one degree of freedom gives a p-value of about 0.0001. That's the SRM alarm going off. The split you're looking at would happen by pure chance roughly one time in ten thousand, so the far more likely explanation is that your test is broken.

Here's the copy-pasteable rule most experimentation platforms have converged on: run the chi-squared test on every experiment, and if the p-value is below 0.001, flag it and refuse to trust the results until you find the cause. Statsig, Eppo, and the analytics-toolkit glossary all land on that same 0.001 line, and Statsig frames a sub-0.001 p-value as strong evidence the split reflects a bug rather than noise.

Why 0.001 and not the usual 0.05

Good question, and it trips people up. You spent your whole stats education treating 0.05 as the magic threshold, so why get stricter here?

Because the cost profile is inverted. For your primary metric, a false positive means you ship something neutral, mildly wasteful. For an SRM check that you run automatically on hundreds of experiments, a 0.05 threshold means you'd cry wolf on 1 in 20 perfectly healthy tests. That's a firehose of false alarms, and teams learn to ignore alarms that are usually wrong. The 0.001 threshold keeps the false-positive rate low enough that when the SRM light turns on, people actually get up and investigate. You're trading a little sensitivity for a lot of credibility, and for a check that gates every experiment, that's the right trade.

The tiny-sample trap

Now the counterintuitive part, and the mistake I made in my first year. SRM is a large-numbers phenomenon. Small imbalances that look alarming are statistically nothing.

Imagine a pilot test with 12 users over 3 weeks, and your split came out 7 in control, 5 in treatment. That feels lopsided. Your gut says something's wrong. Run the math though: expected is 6 per group, so X² = (7−6)²/6 + (5−6)²/6 = 0.167 + 0.167 = 0.33. The p-value is around 0.56. Completely consistent with a fair coin.

The lesson cuts both ways. A 58/42 split on twelve users is fine, and a 50.6/49.4 split on a hundred thousand users is a five-alarm fire. Percentages lie here; absolute counts and the chi-squared test tell the truth. Don't eyeball the ratio and don't panic during the first hour of a test when volumes are low. Check it once the sample is meaningful, and check the counts, not the percent.

Where this goes wrong

Every SRM has a physical cause, and they cluster in a few predictable places. The 2019 KDD paper organizes them by the stage of the experiment where the leak happens — assignment, execution, log processing, analysis — and Lukas Vermeer's SRM Checker documents the same taxonomy in practical terms. Here's my ranked list, ordered by how often I've actually traced an SRM back to each one.

Redirect timing, by a wide margin. If your treatment redirects users to a different URL and control doesn't, you have a built-in asymmetry. Some users close the tab before the redirect fires, and those users never get counted in treatment even though they were assigned to it. Control loses nobody to a redirect it doesn't perform. This alone produces persistent, reproducible SRMs, and it's why redirect tests have a bad reputation among experimenters. If you must run one, measure assignment at the moment of bucketing, before the redirect, not after the destination page loads.

Asymmetric bot or outlier filtering. You filter bots and outliers to clean your data, which is fine, right up until the filter hits one variant harder than the other. If treatment changed page performance or added a new script that a bot-detection heuristic keys on, you can strip more "users" from one side. The fix is a discipline, not a setting: apply every filter identically across variants, and apply it to a variant-independent identifier.

Logging drops tied to the variant. Telemetry isn't free, and variants can change how much of it survives. A slower variant, a variant that adds a heavy component, a variant that fires events from a code path with a flakier logger, and any of these can drop a few percent of events on one side. Because the drop correlates with the variant, it manifests as SRM rather than random noise.

Bucketing and carryover bugs. Faulty hash functions, off-by-one ranges in the assignment logic, user IDs that get reused or corrupted, and carryover from a previous experiment that already sorted users unevenly. These live in the assignment stage and tend to give you clean, constant ratios that are just wrong, like a 60/40 when you asked for 50/50, stable all day.

Analysis-stage segmentation. This one's sneaky because your randomization was perfect. You introduce the SRM yourself by filtering the analysis on a post-treatment condition — say, "users who reached the checkout page" — that the treatment influences. Now you've conditioned on an outcome and re-broken the balance. The test data is clean; your query isn't.

What to do when the light turns on

Do not reweight the groups to force them back to 50/50 and carry on. I know the temptation — you spent two weeks collecting this data and throwing it out hurts. There's a school of thought that says you can statistically salvage an SRM'd experiment by correcting for the imbalance. I think that's wrong, and I'll say so plainly: if you don't know why the split is off, you don't know what else the same bug distorted. Reweighting the counts does nothing for the biased metrics that the same broken pipe produced. Invalidate the test, find the cause, fix it, rerun.

To localize the cause, segment the SRM itself. Recompute the chi-squared test by day, by browser, by platform, by traffic source. An SRM that only appears on Safari points at a client-side redirect or caching issue. One that starts precisely on day three points at a deploy or a logging change. One that lives entirely in a single country points at a bot-filter or CDN quirk. The distribution of where the mismatch concentrates is the fastest debugging signal you have.

And build the check in, don't run it by hand. The whole value of SRM detection is that it fires automatically on every experiment before anyone reads the results. A manual check is a check you'll skip on the busy week when you most need it.

A couple of questions people actually ask

Does SRM apply to unequal splits like 90/10? Yes, exactly the same. The expected counts are 0.9 and 0.1 of the total instead of 0.5 each, and the chi-squared test handles it without modification. Uneven designs are actually more sensitive to certain bucketing bugs, so keep checking.

Can I just widen my confidence intervals to account for it? No. SRM is bias, not variance. Wider intervals address uncertainty from small samples; they do nothing about a systematic tilt in who ended up where. There's no interval wide enough to fix groups that were never comparable.

My test barely crosses 0.001, significant SRM or bad luck? Give it a beat. Treat a borderline result as a prompt to collect a little more data and recheck, and to eyeball the segments. A real SRM usually deepens as volume grows because the underlying bug keeps leaking; genuine noise tends to drift back toward the line. If it's still borderline at scale and no segment lights up, you're probably fine, but log it.

SRM isn't a fancy technique. It's a smoke detector — cheap, boring, and the thing that saves you from confidently shipping a result that was broken before you ever looked at it.