Simpson's Paradox: When Your Aggregate Numbers Lie
The short version: Simpson's paradox is when a trend holds inside every segment of your data yet flips when you pool the segments together. Your conversion rate can rise for desktop users and rise for mobile users and still fall for the site as a whole. Nobody touched the numbers. The aggregate quietly misled you, and the cause is almost always a shift in who showed up, not a change in how well anything actually performed.
That's the whole trap in two sentences. The rest of this piece is about recognizing it in the wild, because it looks exactly like good news or a crisis, and it's neither.
The reversal, with real numbers
Let me give you the smallest example I can that still stings. Imagine you shipped an onboarding change last month and you're reading the before-and-after. You have two kinds of visitors, desktop and mobile, and you track signup conversion for each.
| Visitors | Conversions | Rate | |
|---|---|---|---|
| Before, Desktop | 900 | 90 | 10.0% |
| Before, Mobile | 100 | 4 | 4.0% |
| Before, Total | 1,000 | 94 | 9.4% |
| After, Desktop | 300 | 36 | 12.0% |
| After, Mobile | 700 | 35 | 5.0% |
| After, Total | 1,000 | 71 | 7.1% |
Read the segments. Desktop went from 10% to 12%. Mobile went from 4% to 5%. Both improved. Your onboarding change did what you hoped, in every group you can point to.
Now read the totals. Overall conversion fell from 9.4% to 7.1%. That's a 24% relative drop, and if that's the number on the dashboard your VP is staring at, you're about to spend a very unpleasant afternoon defending a change that worked.
So what happened? The traffic mix moved. Before, 90% of your visitors were desktop, the group that converts well. After, only 30% were. A marketing push (or a seasonal swing, or a new app-install campaign) flooded you with mobile visitors, who convert at a third of the desktop rate no matter what your onboarding does. The average got dragged down by composition, not by performance. Every table cell went up; the weighted blend went down because the weights changed underneath it.
I've watched a genuinely good experiment get rolled back over exactly this. The team never segmented. They saw one number fall and trusted it, the way you'd trust a scale that reads five pounds heavier without asking whether you were holding the groceries.
Why the average betrays you
Think of your overall conversion rate as a city's average commute time. If half the city takes the express train and half takes the slow bus, the "average commute" isn't a fact about either mode. It's a fact about the ratio of train riders to bus riders. Move a thousand people from the train onto the bus and the average commute gets worse, even if both the train and the bus got faster that week. The modes improved. The mix ruined the headline.
An aggregate rate is a weighted average, and a weighted average carries two independent moving parts: the rate inside each group, and the share of the total each group represents. We instinctively read a change in the blended number as a change in the rates. Half the time it's the shares that moved. When the shares move hard enough in the wrong direction, they can overpower real improvements in every underlying rate and flip the sign of the whole thing.
The confounding variable is whatever drives that mix. Device, in the example above. Acquisition channel. Country. New versus returning. Plan tier. Anything that (a) correlates with your outcome and (b) changed its share of the population between the two things you're comparing. If both of those are true, your top-line comparison is unsafe until you've held the mix constant.
The two textbook cases worth knowing
You'll trust the pattern more once you've seen it outside your own funnel, so here are the two that get cited for a reason.
Berkeley, 1973. The university looked like it was discriminating against women in graduate admissions. Overall, it admitted 44% of male applicants and only 35% of female applicants, a gap far too large to shrug off as noise, per the Wikipedia summary of the case. But when statisticians broke the data down by department, women were admitted at an equal or higher rate than men in most of them. The catch: women disproportionately applied to the most competitive departments, where admit rates were low for everybody. Men applied more to departments that took almost anyone. The mix of where people applied produced an aggregate that pointed the opposite way from every department. The finding was published by Bickel, Hammel and O'Connell in Science in 1975, and it's still the cleanest real-world reversal on record.
Kidney stones. A study comparing two treatments found Treatment A succeeded 78% of the time overall and Treatment B 83%. Pick B, obviously? No. Broken out by stone size, Treatment A won for small stones (93% vs 87%) and won for large stones (73% vs 69%), again from the Wikipedia write-up. Treatment A was better for every patient. It only looked worse because doctors reserved the tougher surgery for the harder (large-stone) cases, so its overall number carried more difficult patients. Same shape as your funnel: the better option, buried by an unfavorable mix.
Notice both stories have the same anatomy. A reversal, a lurking third variable, and an aggregate that lies by omission rather than by error.
It isn't a math bug, so which number is right?
Here's the uncomfortable part people skip. When the segments say one thing and the total says another, neither is "wrong." They answer different questions.
The segmented view answers: does this treatment / variant / onboarding flow work better for a given kind of user? The pooled view answers: what did the overall population actually experience? Both are true. Your job is to know which question you're being asked before you quote a number at anyone.
If you're deciding whether to keep the onboarding change, the segment view is what matters, because next month's mix is your choice and your problem, not the change's fault. If you're forecasting revenue for a board that only cares about the blended reality, the pooled number is honest too, mix shift and all. The mistake isn't picking one. It's not noticing there were two.
This is also why I get twitchy about a single top-line "conversion rate" pinned to a wall as the number everyone celebrates. It's a beautiful vanity metric: legible, trend-lined, and structurally incapable of telling you whether it moved because you got better or because your audience changed. A number that can't distinguish those two is decoration, not instrumentation.
The rule I actually use
Before you celebrate or panic over any aggregate rate that moved, ask one question: did the mix change? Then answer it before you touch the interpretation.
Concretely, whenever a blended metric shifts, decompose it the same way every time:
- Split the population by the dimensions most likely to correlate with the outcome: channel, device, geo, new/returning, plan.
- Check the rate inside each segment against the prior period.
- Check each segment's share of the total against the prior period.
- If the rates all moved one way but the total moved the other, you've found a mix shift, and the headline is about composition, not performance.
If your rates held steady and only the shares moved, your product didn't change; your marketing did. If the rates moved and the shares didn't, congratulations, the aggregate is telling the truth for once. It's the crossed case, rates up and total down (or the reverse), that should stop you cold.
One honest caveat: you can't segment by a variable you never recorded. Berkeley could only untangle its data because it had the department on every application. If your events don't carry the dimension that's driving your mix, the paradox is invisible and you'll argue about the wrong thing for a week. Instrument the obvious confounders up front. This is one of the arguments for keeping a clear decomposition of your headline metric so you already know which levers feed it. If you've built something like a north-star metric tree, you've done most of this work already, and the segments are just sitting there waiting to be checked.
A quick sanity checklist
- Weighted averages have two inputs: within-group rates and group shares. A moved total could be either.
- Same-direction segments plus an opposite-direction total equals a mix shift. Every time.
- Neither view is wrong. Ask what you're deciding, then quote the number that answers it.
- You can only catch this if the confounding dimension is in your data. Record it before you need it.
Simpson's paradox has a scary name and a boring cause. It isn't a glitch, a statistical curiosity, or a sign your pipeline broke. It's just the ordinary arithmetic of averages meeting a change in who your users are, and it happens far more often than the two famous case studies suggest, because product traffic shifts constantly. The good news is the defense is cheap: never read an aggregate move without checking whether the composition moved first. Do that, and the paradox stops being a trap and becomes what it should have been all along, a routine second question you ask before you say a single word about the first number.