Vanity Metrics vs. Actionable Metrics: A Two-Question Test
Here's the fastest way to catch a vanity metric: ask whether it can go down. Total signups, cumulative downloads, lifetime page views. Those numbers only ever climb, which is exactly why they feel so reassuring and tell you so little. An actionable metric, by contrast, can drop next week and force you to ask why. That difference, movement you have to explain, is the whole game.
I spent about three years as the person who built the dashboards nobody used. Twelve tiles, mostly big totals, all trending up and to the right. Everyone loved the Monday review. Nobody ever changed a single decision because of it. So let me hand you the two-question test I wish someone had handed me back then, and then a table you can steal.
The two-question litmus test
Before a number earns a spot on your dashboard, make it pass two checks.
One: does it change a decision? If the metric doubled tomorrow, or halved, would anyone do something different? Ship a feature, kill one, reroute the onboarding flow, call a customer? If the honest answer is "we'd feel good/bad and carry on," it's decoration.
Two: is it a rate or ratio, not a cumulative total? Totals accumulate. They can't distinguish a great week from a dead one, because last quarter's wins are still baked into the sum. Rates reset. "Signups this week" beats "total signups" for the same reason a speedometer beats an odometer: one tells you how you're doing right now, the other just tells you you've been driving.
That's the term Eric Ries put a name to back in a 2009 guest post on Tim Ferriss's blog: a vanity metric is any number that makes you feel good but doesn't tell you what to do. He doubled down a year later in Harvard Business Review, warning that gross counts like total registered users paint a picture rosier than the truth. Fifteen years on, the dashboards haven't really caught up.
Why totals flatter you
Think of a metric like a grocery receipt. The total at the bottom tells you what you spent, sure. But if you're trying to figure out why the bill keeps creeping up, the total is useless. You need the line items, the unit prices, the thing you bought three times this month without noticing. Cumulative totals are the number at the bottom. Actionable metrics are the line items.
Here's the mechanical reason it matters. Say you launched eighteen months ago and you've collected 100,000 signups. Wonderful. Now this week you added 400. Is that good? You genuinely cannot tell from the 100,000. The total absorbs the new number and barely flinches, so a bad week and a great week look identical on the chart. A cohort view, "of the people who signed up in week 23, how many came back on day seven," would have screamed at you.
And that day-seven number isn't a toy. Amplitude's 7% retention rule found that if just 7% of an original cohort returns on day seven, the product already sits in the top quartile for activation. More striking: 69% of the top day-seven performers were also top performers at three months. A rate you can measure in a week predicts the outcome you actually care about. A running total predicts nothing except that time has passed.
The rewrite table
This is the part to bookmark. Left column, the metric that looks great in the all-hands. Right column, the version that actually earns its place. Middle, the reason. Most of these are the same move, swapping a cumulative total for a cohorted rate or a ratio.
| Vanity metric | Actionable rewrite | Why the rewrite wins |
|---|---|---|
| Total signups | Weekly activated cohort | Isolates whether this week worked, not the whole company's history |
| Total registered users | 7-day retention per signup cohort | Distinguishes tourists from residents |
| Page views | Conversion rate on the key action | Traffic without an outcome is just server load |
| Social followers | Referral traffic that converts | Followers are an audience; buyers are a business |
| App downloads | Day-1 activation rate | A download is a shelf; activation is a customer |
| Email list size | Active-subscriber engagement rate | A big list of non-openers is a deliverability risk, not an asset |
| Cumulative revenue | Net revenue retention | Shows whether the existing base is expanding or leaking |
| Number of features shipped | Adoption rate per feature | Shipping is output; adoption is outcome |
| Total sessions | Sessions per active user | Reveals depth of habit, not just headcount |
| Time on site | Task-completion rate | More time can mean confusion, not delight |
| Total likes and reactions | Save-to-conversion rate | Applause is cheap; intent is expensive |
| Gross accounts created | Free-to-paid conversion rate | Free accounts cost you money until they convert |
| Cumulative GMV | Repeat-purchase rate | One-time buyers don't compound; repeat ones do |
| Support tickets closed | First-contact resolution rate | Closing fast can just mean reopening later |
| Demos booked | Demo-to-close rate | A full calendar of no-shows is not pipeline |
| Total API calls | Active integrations per account | Volume can be one noisy script; breadth is stickiness |
| Website traffic | Qualified-lead rate | Wrong-fit visitors dilute everything downstream |
| Total impressions | Click-through-to-activation rate | Being seen isn't being chosen |
| Raw monthly active users | Share of MAU hitting the aha action | "Active" means opened the app; "aha" means got value |
| NPS responses collected | Segmented NPS trend by cohort | A blended score hides the segment that's churning |
Twenty rows, one pattern. If you only remember one transformation, make it this: replace the total with "the rate at which the thing that matters happens, for the people who arrived recently."
Where this goes wrong
I don't want to leave you thinking every big number is a lie. That's the overcorrection, and it's its own kind of sloppy.
Some totals are legitimately actionable. Cumulative recurring revenue against a runway is a survival number, not vanity. Total accounts matters enormously the day you're negotiating an acquisition or a fundraise. The metric isn't guilty by its shape; it's guilty when nobody can name the decision it drives.
And vanity metrics have one honest use: direction. Followers, impressions, raw traffic, they're fine as smoke detectors. If traffic falls off a cliff, something broke, and the total will tell you faster than a carefully cohorted rate. Just don't confuse the smoke detector with the thermostat. One tells you there's a fire. The other actually runs the house.
There's also a subtler failure mode where a metric is technically a rate but still useless because you can't act on it. "Global average session length across all users and platforms" is a ratio, congratulations, and it moves for reasons you can never isolate. Marketing ran a campaign, a bot farm hit the site, an enterprise account onboarded 500 seats. The number wiggled and you'll never know which. Ratios earn their keep only when they're narrow enough to trace.
The three A's, and why "accessible" is the one people skip
Ries later sharpened this into a checklist he called the three A's: a good metric is actionable, accessible, and auditable.
Actionable you've already got. Cause and effect you can trace to a decision.
Auditable means you can verify the number. You can trace it back to raw events, check the query, and confirm it means what the slide claims. If two people pull "activation rate" and get two different numbers, you don't have a metric, you have an argument.
Accessible is the one teams quietly skip, and it's the one I'd defend hardest. Can everyone who needs to act on the metric understand it without a stats degree? If your headline number takes a twenty-minute explanation every time it appears, people will nod in the meeting and ignore it at their desks. I've watched a beautifully-constructed retention metric die precisely this way. It was correct. It was auditable. Nobody outside the data team could say what it meant, so it may as well not have existed.
Accessible is why the litmus test is two plain questions and not a formula. A metric that needs a translator isn't going to drive behavior across a company.
Rewiring a dashboard without burning it down
You don't have to nuke everything Monday morning. A softer sequence:
Start by auditing what you've got against question one. Go tile by tile and write down the decision each number would drive if it moved 30% in either direction. The tiles where you can't name a decision are your vanity shortlist. Don't delete them yet. Just move them to a second, clearly-labeled "context" section so nobody mistakes them for scoreboard.
Then, for each promoted metric, run the table's move: total becomes cohorted rate. If your tool supports cohorts and most modern ones do, this is a rebuild, not a rewrite of your whole stack. If you're picking a north-star to anchor the whole thing, that deserves its own careful process, and I'd point you to our walkthrough on building a north-star metric tree before you commit a company to one number.
Finally, gut-check against a benchmark so "good" isn't just a vibe. For B2B SaaS, Wudpecker's 2025 retention benchmarks put top-tier net revenue retention above 120% and gross retention above 95%, with medians around 106% and 90%. Now your retention rate isn't a lonely number climbing a chart. It's a number with a target and a decision attached: below median, and onboarding or expansion needs work this quarter. That's the whole difference. A metric with a decision bolted to it is worth ten that just look nice.
If you want the shared vocabulary to run this exercise with a marketing or finance counterpart who keeps quoting totals at you, the growth-finance metrics glossary is a decent Rosetta Stone.
A short FAQ
Is MAU always a vanity metric? No, but raw MAU usually is. "Opened the app" and "got value from the app" are different events, and only the second one predicts retention. Track the share of your monthly actives who hit the core action, and MAU turns actionable.
What's the single fastest fix? Swap one cumulative total on your main dashboard for its weekly cohorted rate. Signups to weekly-activated-cohort is the classic. You'll feel the difference in the next review.
Are vanity metrics ever worth reporting? As directional smoke detectors, yes. Put them in a labeled context panel, never on the scoreboard, and never in a goal.
Ask the two questions. Does it change a decision, and is it a rate rather than a running total. Most of what's currently glowing on your dashboard won't survive both, and that's not a loss. That's you getting a smaller dashboard that people actually use.