The Test That Tells Vanity Metrics From Decision-Useful Ones

What makes a metric a vanity metric?

Nothing on the usual list of vanity metrics is inherently vain, and a number earns the label only when no plausible value of it would change what you do next. That is the whole test. It is a property of the decision, not a property of the metric, which is why the same figure can be a serious operating input at one company and decoration at another.

The popular framing gets this backwards. "Likes are vanity, revenue is real" sounds decisive and explains nothing: a growth team that buys attention and sells attention can act on likes, and a founder who reads gross revenue without retention is being flattered by a number with a currency symbol in front of it.

Where does the term vanity metrics come from?

Eric Ries popularized it in a guest post published on 19 May 2009, arguing that some numbers "make you feel good, but they don't offer clear guidance for what to do." His examples were cumulative and aggregate: billions of messages sent, ten thousand hits, total pageviews, total registered users.

His proposed replacements were all about structure rather than which metric you pick. Split-tests produce the most actionable results, because the comparison is built in. Per-customer and per-segment views beat aggregates. Cohort analysis tracks groups over time instead of pouring every user into one bucket. And traffic can be segmented by the keyword that brought it, so acquisition and behavior stay connected. His summary line was that metrics are people, too, which is the part most teams skip.

Note what he did not do: he did not publish a blocklist. Every one of his fixes takes a supposedly vain number and cuts it into a shape that can carry a decision.

The three-part test

Apply these in order, and write the answers down before you open the dashboard.

1. Name the decision. Which specific thing changes depending on this number? "We would move budget from X to Y" is a decision. "We would know how we are doing" is not.
2. Name the threshold in advance. At what value do you act, and in which direction? If you cannot commit to a number before you see the number, you are going to rationalize whatever appears.
3. Ask whether it can go up while the business goes down. Every metric that can rise during a collapse needs a paired counter-metric, or it will eventually be used to hide the collapse.

A metric that fails all three is vanity. A metric that fails only the third one is not vanity, it is incomplete, and the fix is pairing rather than deletion.

What the three-part test decides

Fails all three

• No decision named
• No threshold set
• No counter-metric needed, because nothing is at stake

Fails only the third question

• Decision and threshold exist
• Can rise while the business falls
• Not vanity, only incomplete, needs a counter-metric

Passes all three

• Names the decision
• Names the threshold
• Paired against a collapse it could otherwise hide

Which metrics get called vanity, and what is the argument on each side?

Metric	The case for calling it vanity	The condition that makes it decision-useful
Cumulative registered users	Monotonic by construction, cannot fall, so it never signals a problem	Never in cumulative form; split into cohorts and it becomes retention
Pageviews	No relationship to money in a subscription business	It is the literal unit sold in ad-funded publishing
Social followers	Reach is throttled by an algorithm you do not control	You can trace followers to a channel with attributed revenue
Impressions	Platform-defined, redefined without notice, unverifiable by you	You are buying them, and the price is per impression
App downloads	Counts intent, not use; install-then-delete is invisible	Paired with day-7 or day-30 activation from the same cohort
Email open rate	Machine-inflated since 2021; the recipient may never have looked	Comparing two variants within the same untouched segment
Average session duration	Contaminated by how the last page is timed and by idle tabs	Segmented by template, on pages where dwell time is the product
Bounce rate	Definition changed underneath the name, so the trend is an artifact	One template, one traffic source, one definition, compared to itself
NPS	A single lagging score, small samples, heavy rounding	Read as the verbatim comments behind the score, not the score
Gross revenue	Rises while churn rises, if acquisition outruns the leak	Reported next to net revenue retention, never alone

Four of these are worth spelling out, because the argument is not the obvious one.

Cumulative totals are the only shape that is vanity by construction

"Total users ever registered" cannot fall. Neither can total signups, total downloads or total posts published. A number that is mathematically incapable of delivering bad news cannot inform a decision, no matter what it counts. This is the one case where the label is fair without knowing anything about your business. The repair is always the same: convert the total into a rate over a cohort, so that a bad month is allowed to look like a bad month.

Pageviews are payroll for a publisher and noise for a SaaS

The same metric flips sides entirely depending on the business model. If you sell impressions, pageviews are revenue with a one-step conversion, and treating them as vanity is malpractice. If you sell seats, a pageview spike tells you nothing until you know whether it came from buyers, from a Hacker News thread, or from a scraper. Before arguing about it, make sure everyone is counting the same thing; the differences between pageviews, sessions and users account for a surprising share of these disputes.

Email open rate stopped being a measurement in 2021

This one is not a judgment call, it is a mechanism. Apple announced Mail Privacy Protection on 7 June 2021, describing it as a feature that "helps users prevent senders from knowing when they open an email, and masks their IP address so it can't be linked to other online activity or used to determine their location." Mail fetches the tracking pixel whether or not the recipient reads anything. The metric did not become less important, it became partly fictional, and no amount of care in your email tool reverses that. Open rate survives only as a within-experiment comparison on an unaffected segment, and dies as a reported KPI.

Bounce rate and engagement rate are definition risk, not vanity

Google Analytics 4 defines an engaged session as one that "lasts longer than 10 seconds, has a key event, or has 2 or more screen or page views," defines engagement rate as the share of sessions that qualify, and defines bounce rate as "the percentage of sessions that were not engaged." Every one of those thresholds is a design choice. Add a second step to an onboarding flow and your bounce rate improves without a single visitor being better served. The number is not vain, it is sensitive to a definition you can accidentally move, which is a different failure and needs a different guard: freeze the definition, version it, and annotate the chart on the day it changes. Our breakdown of how engagement rate is calculated covers where the thresholds bite.

Average time on page has the same problem for a structural reason rather than a threshold one, since the clock depends on a following event that never arrives on the last page of a visit. We went through the arithmetic in why the time on page metric is flawed.

How do you convert a vanity metric into an actionable one?

Four moves cover almost every case, and they are the same four Ries reached for in 2009.

• Cohort it. Replace "total users" with "users who signed up in week N, still active in week N+4". The total becomes a curve, and curves can fall.
• Segment it. An aggregate that mixes paid, organic and direct hides the only thing you can act on. Splitting traffic by source is usually the difference between a number you look at and a number you spend against, which is the point of attributing revenue back to each channel.
• Ratio it. Absolute counts grow with spend. Rates do not. Signups per thousand sessions survives a traffic spike; signups does not.
• Pair it with a counter-metric. Every number you push on should have a partner that gets worse if you push dishonestly. Conversion rate pairs with refund rate. Session duration pairs with task completion. Velocity pairs with defect rate.

When is a so-called vanity metric the right metric?

When it is the thing you sell, when you are buying it at a known price, or when it is the only leading indicator available for something that takes a quarter to show up in revenue. Awareness campaigns are the honest example. Impressions and branded search volume genuinely lead the revenue they eventually cause, and demanding same-week attribution from them produces worse decisions than accepting a soft number with a stated lag.

The failure mode to avoid is not reporting a soft metric. It is reporting a soft metric without the decision attached, where it exists to be shown rather than to be used.

Frequently asked questions

Are likes and followers always vanity metrics?

No. They are vanity when nothing about your plan changes at any value of them, which is the common case for a B2B product with a long sales cycle. They are decision-useful when you can trace them forward to attributed pipeline, or when your business is literally paid for reach.

What is the difference between a vanity metric and a lagging indicator?

A lagging indicator reports a real outcome late. A vanity metric reports something that would not change your behavior at any value, early or late. Revenue is lagging and useful; total registered users is early and useless.

Is revenue ever a vanity metric?

Gross revenue can be, when it is reported alone. A company can grow revenue every month while retention collapses, because new acquisition is outrunning the leak. Paired with net revenue retention it stops flattering and starts informing.

Why is email open rate still on dashboards?

Because the pixel still fires and the tool still reports a percentage. Apple's Mail Privacy Protection loads that pixel on the reader's behalf, so a large share of recorded opens correspond to nobody looking at anything. The number is still generated, which is not the same as still being measured.

How many metrics should a team actually track?

Few enough that each one has a named decision and a named threshold written next to it. The count matters less than the rule: if you cannot say what you would do differently at a different value, it does not belong on the dashboard.

Who coined the term vanity metrics?

Eric Ries popularized it in 2009, defining the category as numbers that make you feel good without offering clear guidance for what to do, and proposing split-tests, per-customer views, cohort analysis and keyword segmentation as the replacements.

Is pageviews always a vanity metric?

It depends on the business model, not the number itself. For a publisher, pageviews are the unit sold to advertisers, so treating them as vanity is a mistake. For a SaaS built on seats, a pageview spike says nothing until you know whether it came from buyers, a stray Hacker News thread, or a scraper.

Why can't cumulative totals reveal a problem?

Total users, total signups, and total downloads only ever go up, so they cannot report a bad month even when one happens. Ries flagged this shape specifically, because a number that is incapable of falling can't inform any decision. The fix is splitting the total into a cohort rate, so a decline has somewhere to show up.

What should a vanity-leaning metric be paired with to become useful?

A counter-metric that gets worse if the main number is pushed dishonestly. Conversion rate pairs with refund rate, session duration pairs with task completion, and velocity pairs with defect rate. Without that partner, any metric that can rise during a collapse will eventually be used to hide one.

Is NPS a vanity metric?

On its own, NPS is a single lagging score built on a small sample with heavy rounding, which hides more than it shows. It becomes useful once you stop reading the number and read the verbatim comments behind it instead.