The number went up. Did the product get better?

Here’s a made-up design review, though you’ve probably sat in one like it. Two versions of an onboarding flow are on the screen. One asks people to pick their interests before they see anything. The other lets them straight in. Someone opens the dashboard, points at activation, and the conversation is over in about a minute. The version with the higher number ships.

Nobody in that room did anything wrong. The team agreed months ago that activation is the number they steer by, and the test was run properly. What nobody asked is why the number moved. Did more people get to something useful, or did the flow just get better at producing the event we count as “activated”?

That gap, between the thing you count and the thing you wanted, is what Goodhart’s law is about. I think it’s one of the few borrowed mental models that changes what you ask in a review.

The sentence everyone quotes is not Goodhart’s

You know the line: when a measure becomes a target, it ceases to be a good measure. That wording comes from an anthropologist, Marilyn Strathern, in a 1997 paper about audits in British universities.

Charles Goodhart was an adviser at the Bank of England, writing in 1975 about money supply targets, and his own wording is drier: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

I like the original more, because of one word in it: pressure. A number that tracks something real can keep tracking it for as long as nobody is leaning on it. Start steering by it, attach a roadmap or a bonus to it, and the relationship it used to describe starts to come apart. The measure is the same on the day it becomes a target. The pressure on it is new, and a team has some say in how much of it there is.

Nobody has to cheat

The popular reading is about gaming: give people a target and they’ll find a way to fake it. That happens. Jeff Sauro and Jim Lewis at MeasuringU collected examples of what follows once a score like NPS has consequences for the people being scored, including employees at a large bank who deleted customer email addresses to avoid getting low scores from surveys.

The case I find more useful has no villain in it. In 2012 Ron Kohavi and his colleagues at Microsoft published a paper about experiment results that had puzzled them. In one Bing experiment, a bug caused users to see very poor search results. Distinct queries per user went up by more than 10%, and revenue per user went up by more than 30%. People couldn’t find what they wanted, so they searched again and clicked more ads on the way.

On a dashboard, that’s a great experiment. In front of a user, it’s a broken search engine. The authors’ answer was to stop treating queries and revenue as the goal and to optimise for sessions per user, on the reasoning that satisfied people come back more often.

I think designers are well placed to catch this version of the problem, because it lives in the interface. A metric is an event that fires when somebody does something on a screen. There are usually several ways to make that event fire more often, and only some of them involve the person getting what they came for.

What else would move this number?

That’s the question I’d bring to every review where a metric is deciding the outcome. Before you accept the winning variant, list the other ways it could have won.

Time on page goes up when the content is good, and also when the page is confusing. Steps completed goes up when the flow is clear, and also when the skip button got harder to see. Does your team write these down, or does the number get to speak for itself?

It doesn’t take long. I’d expect the usual answer to be that the variant won for the reason you hoped. Sometimes it’s the other answer, and then you’ve found it in a review instead of in a churn report six months later.

A better metric drifts too

The usual fix is to swap the number for a smarter one. It helps, for a while.

In 2012 YouTube wrote that its discovery features had been designed to drive views, which “rewarded videos that were successful at attracting clicks, rather than the videos that actually kept viewers engaged”. So its suggestions moved to watch time.

In 2018 Facebook rebuilt its News Feed ranking around what it called meaningful social interactions. That January Mark Zuckerberg told investors that changes made the quarter before had already reduced time spent on Facebook by roughly 50 million hours every day, so the company was openly giving up one number for another. In 2021 the Wall Street Journal reported, from internal documents, how the new score worked: a like was worth one point and a significant comment or reshare 30. The same article says the company’s own researchers found publishers and political parties reorienting their posts towards outrage and sensationalism. Outrage gets comments, and comments were now what the ranking paid for.

Both replacements were more thoughtful than what they replaced. According to that reporting, the second one still got optimised against. I think a product metric has a shelf life. It describes the product well on the day you choose it, and a little less well every quarter you spend pushing on it. A two-week test won’t show you that, for the same reason it won’t show you what a design does once people have learned from it.

This is not an argument against measuring

I keep seeing the law used to wave numbers away, usually by whoever just lost an argument to one. I don’t think it supports that.

Cedric Chin makes the case well in an essay on Commoncog. He leans on a list that Donald Wheeler credits to Brian Joiner: people under pressure to hit a target can improve the system, distort the system, or distort the data. The first one is the reason anybody sets a target. I read Goodhart’s law as a warning about the other two, which are often less work.

So the useful design question is which of the three your product makes easiest. If improving the real thing takes a quarter and nudging the event takes a sprint, I wouldn’t expect good intentions to hold for long.

What to bring to the next review

Write the metric’s definition next to the design. Skip the name and write down the literal event: which screen, which tap, counted how. I suspect this is where the problem often shows up, because “activated” can turn out to mean “saw the second screen”.

Pair every number you steer by with one that’s harder to fake. Sign-ups next to people still around after a month. Revenue next to sessions, as the Bing paper suggested. The second number should belong to somebody who is in the same room as the first.

Ask who is under pressure to move it, and how much. A number a team watches behaves differently from a number someone’s bonus depends on. If the pressure is high, I’d assume the proxy is already drifting and check it against real people more often. A few recorded sessions or support conversations will do.

You don’t need permission or tooling for any of this, only one extra question in the review: if this number went up and the product got worse, what would that look like? If nobody can answer, I’d hold the decision until someone can.