There is an ad running at the moment for an AI assistant wired into a small business. A cafe owner explains what he did: gave the assistant access to the point-of-sale system, photographed the menu, uploaded it, and asked which items were not selling. It came back with a list. He cut them. He says it feels like living in the future, and he is right that something real happened. The integration is genuine, the answer was correct, the work took minutes instead of an evening.
Then I thought about the cake.
Every decent cafe has one. Expensive, slightly absurd, sitting under glass. People walk past it for months. They think not today. And then one day someone has had a good day, or a terrible one, and they come in specifically for that cake, and they bring a friend, and they order two coffees as well.
On the report, that cake sold four slices this quarter. It is at the bottom of the list. It is exactly what an optimizer removes.
The item was never there to sell itself
The cake has a job, and its job is not to be bought. It sells the visit.
Retail has known this for a century and has names for it. The expensive item is an anchor: next to it, the ordinary espresso stops looking like a splurge and starts looking sensible. The odd item is a reason to visit, and a reason to mention you to someone else. A menu with a hundred coffees tells a customer that whoever runs this place cares about coffee, and that customer will still order the Brazilian, and will come back, because the hundred were the proof of care.
Cut the anchor and the average basket falls. Cut the reason to visit and the visits fall. Neither loss appears in the row you deleted.
Why the mistake feels safe
This is the part that makes it dangerous rather than merely wrong.
The cost of removing the cake arrives months later, distributed across every other line, and attributed to nothing. Footfall drifts down a few percent. The regulars who came once a quarter simply stop appearing, and nobody notices an absence that never had a name. Meanwhile the decision looked free at the time: the row was empty, and removing an empty row cannot possibly hurt.
Decisions whose costs are delayed and unattributable are the ones an organisation makes over and over, because nothing ever comes back to teach it otherwise.
The shape of the error
The owner asked: how much of this did we sell? The assistant answered correctly. Then the answer was used to settle a different question: should this be on the menu?
Those are not the same question, and nothing in the exchange marks the switch. This is the general form, and it has very little to do with how good the model is:
An optimizer answers the question it was given, and never tells you the question was wrong.
Worse, it is not neutral about purpose. Ask a system to improve a business and give it sales data, and it does not proceed without a goal: it adopts the goal the data implies, which is almost always short-term revenue per line item. Nobody chose that goal. It arrived with the dataset.
So the failure mode is not stupidity. It is competence aimed at something nobody meant.
The human job is upstream
The instinct is to put a person after the model, to check the answer. That is necessary but it is not where the value is. By the time the recommendation exists, the framing is already fixed, and a reviewer is mostly reduced to agreeing with arithmetic.
The work is before the numbers, and it is a single sentence the owner has to be able to say: what is this place for.
A quiet place people come to sit in and a fast stop on the way to work are both good businesses, and they are optimized by opposite numbers. The first lives on dwell time, reasons to return, and the strange item under glass. The second lives on seconds per order and seats per square metre, and for that one, cutting the slow-moving menu really is correct. The same recommendation builds one business and kills the other, and the data cannot tell you which you are running.
Once the purpose is stated, metrics become answerable again. You can ask whether removing the cake changed the average basket and the return rate, rather than whether the cake sold. That is still a number. It is simply a number chosen by a human who knew what the place was for.
A test that takes ten seconds
Before accepting a metric, ask two things about it.
What decision would this number change? If no decision hangs on it, it is decoration, and decoration in a dashboard is worse than nothing, because it consumes attention that belongs to the numbers that matter.
What purpose does it serve? Say the purpose out loud. If you cannot, you are about to optimize toward whatever the data happens to encourage, and you will get there efficiently.
This is the same discipline that makes agentic work survivable, and I run my own factory on it: the specification exists before the run, not because writing specs is virtuous, but because a system with no stated purpose will infer one, and the inferred purpose is always the one that is easiest to measure.
Numbers are not the enemy
None of this is an argument for intuition over measurement. Intuition is how the cake gets kept for sentimental reasons long after the neighbourhood moved away. Measurement is how you find out that the cozy place you think you run has a median dwell time of six minutes.
The argument is narrower: a number is only meaningful against a purpose someone chose, and choosing that purpose is not a task you can hand to the thing that reads the numbers. Give it away and you still get decisions, quickly and cheaply, in a direction nobody picked.
That cafe is going to be efficient. The question is whether anyone will want to sit in it.