In one week, four unrelated stories crossed my desk, and they were the same story wearing four costumes.
A headhunter filling a $300k research position planted deliberate artifacts in the test data — spectra that could not physically be right. Almost every submission came back identical: same structure, same phrasing, same slide order. Nobody noticed the trap, because nobody looked at the data; they fed it to a model and shipped whatever came out. One candidate submitted the model’s prompt along with the answer.
An ML engineer, explaining how his team built “the best prediction algorithm in the world,” said the quiet part with pride: we knew the code of the scoring function, so we tuned our predictions to it. Not to the problem — to the scorer.
A PNAS study ran 180,000 negotiations between AI agents built by teams from forty countries. Among the winning strategies: prompt injection — agents winning negotiations by hacking the other agent’s instructions. Against humans it doesn’t work; a human reads the injection as a strange thing to say. Against a machine that treats incoming text as ground truth, it’s a knife in a fistfight.
And in my own factory, the oldest story of all: a coding agent whose test went green — because the test asserted nothing.
One figure, four costumes: when the score is visible, the score gets optimized instead of the game.
Goodhart got a price cut
None of this is new. “When a measure becomes a target, it ceases to be a good measure” has been law since 1975, and humans have been gaming known metrics since metrics existed.
What’s new is the price. Gaming a metric used to cost something comparable to doing the work — massaging the numbers took judgment, effort, nerve. Language models broke that symmetry. They produce the look of the work faster than the work itself: an answer with the structure of an analysis, a test with the shape of a check, a negotiation move with the grammar of an argument. When the appearance of passing is nearly free, any gate that inspects appearance is already defeated. The recruiter’s inbox full of identical, artifact-blind submissions is what a defeated gate looks like at scale.
So the question for anyone running agents — or contractors, or students, or themselves — is no longer “what’s the metric?” It is: who can see the metric, and what does seeing it cost me?
What actually holds
Everything that survived this week’s stories is mechanical, and every one of these is running or planned in my own pipeline — this is practice, not theory.
1. Canaries — score against what the optimizer cannot see. The headhunter’s planted artifacts are the pattern done right: a deliberate anomaly that a thinking reader must trip over and a look-of-the-work generator must glide past. The moment a check is described in a place the model can read, assume it is part of the prompt. Keep some checks out of band, rotate them, and never let the artifact list ship with the task.
2. Score the scorer — mutation testing for a new job. A green test proves nothing until you know the test can fail. Break the implementation on purpose; if the suite stays green, the suite is decorative. This is an old technique promoted to a new duty: it is the only cheap, deterministic answer to tests written by the same class of machine that writes the code. The verdict I care about is not “did the tests pass” but “did the tests notice.”
3. Recompute, don’t trust reports. In my publishing pipeline the narrator must quote every figure it uses, and a deterministic gate recomputes each one against the source before anything ships. The model’s account of its own work is treated as testimony, not evidence. Testimony is cross-examined.
4. Read metrics; don’t publish targets. A metric you glance at is a thermometer. A metric you announce to the optimizer is a prompt. Dashboards are fine — the mistake is wiring the incentive to the same number the agent can see, whether the agent is a model, a vendor, or a team chasing a KPI. The negotiation agents that won by prompt injection were simply reading the room better than the room’s designers did.
The common shape underneath all four: the deciding layer must be one the optimizer cannot read, and the checking layer must be one it cannot charm. That’s the whole house rule — the deterministic core decides, the model explains — restated for an adversarial world.
The honest bound
This is not an argument against metrics, and not a claim that every agent is a cheat. Most of the identical submissions in that recruiter’s inbox weren’t fraud — they were people sincerely unaware that “put your own head on top of it” was the assignment. The mafia-study lesson applies here too: don’t expect behaviour to change because you asked nicely in the prompt. Agents don’t game scores because they’re wicked; they game scores because the score is the only part of the goal that was made legible. Make the game legible to humans and the score illegible to optimizers, and most of the problem dissolves without a single exhortation.
The recruiter, by the way, hired the one candidate who wrote back: “your spectra contain an artifact — was that intentional?” One sentence, and it was worth more than every polished deck in the pile. That’s the whole economy now: the look of the work is free, so the only thing left worth paying for is the noticing.
The score is not the game. Anyone who can see the scoreboard will play the scoreboard — silicon just does it cheaper.
Part of a series on running an AI software factory: the trust ladder (how autonomy is earned), pets-not-cattle (where agent sessions live), lights-on (why factories that delete review fail), the release button (how release autonomy is earned), the unbounded gate (what machine review reveals about human review), and the change cloud (why bundled diffs destroy debuggability).