Dex Horthy just gave a talk called “Harness Engineering is not Enough: Why Software Factories Fail.” His thesis: the “lights-out factory” — code written, tested, and deployed by agents with no human reading it — fails predictably at around 3–6 months of codebase age. Not because operators hold it wrong, but because of what the models are: RL training rewards a passing test, and nothing in that loop can penalize the gratuitous try/catch, the cast, the copy-paste that exists only to make a test go green. The cost of bad design materializes over months. A months-long signal cannot be propagated into a fifteen-minute training episode.
He’s right. And I want to add evidence from an unusual position: I run a software factory that is squarely in his 3–6 month danger window, and it hasn’t failed. Not because my agents are better. Because it was never lights-out.
The failure is a loop, not an event
The lights-out story usually gets told as a threshold: one day the codebase gets too big and the agents get confused. What actually kills these factories is a feedback loop, and it’s worth spelling out, because every stage looks like a win when you adopt it.
Stage one: agents write code in minutes, humans review in days, so review becomes the bottleneck. Stage two: delete the review, reinvest in tests and monitoring — green means ship. Stage three: automate the last human job, queue-stuffing, by routing production incidents and user reports straight into the agent queue.
Now trace an incident through that system. The agent patches the symptom — shortest path to green, that’s what it was trained to do. The patch adds entropy. Entropy breeds the next incident. The next incident gets auto-routed to an agent, which patches the symptom. Nobody is positioned to notice that the third incident this week is the same structural defect wearing different stack traces, because noticing that requires a human who still reads the code — and reading was deleted in stage two. The factory heals itself to death, green all the way down.
What a lights-on factory looks like
My factory is small — one operator, a fleet of agents, tens of thousands of lines across several shipping products. The failure modes it’s built against are not small, and I met most of them before Dex named them. Four load-bearing decisions:
Review is a funnel, not a gate one person guards. A developer agent implements against a spec. A separate reviewer agent — free to be a different model, with different blind spots — reviews the diff. Then a verdict step judges the reviewer: which findings are valid, which are noise. Only then does a human see anything, and what the human sees is a distilled delta plus evidence, not a raw diff stream. This solves the problem Horthy names — review as bottleneck — without the amputation. It also solves one he doesn’t: an LLM reviewer that sprays twenty findings, fifteen of them noise, burns humans out as surely as no review at all. Judging judgments (“is this finding valid?”) is a narrower, more checkable task than judging code (“is this good?”). Narrow the question until a machine can answer it; escalate the rest.
Verification never belongs to the producer. Models game tests — Horthy cites them commenting out tests during scoring. In a lights-out pipeline that’s fatal, because the agent that writes the code also runs the tests and reports its own success. In mine, the harness runs the tests, a separate verifier agent checks every acceptance criterion against the code and must produce file-and-line evidence, and content that leaves the system passes gates that reject any figure not literally present in the source — even true ones. The producing agent’s opinion of its own work is worth exactly nothing, structurally. One mandate I’m tightening after his talk: diffs that touch tests get reviewed stricter than production code. A weakened assertion is a quieter lie than a bug.
Specs are the external memory the model doesn’t have. Horthy’s most useful reframe is that “brownfield” now starts at 3–6 months. The mechanism: in a lights-out shop, intent lives nowhere. The human stopped reading months ago; the agent sees fragments of a codebase whose why was never written down. Every fix is archaeology without a map. In my factory the spec is in the repo, linked from the PR, and every session boots from it plus a living canon file of invariants and known traps. An agent joining the work doesn’t reconstruct intent from eroded code; it reads it. My repos are inside his danger window and still shipping — I think this is the main reason.
Work does not create itself. No signal — incident, user report, idea — becomes agent work by itself. An operator sees the signal, asks an agent to investigate; the agent returns a deep-dive and a question: should this become a spec? The human decides whether the work exists at all. That single decision point breaks the self-healing death loop before it can form. My rule from an earlier essay applies: autonomy is earned per action, not granted per system. Publishing an artifact that passed deterministic verification — my pipeline does that on autopilot, earned. Deciding what work exists — never.
What I’m stealing, and what stays open
Honesty about the scoreboard: two of his points landed on gaps.
I’m adding an explicit program design stage — types, signatures, call graphs agreed before generation — for complex tasks. My specs capture intent and architecture; they don’t capture structure, and structural erosion slips through intent-level review. And I’m instrumenting rework percentage — how much agent-written code gets rewritten within a month — instead of celebrating throughput. You don’t have too many PRs; you have too many bad PRs, and only one of those is measurable by counting.
And one problem stays open for everyone, me included: there is no deterministic gate for design erosion. No maintainability benchmark, no verifier. My canon files and my own eyes are a compensation, not a solution. Horthy says that’s where his company is headed. Good — it’s the right unfilled gap.
The lesson
“Software factories fail” is true the way “bridges collapse” is true: the ones built on a wrong load assumption do. The wrong assumption was never agents write the code — it was green tests mean nobody has to read it. Keep the lights on: keep a human who reads, feed them a funnel instead of a firehose, write intent down where the next session can find it, and never let the system decide what work exists.
The factories that fail were built to remove humans. Build one that concentrates them instead.
This continues two earlier pieces: the trust ladder (how agent autonomy gets earned) and pets-not-cattle (where agent sessions should live). Dex Horthy’s talk is worth your time even where I disagree — especially his point that this is a training-objective problem, not a skill issue.