codecot

The Model Should Not Guess What It Doesn't Know

EXPLAIN Before You SELECT

I once asked ChatGPT to measure a twenty-second pause.

Twenty seconds later, it told me that twenty minutes had passed.

It had not measured the time incorrectly.

It had not measured anything at all.

There was no clock in its context. No timestamp before the pause, no timestamp after it, nothing from which twenty seconds—or twenty minutes—could be calculated.

But the interface looked like a continuous conversation. I had experienced the pause, so I instinctively assumed that the model had experienced it too.

It had not. It received another block of text and produced the most plausible continuation. Somewhere inside that continuation, a duration had to appear. “Twenty minutes” sounded believable enough, so twenty minutes became the answer.

I saw the same failure in a coding agent. A pull request had been merged the day before, but the agent had previously seen it while it was still open. When we discussed it again, the agent confidently continued treating the PR as active.

The repository knew it was merged. GitHub knew it was merged. The agent did not.

Worse, it did not know that it did not know.

These looked like two unrelated bugs: one involving time, another involving repository state. They were the same structural failure.

The model answered from memory about something memory could not reliably contain, then dressed the guess as truth.

A Conversation Is Not a Continuous Experience

A language model does not sit inside the conversation watching the world move. It receives text.

Depending on the surrounding system, that text may include timestamps, retrieved documents, tool results, repository state or nothing beyond the visible messages. The model can reason over whatever is there. It cannot directly observe whatever is not.

Between two messages, no time “passes” for the model unless the runtime tells it that time passed. A pull request does not merge inside its awareness. A stock price does not change. A deployment does not fail. A customer does not cancel an order.

The next inference begins with another snapshot of context.

This is easy to understand technically and surprisingly difficult to feel while using a chat interface. The interface manufactures continuity: one thread, one avatar, one voice, one apparently persistent intelligence.

The experience is continuous for us. The state underneath it often is not.

That becomes dangerous when the conversation moves from stable knowledge to volatile facts. An explanation of how Git branches work can survive in model memory; the current status of pull request 482 cannot. The concept of elapsed time is stable; the amount of time that passed during my pause does not exist unless somebody measured it.

The problem is not that the model forgot these facts.

It never had a reliable source for them.

Because language models are trained to continue, answer and resolve, they often fill missing state with something that fits. Plausibility quietly substitutes for observation.

This is why saying “the model lies” is both accurate and unfair. There is no malice in it, and perhaps no meaningful internal representation of deception at all. But from the user’s side, that distinction does not help much. A confident false statement presented as known truth is still a lie in the behaviour of the system.

Do Not Let the Model Decide From Memory What It Knows

The common response is to improve the prompt:

Ask for clarification when information is missing. Never make assumptions. Say that you do not know.

Those instructions help, but they do not solve the structural problem. Before the model can ask for missing information, it must first recognise that the information is missing.

That recognition is performed by the same system that already feels able to answer.

The coding agent did not think, “I need the current PR status.” It thought it knew the current PR status. Telling an overconfident system to request information when uncertain does little when the failure is false certainty.

The better solution is to separate determining what an answer requires from producing the answer itself.

Before generation, run a planning pass.

Not a hidden chain of thought. Not a polished essay about how the model intends to solve the problem. A small, structured, inspectable declaration of needs:

I think of it as EXPLAIN before SELECT.

In SQL, I can expose the execution plan before committing to the query. The value is not that the plan is always correct. The value is that it becomes visible. I can see the full-table scan, the ignored index or the obviously stupid decision before execution.

An AI system should expose the equivalent of a query plan for its answer:

This answer needs the live status of PR 482. The conversation contains an earlier status, but it may be stale. Fetch the PR from GitHub now. Verify state, merged_at and the merge commit. If the fetch fails, do not state whether the PR is open or merged.

That is not the final answer.

It is the answer’s dependency plan.

For the timing question, the plan should have failed immediately:

Elapsed time cannot be derived from the conversation. A start timestamp and an end timestamp—or access to a timer—are required.

No timestamp, no duration. The system should not be permitted to turn missing instrumentation into prose.

A planning pass is not free either. It costs a call, latency and tokens, and most exchanges do not need one—nobody should EXPLAIN a SELECT 1. Explain a concept, rewrite a paragraph, reason over evidence already on the table: when the answer is fully contained in the current context, planning is ceremony.

But notice who must not make that decision. If the answering model decides whether it needs a plan, we are back at the original failure: it will skip the clock precisely because it feels certain about the time.

The trigger belongs outside the answering model.

Volatility is the clearest signal: “now”, “today”, “how long”, “the latest”, “the current status”, or any reference to live system state should make planning mandatory. But volatility is only one class of external dependency. A private document, a repository, a permission boundary or a decision buried in project history may be perfectly stable and still absent from the context.

The broader rule is mechanical enough: if the truth of the answer depends on a source outside the verified context, assemble that source before generation. If the evidence is already present, do not add ceremony.

Pay where grounding is at stake, and do not let the thing being checked decide whether checking is required.

Assemble the Context, Then Generate

Once the needs plan exists, the model should not be responsible for pretending that the plan has been satisfied.

An orchestrator reads it and assembles the context.

Some dependencies belong to retrieval. A decision may live in an architecture document; a product constraint may be buried in a specification; a previous conclusion may need to be recovered from project history. That is knowledge context. RAG may be the appropriate mechanism.

Other dependencies are volatile or structured. The current pull-request state belongs to GitHub. The latest deployment belongs to the CI system. Today’s price belongs to a market-data source. That state should be fetched through tools—MCP or otherwise—whenever freshness matters.

The distinction matters more than the names of the technologies: retrieval reconstructs relevant knowledge, tools observe current state.

The orchestrator combines both into a grounded context and gives the model a narrower job:

Here are the verified facts. Here are the relevant documents. Here are the unresolved gaps. Now write the answer.

The model still reasons, explains, compares and synthesises. But it no longer has to impersonate a clock, a database, a repository or a monitoring system.

I am not demoting the model. I am moving it to the part of the pipeline where its strengths are real.

A language model is exceptionally good at turning heterogeneous evidence into a coherent human answer. It is much less reliable when asked to invent the evidence and compose the answer in the same motion.

Think in the orchestrator. Execute in the model.

The Plan Must Also Be Evidence-Bound

Adding a planning pass does not make the system correct. The planner is still a model. It can omit a dependency, select the wrong document or drift slightly into another plane while sounding perfectly methodical.

I have seen plans that were tidy, detailed and wrong.

This is where inspectability matters. The needs plan must be a first-class artifact, not invisible internal reasoning. When it is visible, I can ask a few blunt questions:

This is validation applied to the detection step itself. The detector does not get to remain a black box merely because it runs before the answer.

The same principle appears in semantic caching. A semantic cache says that two requests are sufficiently similar for one answer to serve both. That can save substantial work. It can also produce a beautifully written category error.

Similarity is not identity.

“How do I cancel my card?” and “Cancel my card” are close in language but not in intent. “Show my deployment status” and “Show the deployment status for another customer” may be nearly identical in wording but completely different in scope and permission.

Freshness and generation context create the same problem. An answer cached yesterday may be semantically perfect and operationally false today. The same prompt under a different policy, toolset or tenant is not necessarily the same request.

A cache hit is itself a claim: this old answer remains valid here.

That claim must be grounded too.

Honesty Before Economy

At this point, the architecture begins to resemble a cost-optimisation strategy: plan first, retrieve only what is needed, call tools selectively, reuse safe results and avoid sending the entire universe into the largest model.

Yes, this can save tokens.

That is not why I care about it.

I care because a system should not confidently state what it has not established.

The real enemy is the artificiality of the conversation: an interface that sounds present, informed and continuous while silently operating on partial, stale or imagined state.

When it says, “The PR is still open,” the user should not need to wonder whether it checked. When it says, “According to the specification,” there should actually be a specification behind the sentence.

Reliability is not merely producing the correct answer more often. It is preserving the boundary between what was verified, what was inferred and what remains unknown.

A serious system must be allowed to say:

I cannot determine that from the available context.

Or:

The only status I have is from yesterday, so it may be stale.

Or:

I found conflicting evidence and cannot resolve it safely.

Those are not product failures. They are signs that the product has stopped manufacturing certainty.

I would rather spend another tool call and receive an answer that deserves to be believed than save a fraction of a cent on a confident lie.

Economy is a side effect of honesty, never its substitute.

”Unimportant” Is Not a Model-Selection Strategy

The same mistake appears in model routing:

Important task: use the strong model. Unimportant task: use the cheap model and accept lower quality.

I dislike this rule more every time I encounter it.

“Unimportant” does not mean “allowed to be bad.” I do not want a broken page because it is only a weekend project, malformed data because it belongs to an internal tool, or a false answer because the question was casual.

That is not engineering.

It is a wall of shame that we agreed in advance to build.

There are legitimate reasons to use a cheaper model, but they belong to two regimes.

The first is need: the budget, latency envelope, hardware limit or throughput requirement makes the stronger model unavailable.

The second is proof: evaluation has shown that, for a well-defined class of work, the smaller model produces equivalent results within the required quality boundary.

Not “the task feels easy.” Not “nobody will care.” Not “the output is probably good enough.”

Necessity or calibration—not a guess about importance.

This does not mean every token should pass through the largest available model. The mistake is treating the whole task as atomic.

Decomposition Dissolves the Dilemma

Consider turning a product specification into a reviewed pull request. As one giant prompt, it looks expensive: understand the specification, inspect the repository, choose an implementation, write code, update tests, interpret failures, repair the change and review the final diff.

A router may label the whole object “hard” and send everything to one strong model. Or it may decide the project is low priority and send everything to a cheaper one.

Both choices start from the wrong unit.

The task is not one decision. It is a graph of different decisions.

Some nodes are narrow and checkable: extract acceptance criteria, locate files matching a symbol, classify a compiler error, compare a response with a schema, verify whether tests passed. A small model may be fully sufficient because the input is constrained and the result can be checked immediately.

Other nodes require broader judgement: resolve an ambiguous requirement, choose between architectural trade-offs, recognise that a locally correct change violates a system boundary, or synthesise findings into one coherent implementation.

That is where the stronger model earns its cost.

Decomposition does not lower the quality bar. It creates the conditions under which cheaper execution can meet it. Small steps create checkability; validation nodes catch errors near the point of introduction; integration nodes test whether the pieces agree. The strongest model is focused on synthesis and judgement instead of spending its capacity on every mechanical operation.

Quality is not sacrificed for economy.

It is protected by structure, and economy follows.

There is an honest limit. Decomposition has overhead and is absurd for trivial work. It pays off when a task is sufficiently complex, expensive, stateful or risky.

There is another limit that matters more: synthesis remains hard.

Correct pieces do not automatically form a correct whole. Five sensible modules can still produce a Frankenstein system. Ten accurate summaries can still become an incoherent decision. A collection of locally valid patches can still violate the architecture when combined.

Synthesis is often the least checkable, most intellectually demanding and most expensive node.

Decomposition does not eliminate judgement. It concentrates judgement where it is genuinely needed.

The Task Was Never Atomic

This is already how my spec-to-code factory works.

A specification enters. The system decomposes it into steps. Different roles execute different parts. Outputs are validated. Failures enter repair loops. The pieces are synthesised into a reviewed pull request rather than an unexamined block of generated code.

The factory does not run one model on everything and hope that intelligence averages out.

It drops, checks and synthesises.

The model choice lives at the level of the decomposed node, not at the level of the original request.

And this brings me back to the first failure.

A model answering from memory asserts a fact it has not grounded.

A router accepting worse output because a task seems unimportant asserts that quality is sufficient without proving it.

A system presenting retrieved, cached or generated content as true asserts validity it has not checked.

These are not three separate design mistakes.

They are one mistake:

letting the machine assert what it has not grounded.

The discipline that fixes them is also one: assemble and verify before you assert.

Fetch the state. Expose the plan. Inspect the dependencies. Decompose the work. Validate the pieces. Use the strongest judgement where synthesis genuinely requires it.

And when the evidence is absent, let the system stop.

The model can write the final sentence.

We still have to build the conditions under which that sentence deserves to be believed.

That is the last mile.

← All notes