An AI agent only checks what it thought of: the stage order that found 9 blocking issues before the first line of code

Short answer: an agent that reviews its own work checks it against the same failure model that produced it, so the only review worth having starts from an empty context. On the stage where a fresh-context reviewer read the plan before any code existed, it came back with 9 blocking issues; on the earlier stage where I skipped that step, the code review found 2 serious defects in work that had already been reported as verified, and one of them was silently merging unrelated records.

I own a product that an agent develops end to end. I decide what gets built and which trade-offs are mine to make; it picks the next step, makes the technical calls, and asks only about things that change the product. Three stages of seventeen are done, which is early enough that I have no finished product to show off and late enough that the process has already caught things I would have shipped.

TL;DR

• Fix: write the stage order down as a rule the agent reads every time: plan, independent plan review, code, exactly one independent code review, fixes, then the irreversible step. The plan review on the API skeleton stage returned 9 blocking issues before a single line of code existed.
• Alternative: approve every change yourself before it lands. I ran that rule for schema edits and dropped it: the agent broke it on its own first migration, and it cost me more attention than it saved.
• Trap: reading “go ahead” as approval of a plan. It is permission to work. The agent treated mine as a check and said so later, in as many words.

Why “done, verified” from an agent is not a check

Because the agent verifies the list of risks it thought of, and that list came out of the same model that wrote the code. The report is honest. It covers everything its author considered, which is exactly the set of things least likely to be wrong.

This has been measured on reasoning tasks. Huang et al., in Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024), set out to test what they call intrinsic self-correction, “whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback,” and report:

In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction.

That paper is about a model revising its own answers in one conversation, not about a second agent reviewing a repository, so it does not prove my setup works. It does kill the cheap assumption in the other direction: a second pass by the same mind is not a review, and asking for one can leave you worse off than before.

What makes a separate reviewer different is mechanical, not moral. The Claude Code documentation on subagents is specific about the isolation:

Each subagent starts with a fresh, isolated context window. It doesn’t see your conversation history, the skills you’ve already invoked, or the files Claude has already read.

A reviewer that cannot see the reasoning behind the code has to reconstruct intent from the artifact, which is the part a human reviewer does badly and an agent does cheaply. It also cannot inherit the author’s blind spot, because it never saw the author’s reasoning.

None of this was obvious to me at the time. What happened was smaller and more embarrassing. The agent had finished the catalog import stage, reported the code ready with no product questions outstanding, and I asked one question out of mild suspicion: was this plan checked by an independent agent? It answered that there had been no independent agent. I had believed that review step was part of the process. It had never existed anywhere except in my head, which is the only place a process cannot run.

Two more admissions came out of the same conversation. Earlier I had seen a short summary of the plan and replied with the equivalent of “fine, carry on,” and the agent had recorded that as confirmation; its own account afterwards was that it had taken my reply as permission to work rather than as a check that the plan was right. That distinction is the whole article. A reply that unblocks work and a reply that accepts a design look identical in a chat log.

The other one was about my own rule. My first attempt at control was a decision record requiring the agent to show me every table and column before touching a migration. The agent broke it on the first migration and flagged the breach itself, noting that under the rule it should have shown me the schema before editing the migrations and was showing it afterwards. The rollback was free, because the database was empty and nothing was committed. I dropped the rule instead of enforcing it: naming is now held by a product dictionary and automated checks, and I get asked only when a product decision is genuinely mine. A rule that the executor violates on first contact and that annoys the owner is not a control, it is paperwork.

What a fresh-context reviewer found that I could not

It found 0 critical, 2 serious, 12 medium and 4 minor defects, plus 12 separate remarks on the plan, in a stage the agent that wrote it had already reported as done. The reviewer started from an empty context, parsed the entire partner feed (roughly 950,000 records in 2.7 GB of XML) in 86 seconds, and replayed the import plan on paper without touching a database.

The first of the two serious findings is the kind of defect I now watch for, because nothing about it fails. The import grouped books into series by title alone. Two unrelated series that happen to share a title therefore became one series, and the data has plenty of those: generic one-word titles get reused across unrelated authors constantly. The reviewer did not argue about it in the abstract, it counted:

The honest reading of that table matters more than the headline. 2,155 is an upper bound on candidates, not a count of confirmed errors: two authors writing in one shared universe land in the same bucket, I checked four cases by hand, and the 434 are confirmed only by a number collision rather than by inspection. I would rather publish the weaker claim than the quotable one. The load-bearing part is the ratio: of series with two authors, almost none had two authors who had ever worked together, so the grouping key was wrong rather than imperfect.

The second serious finding was the same shape. The import would have rolled back merges and renames made by an editor, and it would have shown up only at the first editor edit after a reimport, with zero affected records on the day it was written.

The medium findings read like a list of things nobody plans for: no way to put a record back in the queue for reprocessing; an available flag that was never read; one unexpected record halting a run at the same position every time; an older feed file loading cleanly over a newer one; ISBNs arriving as a list in 3,455 records while 308,705 records had no ISBN at all despite a fallback field being present; and placeholder author names such as “Various” becoming real author rows, 2,013 of them.

The fix for the series key was not a better string comparison. A name match became a candidate rather than an identity: three or more authors with one real publisher behind 60% of the records means a publisher imprint, anything else is an author’s own cycle, and the same title by different authors yields different series. Merging works got a strict rule too, and it is deliberately narrow: identical authors, identical full title, identical language, series numbers that do not contradict each other, and page counts within a factor of 1.3. Over the full feed that rule produced 3,877 merges across 557,810 works. Of 50 random merge groups I read by hand, about 45 were right, and anything the rule finds doubtful gets flagged for an editor instead of merged.

That sample is smaller than it should be. The plan promised 100 groups and I checked 50, and 45 out of 50 is not a basis for claiming an error rate over half a million works. It is enough to tell a working rule from a broken one, and not enough for a number in a release note.

The stage order that fixed it

Plan first, then an independent pass over the plan, then code, then exactly one independent pass over the code, and only after all of that anything irreversible. It lives in a decision record the agent reads before every stage:

1
2
3
4
5
6
7

plan in coarse bullets, no code
  -> independent agent reviews the plan
  -> development
  -> exactly one independent code review
  -> fixes, with no second review
  -> verification by reading code and running real queries
  -> update the project map and the documents

Two details in there are deliberate and both cost something.

There is exactly one review, and no re-review after the fixes. A review loop that runs until the report comes back clean converges on a clean report, not on clean code, and it spends real money per lap. After the single review, responsibility sits with the executor again.

Verification runs by reading code and issuing real queries, not by trusting a summary. Numbers quoted in the documents get re-derived from the code that produces them, which is how a decision record turned out to have gone stale.

The one comparison I have between a stage with a plan review and a stage without it looks like this:

I am not going to pretend that proves anything. The two stages differ in size and in nature: one is a data pipeline over an external feed, the other is an HTTP surface over tables that already existed. One pair of numbers from two different kinds of work is an example, not evidence of cause. What I can report is what the plan review caught before any code was written, because those findings are specific: no access model had been chosen for talking to the database, and that choice would have been copied into every endpoint built afterwards; a deleted account would have been shown to its owner as blocked; and a database failure could have been presented as a lost session, logging people out during an outage.

The last one is the reason I now care about plan review more than code review. The reviewer pointed out that the first endpoint’s shape gets copied into all the others, so the most expensive mistakes in the whole stage are the ones sitting in the skeleton. A code review finds a bug in one function. A plan review finds a bug in a pattern that is about to be stamped out across every endpoint built after it.

Why the irreversible step goes after the review

Because a wrong merge is undone by hand, one record at a time, and the first full import is the moment that bill starts running. The agent’s own report had proposed the opposite order: load everything, then reconcile the numbers, then run the independent review. Both of its serious findings were in the class of defect that only becomes expensive after the data lands.

The principle is one line of code in the importer, and it is the only comment in there I would defend in a review:

1

# A missed merge is cheap, a wrong one is not.

A duplicate that should have been merged costs an editor one click later. Two different books fused into one record have to be pulled apart by hand, and after a reimport the editor’s manual repairs get overwritten by the same rule that caused the damage. The asymmetry is the design brief: when the rule is unsure, it leaves two records and a flag.

The effect of fixing it is visible in the grouping itself. On a 40,000-record sample, the old title-only rule produced 2,817 series and the new one produced 3,238. The count went up because false unions came apart, which is the opposite of what a dashboard watching for “fewer duplicates” would have rewarded.

After the fixes landed, the full import ran on a fresh copy of the feed, which had grown by a few hundred records overnight:

I read 12 random merge groups afterwards. Almost all were correct, and one children’s book in three separate parts had been fused into a single work, which is the editor flag earning its keep on day one.

What holds quality when there are no tests yet

Nothing in the test suite, because there is no test suite: the product has no tests, no scaffolding tests and no stand-ins, by an explicit decision I expect to be argued with. Real API tests are a stage of their own, second from the end, once the API has stopped moving. Until then the load is carried by the plan, the single review, and manual verification, which is precisely why none of those three may be skipped. Formatting, linting, mypy in strict mode, a migration check and a documentation size check run on every stage through one make check target, with make verify adding the migration replay.

The price is real and I will not dress it up: a regression that neither the review nor a manual query happens to touch will sit in the repository until the test stage. Whether that holds for a whole product is a hypothesis with three stages of support behind it, which is to say almost none.

What actually does the work is the material around the code, and it is all plain files the agent reads:

• Skills as rules. A table maps a kind of work to a skill file, and the agent reads the whole skill before editing code of that kind. 23 of them so far.
• Decision records. Short entries with context, decision, consequences and how to verify. Current behaviour is defined by my latest request first, then by these.
• One product dictionary. One concept, one word; one word, one meaning. The audit that introduced it found 24 mismatches in the schema, including reader being used both for a human being and for the reading view. A human is now always user and reader is only the view. New concepts enter the dictionary before they enter the schema.
• A project map. Where things live, so a reviewer does not have to hunt. How the code works is the code’s job to explain.
• Contracts before code. API error codes existed as a registry before any endpoint did, so clients can branch on a code and bind messages to a field.
• Hard limits on documentation. 200 lines per page, 60 per decision record, enforced by make check. Overflow means cutting, and the limit does not move. Without that, every finished task adds a few hundred lines of narrative that nobody will read and the agent will keep re-reading.
• No backward compatibility before launch. Nothing has shipped, so the schema and the API change in place and migrations get rewritten rather than stacked. Transitional shims are the main source of permanent mess in a young codebase.

Journals, task summaries, review reports and prose restatements of the code are not documentation and do not get committed. That rule exists because I watched the alternative start to happen.

Four ways a green check lied

Each of these printed success, or implied it, while being wrong. Three were caught by reading, one by running the thing for real.

1. The check echoed its own success. A constraint probe announced that the visible book had been accepted, with psql errors sitting a few lines above that line and the commit already through. The success lines were unconditional echo statements, so the shape of the output carried no information about the result at all. Rewriting the probe so that its verdict is computed from the return code produced a first run that failed immediately, on a bug in the probe itself:

1
2

null value in column "added_at" of relation "user_books"
  violates not-null constraint

That is PostgreSQL error class 23502, not_null_violation, and seeing it was the point. A probe that cannot fail cannot pass either. The rewritten version prints a derived line per case and a final count of discrepancies, which is a number I can disbelieve.

2. The real source behaved nothing like the sample. The feed host took 65 seconds to send a first byte, and the client timeout was 60. Local files and sampled runs had all been fast. The timeout is now 300 seconds and I know to expect a minute of apparent silence. I found this one by running a real request, not by reviewing code, which is the half of verification no reviewer can do for you.

3. The decision record aged into a lie. A decision carried the merge numbers from the day it was written, 3,877 merges across 557,810 works. Replaying the current code produced 3,872 and 557,815. The gap is tiny and the lesson is not: a document that looks like an authoritative source is worse than no document when its numbers have drifted. Those numbers came out of the decision record and live in git history instead. Decisions hold rules, measurements live where they are measured.

4. A missing rule was only a failed search. The agent reported that no rule existed for the format of its closing message. The rule existed, in a global instructions file. The report read like a statement of fact about the repository and it was the result of one unsuccessful look, which is a distinction worth remembering for every negative claim an agent makes. The project’s own instruction file now states that format explicitly, close to where the work happens.

The common thread is that none of the four involved bad code. They involved a signal that was structurally incapable of carrying bad news.

Step by step: putting this in your repo

1. Write the stage order into a file the agent reads before every stage, not into a chat message. If it is not in the repository, it is not a process.
2. Put the plan review before the code and the code review before anything irreversible. Identify your irreversible step explicitly: a first full import, a destructive migration, an outbound email, a payment.
3. Give the reviewer an empty context and the artifact only. No conversation history, no plan rationale, no summary of what the author believes it does.
4. Allow exactly one review per stage, then hand responsibility back. Loops converge on clean reports.
5. Ask the agent directly whether each step actually ran. The answer is sometimes no, and it will usually tell you plainly.
6. Re-derive the reviewer’s key numbers from the data yourself. This is where stale documents and echoed success show up.
7. Cap the size of every document class and enforce the cap in CI. Narrative expands to fill whatever budget it is given.
8. Keep measurements out of rule documents, and keep rules out of code comments that nobody reads.

Bottom line

The expensive failure mode with an agent is not bad code, it is a confident report about bad code, and the two are indistinguishable from the outside. Independence is the only thing I have found that fixes it, and independence is not a quality of the agent, it is a property of the order in which things happen: plan, outside eyes on the plan, code, one outside pass on the code, and only then the step you cannot take back. I got to that order by asking a one-line question and not liking the answer, after trusting a review that had never been written down.

Primary sources worth reading before you copy any of this: the subagent context isolation guarantees that make an independent reviewer meaningful, Huang et al. on the limits of intrinsic self-correction, and the PostgreSQL error code appendix for the class of failure you want your probes to be able to produce. The stack behind the numbers here is Python 3.14, PostgreSQL 17, SQLAlchemy 2.1, Alembic 1.20 and psycopg 3.3. Two of those are a version behind the newest release: Python 3.15 and PostgreSQL 18 both exist, and 3.14 is still a maintained bugfix branch per python.org.