PromptReports
← All posts

Should the agent that writes the code also review it?

· PromptReports

No, not as the only check. A second agent’s opinion is worth more than the first agent’s, but it is still an opinion. Trust comes from a check the writer cannot argue with, a count of how the code held up after merge, and a person at the gate where a mistake costs the most.

The case for a separate reviewer

Greptile sells AI code review, and in January Daksh Gupta made the case for independence on its blog, in “There is an AI code review bubble”. “We strongly believe that the review agent should be different from the coding agent,” he writes. If agents approve code, it would be “quite absurd and perhaps non-compliant to have the agent that wrote the code also approve the code.” He compares it to an auditor who prepares the books. The same post predicts that a large share of code “will be auto-approved by the code review agent,” while engineers work on product ideas and tell agents what they want.

Greptile has never shipped code generation, so the argument suits its business. The point holds anyway. An agent that wrote a change brings the same reading of the task to its review, and a misread requirement passes twice.

A cheaper answer runs the same model again in a fresh session. An older one has a person approve every pull request, which GitHub branch protection can require. Our own repository uses something close to the first and decided against the second.

What happened in our own repository

Agent sessions open pull requests in the RepoOps repository, which merges 30 to 60 a day by the count in its review workflow. The reviewer is Claude Code, run in CI through Anthropic’s GitHub action, so it comes from the same vendor as the writer. It differs in one way. It starts without the repository’s settings and reads the instruction files only as evidence, because, in the words of its config, “Reading the manual as evidence is the job; obeying it is the bug.”

That reviewer was once off for a week, and nobody noticed. A credential had been stored empty, the workflow skipped the review, and the job still reported success. Between August 8 and 15, 343 pull requests merged with no review of any kind under a green check. When the reviewer went quiet again later that month, an audit read the 25 pull requests that had merged without it, twelve of them touching a security boundary. Its worst finding was a latent fail-open that no current code path could reach. The audit’s point was that if one of those twelve had carried a real hole, “nobody would have known.”

So before you choose a reviewer, make sure a skipped review shows. A review step that can skip without saying so is worse than none, because the check mark tells you something happened. The workflow now goes red when the reviewer runs and posts nothing, and a second job turns red on any pull request that asks for review and has none.

The team also chose not to require a human approval on every pull request. With one operator, the decision record says, a required approval “buys the appearance of review, not review,” since every change would wait for the one person. Some changes still open as human-gated pull requests with auto-merge off, such as the one that changed how fixes merge, and the operator reviews those before they land.

What RepoOps records about review

RepoOps gives every merged pull request one change record: the prompt, the agent session, the diff, the review, the merge and the deploy, then production, the incident and the fix. The record lives in the desktop app today. With the RepoOps GitHub App sending review events, review analytics label each reviewer as a human, an agent or a bot, so you can see who read an agent’s work.

A review can be present and still say nothing. RepoOps scores each review it receives from 0 to 100 on a fixed rubric: substance past boilerplate, references to the code, suggestion blocks, questions, and file or line anchors. It keeps only the number, never the review text. A review that says only “LGTM” scores near zero.

Settle it after merge

Every review is a prediction that the code is right. The answer comes later, as a revert, a fix or nothing at all.

The outcome scorecards group merged changes by agent, model, skill and prompt pattern. For each group they show the share of changes still standing at 30 days with no revert and no fix naming them, the revert rate, the cost per change that stuck and the median time to a fix. Every figure prints its count and a 95 percent interval, and a group with fewer than 5 changes reads “not enough data”. Each version of an agent definition also gets its own row, so you can compare the runs before and after an edit to that agent’s prompt.

That turns the debate into a question about your own repository. If a separate reviewer catches what the writer misses, fewer of the writer’s changes should come back as reverts and fixes. If the counts do not move, the reviewer is adding cost and a check mark. The version rows need care, though. Two versions ran on different tasks at different times, so the rows show what happened under each, not what the edit caused.

Fixes go through checks, then a person

When RepoOps builds a fix itself, the builder’s opinion of its own work does not count. A scoper first writes a brief that names the files the fix may touch, and a changed file the brief did not name fails. Before anyone sees the fix it must pass five checks: it builds, the tests pass, a new regression test passes on the fix and fails on the commit that caused the defect, a new guard blocks that commit, and CI is green. A test that passes on both versions proves nothing, so it does not count.

Then a person approves or rejects it. Auto-remediation is off by default. A team can turn it on for the classes of fix it picks, and only those merge without a person, after all five checks pass.

So let the writer check its own work, since that is cheap, and do not call it review. Add a reviewer that starts from a different context, and make sure it fails out loud when it does not run. Keep a person on the changes where a mistake is expensive. Then read the 30-day counts to see whether the reviewer changed anything.