On a platform I help run as fractional CTO, PR volume roughly quadrupled over two months once the team leaned into AI-assisted development. That’s the part everyone talks about. The part nobody talks about is what it does to review: the bottleneck moves from writing code to judging it, and a human reading every diff line-by-line stops scaling long before the merge queue does.
My answer wasn’t to trust one AI reviewer. It was to make two of them argue, and referee.
The pipeline
Every non-trivial PR goes through the same sequence:
1. The first reviewer: an automated review bot. Wired into the repo, it reviews every PR automatically and leaves inline findings with confidence scores. It’s fast, consistent, and has no ego. It’s also frequently, confidently wrong — which is fine, because it’s not the last word.
2. The counter-review: a second model, told not to trust the first. I run Claude Code against the same PR with an explicit instruction: do not rely on the first bot’s review at all. Form your own view of the diff, then compare. The independence matters. If the second model reads the first review before the code, it anchors — it confirms findings instead of generating them. Told to start cold, it routinely surfaces things the first reviewer missed and, just as usefully, contradicts findings the first reviewer was sure about.
3. Persona passes, when the PR warrants it. For risky changes I’ll run additional passes with different hats: review this as a CTO (architecture, blast radius), as QA (what test is missing), as the end user (what does this actually do to the product). It sounds like a gimmick. It isn’t — the same diff genuinely yields different findings under different questions. The QA pass finds the untested branch; the user pass finds that the error message is nonsense.
4. All of it in a worktree. Reviews run in a separate git worktree, not my working checkout. Reviewing a branch shouldn’t disturb whatever I’m actively building, and worktrees make “check out the PR, run it, poke at it” free.
5. The referee: me. The output of all this is a pile of findings with overlapping, conflicting claims. My job is to decide which are real, which matter, and which are noise. That’s the part that doesn’t delegate — but it’s a much better use of a lead’s hour than reading raw diffs. I’ve even found myself replying to the first bot’s comments to argue its confidence score down, which is an odd sentence to type, but the audit trail of why a finding was dismissed turns out to be worth keeping.
What it has actually caught
Two examples, described generically because they’re from client work:
- An SSO authorization code consumed before it was validated. A homegrown single-sign-on flow marked the one-time code as used before checking the things that make it safe to use. The counter-review flagged the ordering. That’s a real vulnerability class, in code that had already passed a first review — human and machine.
- A race that allowed duplicate billing. A cost-tracking change charged users from a webhook without an atomic claim on the billing event, so a retried webhook could bill twice. The first bot flagged it; the counter-review confirmed it with a concrete interleaving. Concurrency bugs are exactly where machine review shines, because nothing about a race looks wrong on any single line.
Findings like these are why the pipeline exists. Either would have been an incident. Both were caught pre-merge, at the cost of a few minutes of compute.
What it gets wrong, honestly
If I only told you the wins, this would be the AI-generated filler this site pledges not to publish. The failure modes are real and you have to manage them:
Confident wrong findings. Both reviewers will occasionally assert a bug that does not exist — a “race” the framework already serializes, a “missing null check” on a value that can’t be null. The confidence score does not distinguish these from real findings. This is precisely why the referee role exists and why findings must come with quoted evidence you can check in the code. I verify before I forward; a false finding sent to a teammate costs trust the pipeline can’t afford.
Style noise. Left unconstrained, reviewers pad reports with naming nitpicks and speculative refactors. I prompt against it — findings must describe a failure scenario, not a preference — and dismiss the rest without guilt.
Agreement is not evidence. When both models flag the same line, that’s a strong signal. But when both miss something, they often miss it for the same reason: the bug lives in context neither was given — a business rule, an operational constraint, a promise made to a customer. The pipeline reviews the diff; only the humans know what the diff was supposed to do.
The economics
The pipeline costs a few minutes of wall-clock and pennies of compute per PR. What it buys is that my review time is spent almost entirely at the judgment layer: is this finding real, is this design right, should this merge. On a solo project, that’s a nice-to-have. On a platform with multiple contributors shipping AI-accelerated PRs daily, it’s the difference between review being a gate and review being a rubber stamp — because the honest alternative at that volume isn’t “careful human review of everything.” It’s skimming.
Two flawed reviewers plus disagreement plus a referee beats one reviewer of any kind. That’s not a new idea — it’s how good human review already worked. The models just made it cheap enough to do on every PR.