Engineering on donchev.is

Quality walls: the bottleneck is verification

Thu, 08 Oct 2026 00:00:00 +0000

In September we merged about 10,900 pull requests. In January it was about 2,500. Same people: then and now, roughly 97 people merge code and roughly 80 approve it. What changed is that most of them now work with an agent all day, and the agents got good enough that a typical engineer merges 10 PRs a week instead of 4.

Earlier this year I wrote that writing code got cheap and reviewing it didn’t. This post is about what we did once that stopped being a prediction.

The numbers

Human-authored PRs merged across all our repositories, bots excluded, in two 14-day windows:

	Jan 12-25	Sep 7-20	Change
PRs merged	1,111	3,683	3.3x
People who merged at least one PR	97	97	same
PRs per author per week	5.7	19.0	3.3x
Median author, PRs per week	4.0	10.5	2.6x
Median PR size (lines changed)	52	145	2.8x
Median time to first human review	25 min	48 min	1.9x
People approving PRs	80	80	same
Approvals per approver	14.4	41.2	2.9x
Approvals by the busiest approver	145	263	1.8x

Two things to get out of the way. First, this growth is our engineers and their own agents (Claude Code, Codex, Cursor). Cody, our background agent platform, authored about 2% of September’s merged PRs. Second, the Claude Co-Authored-By trailer is on roughly half of the human commits in our longest-lived repos now, and that’s a lower bound.

The row I keep looking at is the busiest approver: 263 approvals in two weeks, about 26 per working day, on PRs that are almost three times bigger than in January. That’s a lot of careful reading to ask of one person on top of their own work. When volume grows 3x and the number of reviewers stays flat, something has to give.

So the question for this half of the year was: what does a human reviewer need to look at, and what can we verify without them?

Write it down, then enforce it

The first answer was boring and it came from the earlier posts. Everything an agent or a reviewer needs to know has to be in the repo. In August we turned our workspace of about 90 repos into something an agent can navigate: one short map file at the root (a check fails if it grows over 130 lines), a docs folder, and a page of core beliefs, the invariants every repo has to respect. Two lines from that page describe the approach:

Prefer strengthening the enforcer over writing more prose.

Knowledge that only exists in Slack/Docs/heads does not exist for an agent.

The second line I already believed last year. The first one is what this post is about.

Quality walls

Our earlier review skills and CodeRabbit checks were good at conventions: wrong error type, missing validation, a transaction in the wrong layer. What they didn’t do well was the harder stuff: is this change actually tested, can this money calculation lose precision, can a tenant see another tenant’s data.

Quality walls are our answer. A wall is a rule with a structure an agent can investigate. This is one of the 92 generic ones:

{
  "id": "QW-GLOBAL-042",
  "title": "Not Enough Tests",
  "check": "Whether changed behavior, branches, failures, or invariants have evidence in executable tests. Trace the changed outcome through existing tests at the owning boundary ...",
  "reject_when": "A concrete new or changed observable outcome can regress without any focused test failing. Identify the changed outcome and a specific regression that existing assertions would miss ...",
  "severity": "major",
  "remediation": "Add the smallest tests that prove the missing observable outcomes.",
  "exclusions": "Generated code, declarative configuration, behavior already proven at a more appropriate boundary ..."
}

There are 222 rules in the central catalog today: generic ones like this, plus modules for authorization, audit logging, Postgres queries, decimal math, data ingestion and so on. Repos can add their own local rules on top. They can’t disable central ones.

The most important part of the design doc is the list of what they are not:

They are not a linter, a replacement for tests, or another PR-review persona. The evaluator uses each rule as an investigation lead, verifies the actual code path and consequence, and reports only concrete violations.

“A rule is a lead, and a finding requires a verified consequence.” A wall can’t fail a PR because a line looks suspicious. It has to show the regression that would get through.

A few other decisions I like:

A PR can’t change the rules that evaluate it. The policy comes from the merge base, so the head can’t loosen a rule while it’s being reviewed. A rule change only becomes active after it merges.
The same rules are meant to run on main too. The design adds a weekly repair run that checks everything that landed since the last run and opens at most one fix PR, which a human merges.
Local rules graduate. The ones that prove useful get promoted into the central catalog through a normal PR. Both of these loops are designed, but only partly rolled out so far.
Waivers are explicit: /cody waive <rule> <reason>, and only a code owner who isn’t the PR author can do it.

And the most honest paragraph in the whole doc:

Evaluation completeness and judgment correctness are different guarantees. […] It does not prove that an LLM’s judgment was correct. A defect class becomes deterministic only when it has an executable detector.

The agent decides, Cody owns the check

Quality walls and PR review run as Cody lanes, and each lane reports a GitHub Check Run. Early on we had a nasty bug in that, straight from the spec:

A successful Codex turn therefore becomes a successful Check Run even when the PR-review skill found blocking defects

The agent could do its job, find real problems, and the check would still go green, because “the agent run succeeded” and “the review passed” were the same signal. The fix was to split the responsibility:

The agent decides whether the review passed, failed, or was inconclusive. Cody owns the corresponding Check Run from deterministic registration through verified terminal publication.

The agent returns one bounded result tied to the commit it reviewed. Cody maps it to a check conclusion, re-reads the PR head, updates the exact Check Run and reads it back before it calls it done:

Agent result	Check conclusion
`pass`	success
`fail`	failure
`inconclusive`, missing, partial, cancelled	action required
PR head moved on since the review	neutral

Everything that isn’t a clear pass fails closed. A model that times out doesn’t get to approve anything.

Two smaller rules came out of the same thinking. Lanes that only apply to some paths (a docs review, say) still always publish their check, as a successful “Not applicable” for that exact commit, and if Cody can’t read the changed files, that fails closed too. And a stale blocking review can only be cleared when a person, not a bot, asks for a re-review on the same commit and Cody approves it. In the words of the requirement: “Only Cody’s own verdict can be replaced, and only on the head it reviewed.”

One policy file, about 100 repos

Between September 27 and October 1 we made these checks required on main: Cody PR Review, Cody Quality Walls, and the requirements and specs reviews for PRs that touch those documents. The branch rulesets are generated from one fleet file in our skills repo, which lists every repo and its lanes. New repos get five lanes automatically. Changing the policy for 100 repos is one reviewed PR.

The PR description is evidence

As the diffs got bigger and less hand-written, the description became more important than the diff. Every PR now uses the same body, and a quality wall checks it:

## Summary

<what changed and why, in one or two sentences>

## Evidence

- **Before:** <screenshot/output/failing test run>
- **After:** <screenshot/output/passing test run, or per-acceptance-criterion evidence>

## Merge Danger

**Door:** one-way or two-way, and how to walk it back or recover
**Blast Radius:** <affected services, consumers, or environments, and what could break>

Evidence has to be something that was actually observed on the current head commit; the skill says to never fabricate a failing run. Merge Danger is the part I’d steal for any team. A two-way door (revert and redeploy) deserves a different review than a one-way door (a migration that drops a column). It tells the reviewer where to spend their attention.

Shift left, but CI wins

Waiting for CI to tell you about a quality wall you could have caught locally is a waste. Since October 1 our shared Claude Code plugin ships a local precheck that runs the same PR review rubric and the same quality walls, pinned to the same versions CI uses. A SessionStart hook installs a git pre-push guard, and a PreToolUse hook stops the agent from running git push or gh pr create without a fresh precheck. This is what engineers see when they try anyway:

Cody local precheck: Run /cody-precheck, fix findings and commit, then rerun.
Explicit waiver: /cody-precheck --skip "reason".

It’s deliberately easy to skip. From the skill: “Local records are advisory and editable; CI remains authoritative.” The local check is there to save time. CI makes the final call.

What we don’t know yet

Plenty. A few things I want to be upfront about:

We can’t yet measure how good the reviewers are. We have labeled eval corpora for quality walls and simulated PR scenarios for PR review, and the eval README says it plainly: “No live baseline exists yet.” My favorite line in that README is “Do not weaken an expectation or relabel a fixture merely to improve a score.”
Disputes are a signal, not a rate. Every night we collect findings that engineers pushed back on (“inapplicable”, “pre-existing”). The most disputed rule so far is the one about PR descriptions, which tells you something about engineers too.
The review machinery became its own bottleneck. In September we had reviewer pods waiting for CPU and third-party review rate limits holding up merges. When review is required, review capacity is production capacity, and it has to be run like production.
Reverts didn’t go up. They stayed at or below about half a percent of merged PRs in every window I sampled this year. That’s good, but it isn’t proof that quality held. It only shows that nothing obviously broke.

Where the bottleneck is now

A year ago the slow part of shipping was writing the code. In the spring it was reviewing it. Now it is verifying it: proving with evidence that a change does what it says and doesn’t do what it shouldn’t. That’s a better place for the bottleneck to be. Verification can be automated piece by piece, written down as rules, measured, and argued about in PRs. “Somebody senior looked at it” can’t.

As always, if you have any questions or remarks, feel free to ping me on twitter @bobby_donchev.

Writing code got cheap. Reviewing it didn't.

Sun, 05 Apr 2026 00:00:00 +0000

In December I looked at what a year of AI coding assistants did to our pull requests. The conclusion was that authors got faster and reviewers didn’t, and that the review queue is where it would hurt first.

That happened faster than I expected. This quarter Claude Code went from “a few people use it” to the default way a lot of our engineers work, and our merged PRs per month went from about 2,500 in January to 3,900 in February and 4,400 in March. Same number of people.

Same people, more PRs

Two 14-day windows again, human-authored PRs only, across all our repositories:

	Jan 12-25	Mar 15-28	Change
PRs merged	1,111	1,558	+40%
People who merged at least one PR	97	97	same
PRs per author per week	5.7	8.0	+40%
Median time to first human review	25 min	34 min	+36%
PRs reviewed by a bot	28%	65%
People approving PRs	80	80	same
Approvals per approver	14.4	19.3	+34%
Approvals by the busiest approver	145	184	+27%

Claude Code adds a Co-Authored-By trailer to commits by default, so I counted those too. The share of human commits on main with a Claude trailer went from 11% in January to 27% in February and 34% in March. That’s a lower bound, since Copilot and Cursor don’t leave a trailer and some people switch it off.

For the first time the review numbers moved the wrong way. Time to first review went from 25 to 34 minutes. That doesn’t sound like much, but it’s a median, and it moved within a single quarter with the same people on both sides.

The reviewer’s question changed

A year ago when I reviewed a PR, I could assume the author had written every line and could explain every line. The question in my head was “is this correct?”

Now a good chunk of the diff was typed by an agent, and the author’s job was to steer it and check it. Most authors do that well. Some days, honestly, nobody has read every line. So the reviewer’s question became “does anybody understand this change?”, and that is a harder question to answer from a diff.

At the same time a lot of review comments are boring and repetitive. Wrong error type. Transaction opened in the repository instead of the service. A DTO mapper where we pass proto types straight through. Missing config validation. A human shouldn’t spend their limited review attention on that, and at today’s volume they can’t.

So this quarter was about splitting review into the part a machine can check and the part only a human can.

Docs where the agent reads them

The first step was the least exciting one. If a convention only exists in someone’s head, the agent can’t follow it and the AI reviewer can’t check it.

In January we added AGENTS.md and CLAUDE.md files to our main repos, and we started a separate docs repo where an agent keeps the code documentation up to date after every merge. The proposal put it like this:

Instead of passing diffs to a single AI call, we use Claude Code CLI as an autonomous agent

The README of that repo opens with a quote I won’t reproduce in full, but it ends with “Mr. Claude, it’s too much.” That’s the spirit.

Review rules as skills

In late January we created a shared skills repo. Every convention we kept repeating in reviews became a small review-* skill: review-domain-errors, review-cls-transactions, review-connectrpc-only, review-no-dto-mappers, review-type-not-interface, review-config-validation, review-unit-tests and a few more. Thirteen at the end of January.

The skills come with two hooks for Claude Code. One runs after every edit and blocks on lint, type and test failures. The other runs when the agent wants to stop, and spawns one reviewer per principle in parallel against the changed files:

#!/bin/bash
# review-principles.sh - Spawns one Claude agent per principle/skill in parallel
#
# This hook runs on Stop (session end) and reviews all changed files
# against each skill defined in skills/review-*/SKILL.md
#
# Each principle gets its own agent for focused, accurate review.

One agent per principle sounds wasteful. The idea is that an agent looking for exactly one thing is more reliable than one agent juggling a list of thirteen rules.

An AI first pass on every PR

On the PR side we rolled out CodeRabbit in February. One of the first things we changed was turning off its auto-approve. From the commit message:

PRs should still require a human approver. CodeRabbit should review and flag issues but not count as an approving reviewer.

In March we moved its configuration into one central repo. Two parts of that config do most of the work. It reads our AGENTS.md and CLAUDE.md files as review guidelines, so the docs from step one double as review context. And our most common review comments are now pre-merge checks in error mode:

knowledge_base:
  code_guidelines:
    enabled: true
    filePatterns:
      - "**/AGENTS.md"
      - "**/CLAUDE.md"
      - "**/.claude/rules/"
      - "**/.cursor/rules/"
      # ...

reviews:
  pre_merge_checks:
    title:
      mode: "error"   # conventional commits with a ticket number
    description:
      mode: "error"
    custom_checks:
      - name: "Domain Errors Over Plain Errors"
        mode: "error"
        instructions: "Business logic must throw domain error classes (...) instead of plain Error ..."
      - name: "CLS Transactions Over Manual db.transaction()"
        mode: "error"
        instructions: "Database transactions must use the CLS-based @Transactional() decorator ..."
      - name: "Config Validation With Joi"
        mode: "error"
        instructions: "Any new or modified NestJS configuration registered via registerAs() must have a corresponding Joi validation schema ..."

(There’s also poem: true in there. We’re keeping it.)

By the end of March, CodeRabbit reviewed about 60% of human PRs.

What it didn’t fix

The numbers above are after all of this. Time to first review still went up, and the busiest reviewers still approve more every month. The AI first pass removes the boring comments. Someone still has to understand the change and own the approval.

What I take from this quarter:

Generation is solved enough. The next tool worth building is one that helps us verify code.
Conventions belong in the repo, written for an agent. The humans benefit just as much.
A rule that a machine can check should never cost a human reviewer a minute.
The approve button is the scarcest resource we have. We should treat it that way.

Next we want agents doing work in the background, not just in an engineer’s terminal. More on that soon.

As always, if you have any questions or remarks, feel free to ping me on twitter @bobby_donchev.

A year of AI coding assistants: what changed in our pull requests

Sun, 14 Dec 2025 00:00:00 +0000

In 2025 a lot of our engineers started using an AI coding assistant. Copilot in the editor for most people, Cursor for some, and since September Claude Code in our biggest monorepo. Everybody felt faster. I wanted to know what actually changed, so I pulled every merged pull request from GitHub for two comparable two-week windows, one in January and one in October, and compared them.

Short version: the people writing code got a lot faster. The people reviewing it did not.

The numbers

Human-authored PRs merged across all our repositories, bots excluded, in two 14-day windows (January 12-25 and October 12-25):

	Jan 2025	Oct 2025	Change
PRs merged	375	844	2.3x
People who merged at least one PR	57	85	1.5x
PRs per author per week	3.3	5.0	+52%
Median PR size (lines changed)	47	82	+74%
Median time to first human review	29 min	27 min	same
PRs with an AI review comment	0%	36%
People approving PRs	43	62	1.4x
Approvals per approver	9.4	13.5	+44%
Approvals by the busiest approver	36	107	3x

Monthly merged PRs (bots included this time) went from about 1,300 in January to about 2,300 in November.

Part of that is growth. The team grew by roughly half and we started a handful of new services. But per person we merged 50% more PRs a week, and those PRs were bigger. I can’t pin all of that on AI, but it is the shape I expected.

What we set up

Nothing fancy. Most of it was individual engineers picking up tools, and a few things we set up for the whole team.

In September we added Claude Code to our largest Go/Node monorepo. Anyone can mention @claude in a PR comment and it picks up the request:

name: Claude Code
on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]

permissions:
  contents: read
  issues: write
  pull-requests: write

jobs:
  claude:
    runs-on: ubuntu-latest
    if: contains(github.event.comment.body, '@claude')
    steps:
      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

Next to it we wrote a CLAUDE.md with the title “Rules for Agents”. It is around 340 lines and reads like this:

**MUST** rules are enforced by CI; **SHOULD** rules are strongly recommended.

- **BP-1 (MUST)** Ask the user clarifying questions.
- **C-1 (MUST)** Follow TDD: scaffold stub -> write failing test -> implement.
- **C-7 (SHOULD NOT)** Add comments except for critical caveats; rely on self-explanatory code.
- **T-3 (MUST)** ALWAYS separate pure-logic unit tests from DB-touching integration tests.
- **D-1 (MUST)** Use SQLC for type-safe database access with proper Go structs.

Our newest service got Cursor rules in its repo. And by October, Copilot’s PR reviewer and Cursor’s review bot were leaving comments on about a third of our human PRs.

Writing rules for agents is writing rules for humans

The most useful thing that came out of this was the rules file, more than any code the assistants wrote.

We had conventions before. They lived in people’s heads, in old Slack threads and in review comments that got repeated every other week (“please don’t open a transaction in the repository”). Writing a CLAUDE.md forced us to put them in one place for the first time. I’d argue that file is as useful to a new engineer on their first day as it is to the agent.

The second lesson is in the first line of that file. A rule in a markdown file is a suggestion. The agent follows it most of the time and so do humans. If a rule really matters, a tool has to enforce it: a linter, a type check, a CI step. Our MUST rules are the ones we can check in CI, and in practice those are the ones that hold.

Review didn’t change, and that’s the problem

Look at the bottom of the table. Time to first review stayed flat at under half an hour, which sounds great. But the number of approvals per approver went up 44%, and the busiest approver went from 36 approvals in two weeks to 107. That’s more than ten a working day on top of their own work.

Authors scale with tools and hiring. Reviewers only scale with hiring. The AI review bots help a bit: they catch typos, a missing null check, the occasional real bug. They don’t approve anything, though, and they shouldn’t. Somebody still has to understand the change and press the button.

PRs also got bigger, 47 lines to 82 at the median. A bigger diff that the author partly didn’t type is also harder to review.

What I’m watching in 2026

If the assistants keep getting better at this pace (and it looks like they will), the volume on the writing side will keep growing and the review queue is where it will hurt first. So that’s where we want to put our effort next year:

Put our documentation where the agents read it, and keep it up to date with an agent instead of hoping someone does it.
Turn more of the review comments we keep repeating into rules a machine can check, so human reviewers can spend their time on the questions only a human can answer.
Keep measuring. It took me an afternoon to pull these numbers and I learned more from them than from a year of “it feels faster”.

I’ll write a follow-up once we have a few months of data.

As always, if you have any questions or remarks, feel free to ping me on twitter @bobby_donchev.