Follow-up from my post yesterday. Anthropic detailed its risk-based approval approach to code review last week. Changes per day jumped 12x and most important post-coding bottleneck got removed!
The agent infra enables them to do so: production incidents and merge reverts are evals for their code review agent. I am sure they use this same idea for their coding agent too.
Natural question: what would it take for an enterprise engineering team, without a frontier lab’s infinite token budget, to do the same?
𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼 𝟭: 𝗦𝗶𝗹𝗼𝗲𝗱 𝗮𝗴𝗲𝗻𝘁𝘀
Engineer 1 codes locally with Claude Code.
Engineer 2 reviews and merges the PR after Claude Code reviews it in GitHub Actions.
Engineer 3 investigates a production incident using Claude Code locally. A caching change omitted the customer ID from its key, exposing one customer’s data to another.
A patch fixes the bug but teaching all agents the lesson takes more work.
An engineer must connect the incident to the coding agent trace and code review in CI; reconstruct the failure; and propose better skills. Then test whether the agents use that skill as expected.
Claude Code’s skill-creator plugin and eval commands help. The engineers still own the infrastructure setup and coordinated updates.
𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼 𝟮: 𝗔 𝘀𝗲𝗹𝗳-𝗶𝗺𝗽𝗿𝗼𝘃𝗶𝗻𝗴 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗳𝗮𝗰𝘁𝗼𝗿𝘆
Engineer 1 still codes locally. Engineer 2 still uses the CI reviewer. Both setups make their agent traces available to the cloud software factory that also has self-improvement agents. An incident-response agent investigates the leak. Engineer 3 confirms the root cause after this investigation.
Then the self-improvement agents take over:
Evaluator agents analyze the coding and review sessions against the incident, then build repeatable benchmarks.
Healer agents propose improvements and test accuracy, cost, latency against the benchmarks.
An engineer reviews the evidence and approves the exact changes that go live.
In this scenario, evals and healing run autonomously. Behavior changes remain engineer-approved.
𝗪𝗵𝗲𝗿𝗲 𝘁𝗵𝗲 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗮𝗹𝗽𝗵𝗮 𝗰𝗼𝗺𝗲𝘀 𝗳𝗿𝗼𝗺
Both teams learn from the incident. In the first, engineers organize that learning each time. In reality, most teams don't have time for this since they are already working on the next feature. In the second, the factory carries the process through to an approval decision.
That is the alpha worth pursuing: autonomous learning from production failures to get better coding and review. Self-improving engineering is not only for frontier labs anymore.
