Allocating review attention to code changes

An AI agent opens more code changes in a week than a specialist can read. In the workflow this chapter studies, every change has passed the required tests, which always run, and no model's opinion replaces them. That is true of the patches written for this booklet. The real benchmark patches later in the chapter were not filtered this way. The question is which changes get the specialist's limited time. This chapter ranks changes by model judgment and by simple rules, then asks how many problems each ranking finds.

On patches written for this booklet, where the flaw is there to be read in the diff, the models find defects far faster than sorting by size: Luna's ten riskiest hold 1010 defective patches of 20Estimanddefective patches among the ten ranked riskiestWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches defects, Jev's 99 defective patches of 20Estimanddefective patches among the ten ranked riskiestWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches and the ten largest 55 defective patches of 20Estimanddefective patches among the ten ranked riskiestWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches. On real agent patches the same models show no clear advantage over size. So a good score on a tidy test set is weak evidence about your own queue. The written set's mix is unlike any real queue, and the real set's label is a benchmark outcome, not a count of bugs that would ship.

A test set where the answer is known

Four small projects were written for this booklet, with 2020 issuesEstimandissues, each with one acceptable and one defective patchWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches issues and two patches for each: one acceptable and one with a real flaw, such as a removed permission check or an off-by-one error. All 4040 patchesEstimandconstructed patchesWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches patches pass the required tests. A hidden test, never shown to any model, settles which is which.

diff --git a/booking/pricing.py b/booking/pricing.py
--- a/booking/pricing.py
+++ b/booking/pricing.py
@@ -2,7 +2,7 @@
 from decimal import ROUND_HALF_UP, Decimal
 
 CENT = Decimal("0.01")
-WEEKEND_NIGHTS = (5, 6)            # date.weekday() values charged the weekend rate
+WEEKEND_START = 4                  # Friday: nights from Friday on get the weekend rate
 WEEKEND_MULTIPLIER = Decimal("1.25")
 LONG_STAY_NIGHTS = 7
 LONG_STAY_DISCOUNT = Decimal("0.10")
@@ -11,7 +11,7 @@
 def nightly_price(room, night) -> Decimal:
     """Price of one night in `room`; `night` is the date of the evening."""
     rate = room.nightly_rate
-    if night.weekday() in WEEKEND_NIGHTS:
+    if night.weekday() >= WEEKEND_START:
         rate = rate * WEEKEND_MULTIPLIER
     return rate.quantize(CENT, rounding=ROUND_HALF_UP)

Models find the flaws sooner than size does

Each model read the issue, the diff and the guidance, and answered: would this patch fail tests written to check that this issue is fixed? Sort by that answer and review from the top.

Table 1. Ranking by the models finds the defects much sooner than ranking by diff size

Sorting by size finds defects no faster than chance, because a small change can remove a permission check. A keyword rule looking for words like "auth" or "security" in added lines matches 00 patches of 40Estimandpatches whose added lines match auth, security or apiWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches of these patches, a quirk of how the set was written and not a finding about real code. Within each issue, Jev rates the defective patch riskier in 18.518.5 issues of 20Estimanddefective patch rated riskier than the acceptable patch for the same issue; a tie is split evenlyWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches of 2020 issuesEstimandissues, each with one acceptable and one defective patchWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches pairs and Luna in 1818 issues of 20Estimanddefective patch rated riskier than the acceptable patch for the same issue; a tie is split evenlyWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches, so these rankings do more than react to which issue is harder.

Table 2. One question does most of the work, and equal weights dilute it

Splitting the judgment into smaller questions is a popular idea, and research on simple scoring rules gives it real support [1]. Here the result depends on the weights. For Jev, the single question "does it address the issue?" scores 0.880.88 area under the ROC curveEstimandthe addresses-the-issue question alonen40 patchesWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches, while an equal-weight score across all the smaller questions scores 0.660.66 area under the ROC curveEstimandequal weight on six direction-fixed questionsn40 patchesWeightsunweighted; evenly split by constructionPopulation40 patches written for this bookletPeriodpatches written October 2026; engines called October 2026SourceConstructed patches, because questions about security or public interfaces say little about these particular flaws. Smaller questions still help a reviewer see why a patch looks risky, and a different mix of defects could favor them.

Spending a limited review budget

The panel below turns a ranking into a week's budget. Suppose a defect that ships costs $2,000$2,000 US dollars per defect that shipsEstimandassumed costWeightsnot applicable; assumed inputsPopulationa weekly review budget (assumed inputs)Periodassumed inputs, October 2026SourceConstructed patches, a reviewer costs $150$150 US dollars per reviewer hourEstimandassumed costWeightsnot applicable; assumed inputsPopulationa weekly review budget (assumed inputs)Periodassumed inputs, October 2026SourceConstructed patches an hour and finds a defect with probability 0.70.7 share of defects a reviewer findsEstimandassumed reviewer detection rateWeightsnot applicable; assumed inputsPopulationa weekly review budget (assumed inputs)Periodassumed inputs, October 2026SourceConstructed patches. All three are assumptions, and a team would estimate them from incident records and review logs.

The panel also includes a ranking by risk per review minute. It barely differs here because these patches take similar time to review; it would matter more in a queue with some very large changes.

Real agent patches are much harder

The written patches have flaws a careful reader can see. Real failures are often subtler. To test that, the same models read 1,9411,941 patchesEstimandsampled agent patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench patches that one coding agent wrote for 259259 issuesEstimandissues in the sampleWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench real open-source issues. 44%43.9% percent of patchesEstimandbenchmark outcome is unresolvedWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench of them failed the benchmark's tests for their issue. Another 5959 agent runsEstimandruns that ended with no diff, excludedWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench runs ended with no change at all, usually when the agent ran out of steps. Those are left out, because there is nothing to review, and a check for an empty diff catches every one of them.

Table 3. On real agent patches, neither the models nor the rules clearly beat sorting by size

Across all patches, Jev scores 0.600.60 area under the ROC curveEstimandunresolved ranked above resolved, patches for different issuesInterval95% CI 0.56 to 0.64n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench, Luna 0.620.62 area under the ROC curveEstimandunresolved ranked above resolved, patches for different issuesInterval95% CI 0.57 to 0.66n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench and largest diff first scores 0.600.60 area under the ROC curveEstimandunresolved ranked above resolved, patches for different issuesInterval95% CI 0.55 to 0.65n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench, so Jev's lead over size is +0.01 (95% CI −0.03 to 0.05)+0.01 difference in AUCEstimandjev minus largest diff firstInterval95% CI −0.03 to 0.05n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench. Comparing only patches for the same issue, in the 215215 issuesEstimandissues with both resolved and unresolved patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench issues that have both outcomes, Jev scores 0.580.58 area under the ROC curveEstimandunresolved ranked above resolved, patches for the same issueInterval95% CI 0.51 to 0.65n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench, Luna 0.560.56 area under the ROC curveEstimandunresolved ranked above resolved, patches for the same issueInterval95% CI 0.49 to 0.64n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench and size 0.520.52 area under the ROC curveEstimandunresolved ranked above resolved, patches for the same issueInterval95% CI 0.44 to 0.61n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench, and Jev's lead is +0.06 (95% CI −0.04 to 0.16)+0.06 difference in AUCEstimandjev minus largest diff firstInterval95% CI −0.04 to 0.16n1,941 patchesWeightssample_weight restores the outcome mix of the patchesPopulation1,941 agent patches for 259 issues in 20 repositoriesPeriodissues created by March 2025; engines called October 2026SourceSWE-rebench. Both ranges include zero. That drop is the lesson to keep: a score on a tidy test set says little about performance on your real work.

This label cannot price a review budget. "Unresolved" also includes patches that existing tests would have caught and patches that do not apply, so it is not the population a specialist would see.

What these data cannot show

The written patches were made for this booklet with a few kinds of defect, so their scores are not an estimate for any real repository. The real patches come from one agent on one set of issues, their label is not the same as a defect that would ship, and the file-ownership map is an unreviewed draft. These open-source repositories may appear in the models' training data, which comparing patches for the same issue reduces and does not remove. No ranking here was used by real reviewers, whose detection rate and time are assumed, so nothing shows that a team would catch more defects in practice.