# About this booklet: report
Introduction of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/about-this-booklet. Table n of this report is figure n of the chapter.
## Identification
- Title: About this booklet
- Chapter: Introduction
- Question: What does this booklet cover, how are its numbers produced, and how should a reader use its interactive scenarios?
- Dek: This booklet explores the decision endpoints that major AI labs have recently released, an interface popularized by TypeSafe's Jev, by posing business and workflow decisions as typed questions and checking whether acting on the answers beats the simplest credible alternative.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- Authored examples. This page describes the booklet and reports no data; weights: not applicable; no data are sampled
- Canonical URL: https://vishalsingh.org/decisionai/about-this-booklet
- Report: https://vishalsingh.org/decisionai/about-this-booklet/report
- Markdown: https://vishalsingh.org/decisionai/about-this-booklet/report.md
- BibTeX key: singh2026aboutthisbooklet
```bibtex
@misc{singh2026aboutthisbooklet,
author = {Singh, Vishal},
title = {About this booklet},
year = {2026},
url = {https://vishalsingh.org/decisionai/about-this-booklet},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The dek and the opening prose cite no fact.
## Findings
This booklet is for people who build or buy systems that act on a model's judgment: MBA students who have studied classification, and practitioners deciding whether a decision model, a language model, a trained classifier or a written rule should sit behind an action. Its subject is the decision endpoints that labs such as OpenAI and Perplexity now offer, an interface popularized by [TypeSafe's Jev](https://typesafe.ai/). It asks one question in every chapter. When does a model's judgment improve the action that a person or a system takes?
### How a chapter is built
A case chapter starts from one decision with an owner and a fixed set of actions: contact a customer or not, route a complaint or escalate it, send a code change to a specialist, let an agent stop or make it continue. It states what evidence exists when the decision is made and what establishes a correct answer. It poses the judgment as typed questions, compares complete policies built on the answers, and prices them with costs that the reader can change. Where the data allow, a chapter shows a failure a reader can inspect, and each ends with what its data cannot show.
The comparison always includes the simplest credible alternative, such as a keyword rule, a trained classifier or a fixed step limit. When that alternative matches the model, the chapter says so, because a tie is a finding about whether the model is worth its cost.
### How to read the numbers
Every statistic in a chapter is printed from that chapter's data file. Hover over a number, or focus it with the keyboard, to see its unit, population, period, source and interval. Costs in a scenario are assumptions and carry the tag "assumed"; quantities computed from the data carry the tag "measured". The [reading guide](https://vishalsingh.org/decisionai/reference/reading-guide) explains the kinds of claims a chapter can make, and the [methods page](https://vishalsingh.org/decisionai/reference/methods) gives the rules every chapter shares.
No reader needs an API key. Every call to a model was made once and stored, so a chapter's numbers do not change when an engine does. Each chapter's methods name the engines and when they were called.
### What is finished
All seven chapters are drafts, revised after a review of their metrics on October 30, 2026. Their labels are records that already exist, such as star ratings, company responses and task graders, and no one rated anything by hand. The [changelog](https://vishalsingh.org/decisionai/reference/changelog) records each change.
Instructors can add `?mode=instructor` to a chapter's address to show teaching notes beside the sections they concern.
## Tables
The chapter has no figures.
## All estimates
The chapter has no facts.json.
## Definitions
The chapter marks no glossary term.
## Methods
This page reports no estimate. The rules every chapter shares are on the methods page.
### Where the shared rules are
Samples, weights, intervals, cross-fitting, refusals and the caching of engine calls are described once on the methods page, and each chapter's methods note gives its own sample, engines and run dates.
## What these data cannot show
This page describes how the booklet is built and how to read it. It reports no result, so it cannot show whether any engine or policy performs well.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/about-this-booklet/build.py (sha256 7fb9b7091c511d4746eaf753d0a98213c5df7b4b1adfcc7ab4b77d41ae5e02b5)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
| meta.json | 49afb3b4d16fc4197c82ec050aca93ff6217c238794f8e4a3e47ddc74076317b |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/about-this-booklet/report
- Report (Markdown): https://vishalsingh.org/decisionai/about-this-booklet/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/about-this-booklet/facts.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# Decision AI in applications and agents: report
Chapter 1.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents. Table n of this report is figure n of the chapter.
## Identification
- Title: Decision AI in applications and agents
- Chapter: 1.1 (Part I, From judgment to action; Case studies)
- Question: Where do bounded judgments with a fixed answer space occur in applications and agents, and what does each kind of output establish?
- Dek: An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- Authored examples. The bug-report example and its costs are written for this booklet; weights: not applicable; no data are sampled
- Canonical URL: https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents
- Report: https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report
- Markdown: https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report.md
- BibTeX key: singh2026decisionaiinapplicationsandagents
```bibtex
@misc{singh2026decisionaiinapplicationsandagents,
author = {Singh, Vishal},
title = {Decision AI in applications and agents},
year = {2026},
url = {https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| break-even probability, 1 minus delay cost over misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | 0.556 | | | not applicable; assumed inputs | Authored examples | break_even_p_repro |
| assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $90 | | | not applicable; assumed inputs | Authored examples | cost_misroute |
| assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $40 | | | not applicable; assumed inputs | Authored examples | cost_delay |
| assumed probability | one illustrative bug report (assumed) | assumed inputs, October 2026 | 0.6 | | | not applicable; assumed inputs | Authored examples | p_repro_example |
| (1 - p) times the misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $36 | | | not applicable; assumed inputs | Authored examples | expected_cost_route |
## Findings
A bug report reaches a software team, and someone has to decide whether to route it to an engineer now or ask the reporter for steps that reproduce the problem. Suppose that routing a report nobody can reproduce wastes $90 of an engineer's time before the engineer sends it back for details, and that asking first costs $40 of delay whether or not the report was complete. A model reads the report and returns the probability that it can be reproduced.
Both costs are assumptions written for this example. A team would estimate the first from engineers' time logs on reports that were later closed as not reproducible, and the second from what a day of delay costs the people waiting on a fix.
Above what probability of reproducibility should the team route the report instead of asking first?
Routing costs the misrouting cost only when the report cannot be reproduced, so a report with a probability of 0.6 has an expected cost of $36 if routed, against $40 if the team asks. Routing wins. The two actions cost the same at a probability of 0.556, so that is the cutoff: route above it and ask below it.
(Interactive illustration: a cutoff on the probability p that a report can be reproduced. With an assumed cost of 90 for routing a report nobody can reproduce and 40 for asking first, routing has the expected cost (1 − p) × 90 and costs less than asking above a probability of 0.556.)
The example separates two things that are easy to run together. The probability is a judgment about the report, and it could come from a written rule, a trained classifier, a language model or a decision model. The cutoff is a policy, and it came from the costs. No model supplied it, and a better model would not change it. This booklet is about how judgments become actions, and about when a model's judgment improves the action compared with something simpler.
### A decision system has more parts than its model
A model supplies one part of a decision system. The table follows the bug report through every part and names who owns each one.
| Part | In the bug-report example | Who owns it |
| --- | --- | --- |
| Evidence | The report's text, the component it names, the reporter's earlier reports | The reporter and the issue tracker |
| Judgment | The probability that the report can be reproduced | A rule, a classifier or a model |
| Policy | Route when the probability exceeds the cutoff, otherwise ask | The engineering manager, who states the costs |
| Action | Route the report or send it back with a request | Application code |
| Outcome | Engineer time spent and days of delay | Recorded later in the tracker |
| Feedback | Whether a routed report was reproduced | Recorded only for reports that were routed |
(Interactive illustration: one bug report moves through evidence, judgment, policy, action, outcome and feedback. Only a report that was routed returns a label as feedback.)
The last row recurs in every case in this booklet. A team learns whether a routed report could be reproduced, because an engineer tried. It learns much less about the reports it sent back, because nobody tried them. The policy decides which outcomes the organization gets to observe, so the data available for judging the policy later are shaped by the policy itself.
### Where the judgment can come from
Decision models put a bounded judgment behind an interface with a fixed answer space. The application supplies evidence and typed questions: a yes-or-no question, a choice among named options, or a score on a written rubric. The model returns a probability, a choice with a probability for each option, or a score, and ordinary code decides what to do with it. TypeSafe calls models of this kind "System One" models, borrowing Kahneman's term for fast, intuitive judgment [1]. Its Jev model is a prominent early example, and OpenAI offers a similar interface in its Decisions API [2]. Software libraries such as the AI SDK put several of these engines, and general language models, behind one function [3]. (Note: Interfaces change faster than the ideas in this chapter, so the chapters name engines only where a result depends on one.)
None of this makes the judgment new. Classification and statistical decision theory are much older, and a long line of research has found that simple statistical rules often predict as well as expert judgment does [4][5]. A decision model is one more way to supply the judgment row of the table above. Each way produces an output that establishes something different.
| Output | What it establishes | What still needs checking |
| --- | --- | --- |
| A decision model's answer | A choice, rubric score or probability from a fixed answer space | Whether it is correct and calibrated for this application's records |
| A language model's structured output | Text that fits a schema, within what the interface guarantees | Whether the content is correct, and whether any "confidence" field predicts anything |
| A tool call | A proposed operation and its arguments | Whether the operation is appropriate, permitted and likely to work |
| A rule or an executable test | A reproducible result for the condition it checks | Whether that condition captures the actual requirement |
### Evidence that a policy works comes in kinds
A claim that a model "works" can mean any of several things, and each needs different evidence. The case chapters keep them apart.
| Kind of evidence | The question it answers | What can establish it |
| --- | --- | --- |
| Judgment validity | Does the answer match a defensible reference label? | A labeled benchmark drawn from the records the policy will see |
| Probability validity | Among records given a probability near some value, does the event happen at about that rate? | The same benchmark, read for calibration in the relevant population |
| Policy value | Does acting on the answers lower expected cost under stated costs and constraints? | A comparison of complete policies on the benchmark, with the costs labeled as assumed |
| Observed effect | Did running the policy improve what actually happened? | Executing the policy and measuring outcomes, ideally against a randomized comparison |
A labeled benchmark can establish the first two kinds. A cost comparison on the benchmark shows the third, and only under its assumed costs. Neither shows the fourth. The cases climb this ladder in order. Chapter 1.2, From evidence to typed questions (https://vishalsingh.org/decisionai/from-evidence-to-typed-questions) asks about judgment validity, Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) adds probability validity and policy value, Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) asks about policy value when a person's time is limited, and Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) runs its policies and observes what happens, on tasks written for this booklet.
### Four quantities that are easy to confuse
The probability that a label is true, a vendor's confidence statistic, the probability that an action succeeds, and a rubric score are different quantities. A decision model that says a complaint is about a mortgage with probability near one has said nothing about whether routing it to the mortgage team will resolve it. A confidence field may be computed from the spread of the answer probabilities and measure how peaked they are, which is a different quantity from the chance of being right. A rubric score places a record on a written scale, and no outcome has that score as its probability. TypeSafe's documentation makes a related point about its own outputs: calibration is a property of many predictions taken together, and it does not make any single answer correct [6].
### Costs, constraints and the simplest alternative
Each case chapter reports the outcomes it measures first, such as missed escalations, hours of review or reports finished too early, and then shows how stated costs turn them into one comparison. The costs are assumptions, tagged as such, with a sentence on where a firm would find the real number. Some choices are not a matter of cost. An organization may require that certain complaints always reach a person, or that required tests always run. Those constraints rule out actions before any cost comparison starts, and a low assumed cost does not override them.
Every case also compares the model's policy with the simplest credible alternative: a keyword rule, a trained classifier, a fixed step limit, or sending everything to a person. A chapter can end with the simple policy matching or beating the model. That result is a finding about whether to buy or build the system, and the chapter reports it as one.
### One report through a workflow
The bug report can continue past the first decision. Each step below poses a different kind of question, and each links the chapter that studies a decision of that kind.
| Step | Question at that step | Kind of answer | Studied in |
| --- | --- | --- | --- |
| Triage | Are the reproduction details sufficient? | Yes or no, with a probability | This chapter |
| Routing | Which team owns the report? | A choice among teams | Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) |
| Repair | What change fixes the problem? | Generated code | Outside this booklet's scope |
| Testing | Do the required tests pass? | An executable result | Outside this booklet's scope |
| Review | Does the change need a specialist's review? | Several yes-or-no questions, combined | Chapter 2.2, Allocating review attention to code changes (https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes) |
| Stopping | Is the work finished, or should the agent continue or ask? | A choice among next actions | Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) |
The same pattern appears outside software: a company decides which customer to contact after a bad review (Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions)).
Which decisions in your organization are made today by an informal rule that someone wrote once? Who wrote it, who would notice if it were wrong, and what does a wrong decision cost?
Ask students to derive the break-even probability as one minus the delay cost over the misrouting cost, then to say which direction the cutoff moves when delay becomes more expensive. The point to draw out is that the cutoff changed while the model did not.
### What these data cannot show
This chapter uses no data. Its example shows how a policy follows from a stated probability and stated costs, and says nothing about how often any engine returns a correct or calibrated probability. Whether a decision model's policy beats a rule or a classifier is a question each case chapter answers on its own records, and the answer can differ from case to case.
## Tables
The chapter has no figures.
## All estimates
Every fact in facts.json (5), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| break_even_p_repro | 0.556 | probability that the report can be reproduced | break-even probability, 1 minus delay cost over misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Authored examples | |
| cost_delay | $40 | US dollars per report sent back to the reporter | assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Authored examples | Assumed: one day of delay, charged whether or not the report was reproducible. |
| cost_misroute | $90 | US dollars of engineer time per report routed that cannot be reproduced | assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Authored examples | Assumed: one engineer hour. |
| expected_cost_route | $36 | expected US dollars per report routed now | (1 - p) times the misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Authored examples | |
| p_repro_example | 0.6 | probability that the report can be reproduced | assumed probability | one illustrative bug report (assumed) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Authored examples | |
## Definitions
- Expected loss: The average cost of an action, weighting each possible outcome's cost by its probability.
- Cutoff: The probability above which a policy takes an action.
- Calibration: Whether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive.
## Methods
No engine was called and no data were read for this chapter. The opening example states its costs and its probability as assumptions, and every other number on the page is arithmetic on those assumptions.
### The bug-report example
Routing a report that cannot be reproduced is assumed to waste $90 of engineer time, after which the engineer sends it back for details; that second step is not counted. Asking the reporter first is assumed to cost $40 of delay whether or not the report could have been reproduced.
With p the probability that the report can be reproduced, routing costs (1 − p) times the misrouting cost in expectation and asking costs the delay cost. The two are equal at p = 1 − delay cost / misrouting cost, which is 0.556 at these inputs.
## What these data cannot show
This chapter uses no data. Its example shows how a policy follows from a stated probability and stated costs, and says nothing about how often any engine returns a correct or calibrated probability. Whether a decision model's policy beats a rule or a classifier is a question each case chapter answers on its own records, and the answer can differ from case to case.
## References
1. TypeSafe (2026). System One. TypeSafe documentation, read October 24, 2026. https://docs.typesafe.ai/concepts/system-one
2. OpenAI (2026). Decisions. OpenAI API documentation, read October 24, 2026. https://developers.openai.com/api/docs/guides/decisions
3. Vercel (2026). Decisions. AI SDK Core documentation, read October 24, 2026. https://ai-sdk.dev/docs/ai-sdk-core/decisions
4. Meehl, Paul E. (1954). Clinical versus Statistical Prediction. University of Minnesota Press.
5. Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571-582. https://doi.org/10.1037/0003-066X.34.7.571
6. TypeSafe (2026). Confidence. TypeSafe documentation, read October 24, 2026. https://docs.typesafe.ai/confidence.md
## Provenance
- Build script: decision_ai/chapters/decision-ai-in-applications-and-agents/build.py (sha256 cd480dae075fe8e5385ca75321678ad77a4eb35316571f70d3301ad617c78ce2)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 0aa468f3b6cc5d0c9f5a84a9622ac2d2d31ec50713c5c494e57d3b8091aea0b5 |
| meta.json | b233653547d7465ddda26fd43d738eb84873feb2090388e1764a897798ddf381 |
| refs.json | 508fd36e6b882c96fffea4f376a94a0b9e79ae78349bb37433a4427fac1baf5f |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report
- Report (Markdown): https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/decision-ai-in-applications-and-agents/facts.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# From evidence to typed questions: report
Chapter 1.2 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/from-evidence-to-typed-questions. Table n of this report is figure n of the chapter.
## Identification
- Title: From evidence to typed questions
- Chapter: 1.2 (Part I, From judgment to action; Case studies)
- Question: How does the way a judgment about a review is posed change the answer a decision model returns, and how do engines compare with older classifiers?
- Dek: On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating, and reviews of other coffee products for training the classifier; weights: sample_weight restores the star mix of the coffee reviews
- Canonical URL: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions
- Report: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report
- Markdown: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md
- BibTeX key: singh2026fromevidencetotypedquestions
```bibtex
@misc{singh2026fromevidencetotypedquestions,
author = {Singh, Vishal},
title = {From evidence to typed questions},
year = {2026},
url = {https://vishalsingh.org/decisionai/from-evidence-to-typed-questions},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 5,000 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_reviews |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.979 | 95% CI 0.976 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_luna |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.983 | 95% CI 0.980 to 0.985 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_adapter |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.968 | 95% CI 0.964 to 0.972 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_tfidf |
| distinct products in the sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 2,082 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_products |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.980 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_jev |
| training size in the small-sample refit | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | a thousand | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_n_small |
| mean AUC over refits on 1,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.928 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_auc_1000 |
## Findings
Bean & Basket, a coffee roaster that sells bagged coffee online, wants to know which of its reviewers are unhappy and why. The firm is illustrative; the reviews come from 2,082 coffee products sold on Amazon. This chapter poses the judgment as typed questions to two decision engines, Jev and OpenAI's Luna, and to a general language model behind the same interface, on 5,000 sampled reviews. Every review comes with the reviewer's own star rating, which serves as the check. The chapter takes no business action yet; Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) turns the answers into a contact policy.
On this reading task, Jev ranks unhappy reviews a little better than a word-counting classifier fit to the full labeled sample, 0.980 against 0.968 on a ranking score explained below, and well above the same classifier fit to a thousand labels (0.928). So an endpoint is a strong candidate when a firm has few labels of its own. Star ratings are only a stand-in for dissatisfaction, and the issue answers have no label to check them against.
### The data and what counts as correct
The reviews come from Amazon Reviews 2023, restricted to coffee products in the Grocery and Gourmet Food category. The sample holds the same number of reviews for each star rating, so that rare one- and two-star reviews are well represented, and each review carries a sample weight that restores the star mix of the 1,428,867 coffee reviews it was drawn from. Weighted that way, 14.2% of coffee reviews have one or two stars.
The star rating is a proxy. A reviewer can give three stars and still complain bitterly, and a one-star review can be about a late delivery. The chapter uses the rating because it exists for every review without anyone reading them, and it shows below where the proxy and the engines part ways.
### Questions as asked
Each engine received the review's title and text and three typed questions: a yes-or-no question, a score on a five-level rubric and a choice among named problems.
```ts
const questions = {
dissatisfied: { type: 'boolean',
instructions: 'Is the reviewer dissatisfied with this coffee overall?' },
stars: { type: 'score',
instructions: 'Which rating level best matches this review?',
criteria: ['Lowest level: very negative, would not buy again',
'Mostly negative', 'Mixed or neutral', 'Mostly positive',
'Highest level: very positive, would recommend'] },
issue: { type: 'choice',
instructions: 'What is the main problem the reviewer raises, if any?',
criteria: { taste: 'Flavor, bitterness, freshness',
roast_grind: 'Roast level or grind not as expected',
packaging: 'Damaged, leaking or unsealed package',
shipping: 'Late, lost or wrong item', price: 'Price or value',
caffeine: 'Caffeine content or effect', none: 'No problem raised' } },
};
```
A decision engine answers with more than a label. For the yes-or-no question it returns a probability. For the choice it returns a probability for every option. For the rubric it returns a probability for every level and a score that averages the level positions by those probabilities, so a score can fall between two levels. Jev's star score falls clearly between two whole levels for 24.5% of reviews. The Haiku adapter is a general language model behind the same interface; it returns a single answer and a prompted estimate of the probability, with no distribution behind it.
Find a review whose score falls between two levels. What does a fractional star mean, and when would you rather have the distribution than the single most likely level?
### How the engines compare with a trained classifier
Start with the yes-or-no question. Jev, Luna, the language model and a TF-IDF classifier that counts words and word pairs and was trained on reviews of other coffee products, sort unhappy reviews from happy ones well. The useful lesson is in the small differences between them.
Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews
The first view scores ranking with AUC: the chance that a randomly chosen unhappy review gets a higher probability than a randomly chosen happy one. Jev reaches 0.980 and the classifier 0.968, so the engines rank better with no local fitting on these labels. Switch to the log loss view and the classifier moves ahead of Jev and Luna. Log loss also asks whether the probabilities themselves are right, and the classifier was calibrated on reviews like these. Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) shows what that costs once a probability sets a cutoff.
A keyword rule that flags reviews containing words such as "stale", "disappointed" or "refund" is the cheapest alternative of all. It catches 46.1% of dissatisfied reviews, and 39.4% of the reviews it flags are dissatisfied.
The Haiku adapter, a general language model, ranks reviews slightly better than Jev, by +0.0024 (95% CI 0.0012 to 0.0037). These results show no advantage for a specialized endpoint over a language model on this task.
The classifier needed labeled training reviews and a calibration step, and each prediction then costs close to nothing on a laptop. The engines needed neither, but every call costs money and time.
Table 2. Jev is the cheapest of the three models per review
The rubric question gives a second check. Each engine's star score lands near the reviewer's own rating, off by about 0.4 stars on average for Jev, and the three models are close to one another.
Table 3. The three models' star scores miss the reviewer's rating by similar amounts
### Training a classifier or asking an endpoint
The choice between these two routes runs through the rest of the booklet, and neither wins everywhere. The classifier here means TF-IDF with logistic regression, one simple kind of supervised model, and these cases do not speak for every method.
| | A classifier you train | A decision endpoint you ask |
| --- | --- | --- |
| Defining the task | Pick a target, collect past labels, fit a model | Write the evidence, typed questions, options and rubrics |
| Labels you need | Enough to fit and calibrate | None to get a first answer; some to check and tune how you use it |
| Changing the task | A new target needs new labels and a new fit | Edit the question, then check it again, because it is now a different system |
| Trusting its probabilities | Depends on the labels and the population | Must be checked against your population; a returned distribution is not proof of calibration |
| Ongoing work | Run a pipeline and watch for drift | Maintain questions, provider access and version records, and watch for drift |
| Costs to count | Labeling, fitting, maintenance, serving | Integration, checking, calls, retries, maintenance |
| Shown in these cases | Stronger money-back prediction, product agreement level with Jev on the actual complaint mix | Better review ranking with no labels, higher contact value once recalibrated |
### What labels buy the classifier
The classifier above trained on 20,000 labeled reviews, which many firms do not have. To see what labels buy, the classifier was refit on smaller random subsets of those reviews and each fit was scored as a contact policy, using the costs that Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) derives.
Jev's raw probabilities earn $5,076 per thousand reviews at those costs. What would you expect a classifier trained on a thousand labeled reviews to earn?
Table 4. The review classifier needs thousands of labels to approach what Jev earns with none
Each label count covers everything the classifier used: some of those labels go to calibration and the rest to fitting, and none comes from the reviews it is scored on. Raw Jev is the reference with no local fitting, which does not mean no labels at all, because checking or recalibrating it takes some. These numbers describe this classifier, sampled and fit this way. Another product, target or class mix could move them, so treat them as a guide for a pilot and not as the labels any firm needs.
The classifier earns $3,300 with two hundred labels, $4,500 with a thousand and $4,800 with five thousand. It reaches about what Jev earns only near the full sample, at $5,000. Labels still have a job on the endpoint's side: the next chapter shows that a modest labeled sample is what fixes Jev's probabilities near the cutoff.
### The wording of a question is part of the policy
A question spec is a small piece of policy, and editing it changes answers. A random subset of the sample went to Jev and Luna with one edit at a time, asked as a single question. The same subset was also sent the unedited question on its own in two separate calls, so that each edit can be compared with the change between two identical calls.
Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few
Rewording the question changes 2.5% of Jev's answers and reversing the options changes 1.9%, against 0.2% and 0.4% between identical calls. Removing the "no problem raised" option looks larger, at 67%, but that is almost entirely forced: Jev chose that option for 67% of these reviews, and those answers must change once it is gone. Among reviews where Jev named a problem, only 0.9% of answers change. Jev and Luna choose the same issue for 94.1% of reviews, and for 81.4% of those where at least one of them names a problem.
The three panels below let you inspect the reviews behind each edit.
### Where the engine and the rating disagree
Jev gives a probability of dissatisfaction near one to some four- and five-star reviews, and near zero to some one-star reviews; together these confident disagreements make up 0.2% of reviews. Reading them is the fastest way to learn what the label measures. Some are the engine's mistakes. Others are reviewers who wrote a complaint and gave five stars anyway, or gave one star for a reason the question does not ask about.
The question spec is policy. Who should own its wording, who reviews a change, and how would you version it so that last month's answers can be compared with this month's? When is a word-counting classifier still the right tool?
Ask students to predict, before showing the wording comparison, whether a synonym will change more answers than asking the same question again. Then ask why removing the no-problem option changes most answers and yet says little about how sensitive the engine is to wording.
### What these data cannot show
The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.
## Tables
### Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews
AUC for dissatisfied (one or two stars), log loss and the gain in AUC over the classifier, sampled coffee reviews, weighted
| Engine | Measure | Estimate | Lower | Upper |
| --- | --- | ---: | ---: | ---: |
| Jev 1.14.0 | auc | 0.980 | 0.978 | 0.983 |
| OpenAI Luna | auc | 0.979 | 0.976 | 0.982 |
| Claude Haiku adapter | auc | 0.983 | 0.980 | 0.985 |
| TF-IDF + logistic | auc | 0.968 | 0.964 | 0.972 |
| Jev 1.14.0 | logloss | 0.179 | 0.171 | 0.187 |
| OpenAI Luna | logloss | 0.187 | 0.175 | 0.200 |
| Claude Haiku adapter | logloss | 0.165 | 0.158 | 0.173 |
| TF-IDF + logistic | logloss | 0.157 | 0.148 | 0.167 |
| Jev 1.14.0 | gain | 0.012 | 0.009 | 0.016 |
| OpenAI Luna | gain | 0.011 | 0.007 | 0.015 |
| Claude Haiku adapter | gain | 0.015 | 0.011 | 0.018 |
Views: Ranking (AUC) (metric = auc); Gain over the classifier (metric = gain); Probability fit (log loss) (metric = logloss).
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json
### Table 2. Jev is the cheapest of the three models per review
Mean billed cost per thousand reviews and median latency per call, by engine, October 2026
| engine | measure | n | value |
| --- | --- | ---: | ---: |
| Jev 1.14.0 | cost | 5,000 | 0.023 |
| Jev 1.14.0 | latency | 5,000 | 123 |
| OpenAI Luna | cost | 5,000 | 0.053 |
| OpenAI Luna | latency | 5,000 | 259 |
| Claude Haiku adapter | cost | 5,000 | 0.124 |
| Claude Haiku adapter | latency | 5,000 | 893 |
Views: Cost (measure = cost); Latency (measure = latency).
Note: Cost is the mean billed cost at list price and latency the median of the calls made from one machine; neither carries an interval.
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json
### Table 3. The three models' star scores miss the reviewer's rating by similar amounts
Mean absolute error of the star score and the share of reviews whose most likely level equals the rating, sampled coffee reviews, weighted
| conf_high | conf_low | engine | estimate | metric | n |
| ---: | ---: | --- | ---: | --- | ---: |
| 0.397 | 0.353 | Jev 1.14.0 | 0.374 | mae | 5,000 |
| 0.418 | 0.374 | OpenAI Luna | 0.395 | mae | 4,997 |
| 0.427 | 0.384 | Claude Haiku adapter | 0.405 | mae | 5,000 |
| 73.98 | 70.53 | Jev 1.14.0 | 72.22 | exact | 5,000 |
| 69.78 | 65.83 | OpenAI Luna | 67.76 | exact | 4,997 |
| 74.75 | 71.54 | Claude Haiku adapter | 73.18 | exact | 5,000 |
Views: Average miss (metric = mae); Exact level (metric = exact).
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json
### Table 4. The review classifier needs thousands of labels to approach what Jev earns with none
Net value of the contact policy per thousand reviews against labeled reviews used to train the classifier, weighted
| conf_high | conf_low | labels | net |
| ---: | ---: | --- | ---: |
| 3009 | 0 | 50 labels | 1772 |
| 3204 | 0 | 100 labels | 2224 |
| 3758 | 2882 | 200 labels | 3307 |
| 4191 | 3684 | 500 labels | 4022 |
| 4559 | 4208 | 1,000 labels | 4466 |
| 4673 | 4331 | 2,000 labels | 4551 |
| 4877 | 4731 | 5,000 labels | 4816 |
| 4986 | 4906 | 10,000 labels | 4944 |
| 5058 | 5036 | 20,000 labels | 5047 |
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; ten refits per size, three at the largest; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json
### Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few
Percent of reviews whose answer differs from the original question, after one change and after asking the same question again, Jev and Luna, weighted
| again | again_hi | again_lo | change | changed | changed_hi | changed_lo | engine | n_again | n_changed | size |
| ---: | ---: | ---: | --- | ---: | ---: | ---: | --- | ---: | ---: | --- |
| 0.220 | 0.400 | 0.070 | Reworded question | 2.49 | 3.95 | 1.38 | Jev 1.14.0 | 1,000 | 1,000 | small |
| 0 | n/a | n/a | Reworded question | 1.69 | 2.66 | 1.03 | OpenAI Luna | 999 | 999 | small |
| 0.390 | 0.670 | 0.150 | Issue options reversed | 1.86 | 3.01 | 0.930 | Jev 1.14.0 | 1,000 | 1,000 | small |
| 0 | n/a | n/a | Issue options reversed | 2.98 | 3.69 | 2.29 | OpenAI Luna | 1,000 | 1,000 | small |
| 0.390 | 0.670 | 0.150 | No-problem option removed | 67.36 | 70.59 | 63.89 | Jev 1.14.0 | 1,000 | 1,000 | large |
| 0 | n/a | n/a | No-problem option removed | 69.61 | 72.82 | 66.14 | OpenAI Luna | 1,000 | 988 | large |
Views: All three changes (every row); Small edits only (size = small).
Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json
## All estimates
Every fact in facts.json (37), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| auc_adapter | 0.983 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.980 to 0.985 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_diff_adapter_jev | +0.0024 | difference in AUC | general language model minus Jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.0012 to 0.0037 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_jev | 0.980 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_luna | 0.979 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.976 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_tfidf | 0.968 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.964 to 0.972 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_escape_jev | 67.4% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 63.9% to 70.6% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_escape_named_jev | 0.9% | percent of reviews | answer differs, among reviews where Jev named a problem with the option present | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | | 682 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_order_jev | 1.9% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.9% to 3.0% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_wording_jev | 2.5% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 1.4% to 3.9% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_adapter | 0.0% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_jev | 0.0% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_luna | 0.1% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| issue_agree | 94.1% | share of reviews | Jev and Luna choose the same issue | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| issue_agree_named | 81.4% | share of reviews | Jev and Luna choose the same issue, among reviews where at least one names a problem | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 3,412 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_1000 | 0.928 | area under the ROC curve | mean AUC over refits on 1,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_200 | 0.821 | area under the ROC curve | mean AUC over refits on 200 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_20000 | 0.968 | area under the ROC curve | mean AUC over refits on 20,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_5000 | 0.957 | area under the ROC curve | mean AUC over refits on 5,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_n_small | a thousand | labeled reviews | training size in the small-sample refit | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_1000 | $4,466 | US dollars per 1,000 reviews | mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_200 | $3,307 | US dollars per 1,000 reviews | mean net value over refits on 200 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_20000 | $5,047 | US dollars per 1,000 reviews | mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_5000 | $4,816 | US dollars per 1,000 reviews | mean net value over refits on 5,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_jev | $5,076 | US dollars per 1,000 reviews | Jev's raw probabilities at the same contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| mae_jev | 0.37 | stars | weighted mean absolute error of the score against the rating | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.35 to 0.40 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_confident_disagree | 0.2% | share of reviews | Jev P(dissatisfied) at least 0.9 on four or five stars, or at most 0.05 on one star | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_population | 1,428,867 | reviews | coffee reviews the sample stands for | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_products | 2,082 | products | distinct products in the sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_reviews | 5,000 | reviews | sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_train | 20,000 | labeled reviews | classifier training sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| noise_order_jev | 0.4% | percent of reviews | answer differs between two identical calls | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.1% to 0.7% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| noise_wording_jev | 0.2% | percent of reviews | answer differs between two identical calls | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.1% to 0.4% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| rule_precision | 39.4% | share of flagged reviews that are dissatisfied | weighted precision | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| rule_recall | 46.1% | share of dissatisfied reviews the rule flags | weighted recall | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_dissatisfied | 14.2% | share of reviews with one or two stars | weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_fractional_jev | 24.5% | share of reviews | Jev score at least a quarter star from a whole number | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_none_jev | 67.1% | percent of reviews | Jev answers no problem raised | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
## Definitions
- Sample weight: A number attached to each sampled item so that weighted results describe the population the sample was drawn from rather than the sample's own mix.
- AUC: Area under the ROC curve: the chance that a randomly chosen positive record gets a higher score than a randomly chosen negative one. It measures ranking, not calibration.
## Methods
Each sampled review went to two decision engines and a general language model with the same three typed questions, and to a classifier trained on reviews of other coffee products. Answers are compared with the reviewer's own star rating.
### The sample
The eval sample holds the same number of reviews for each star rating, drawn with a fixed seed from the coffee reviews. Weights restore the star mix of all coffee reviews, so a share in the text describes that population rather than the balanced sample.
### Variants and repeats
A random subset of the sample was sent again with one change to the question: a reworded dissatisfied question, the issue options in reverse order, and the issue question without its no-problem option. The same subset was also sent a second time with the original question, so a change from rewording can be compared with the change from simply asking again.
### Intervals
Intervals are 95% bootstrap intervals that resample products within star strata, with the same resamples for every engine, so the interval on a difference reflects the pairing.
## What these data cannot show
The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/from-evidence-to-typed-questions/build.py (sha256 614bfa013c23aaad7fb46c684c28d74a47931f487d9924d97ba5996ce54c6dd0)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | f2bea860b34a80b68cba30c4778a50fb56cb894d00dbe90d0835b4239fc38f7f |
| figures/engine-cost-speed.json | f8da6d8e7a6ce6135edc3aa5e3f05cbcd27e0b423bd8aea06b90bd0395b91218 |
| figures/engine-scores.json | aa494371dfbbd60a3c5eda50ee6440fbf0b015811042f4031a8b815a2bf0d80f |
| figures/labels-to-catch-up.json | 78a2da5df8b850d5a954f667a6c17df5cff672112bec4baceda69a9ed46f8e68 |
| figures/question-changes.json | c1ce35a86b5b074236a5853a52829be7f7dafca85ef0b4fe6dfb7719ff9c7f29 |
| figures/star-score-error.json | deb07210f374f86111280492372ed21de8309d4d5f327f4a241f27afead1eecd |
| meta.json | e5508615adc538ad383df2ea75ac289022ee9b420f73a543c4e3621ae6e71773 |
| public/data/from-evidence-to-typed-questions/answer.json | 18a7a97e8c330fa254f4c34c2f6e9f0257c148dbb0f5912ef6dfd23073dfd1e7 |
| public/data/from-evidence-to-typed-questions/excerpts.json | db37452cad35c16930ad5bb119733b88e8479163f660dd69ce4f7359865cbd92 |
| public/data/from-evidence-to-typed-questions/frontier.json | 0c8b1e929247ec1535942e6a642fbdd77d65609b55fe7d06e9918568ac5f6d79 |
| public/data/from-evidence-to-typed-questions/reveal.json | 5d4a01c21a94e63e4b03b7e6e809425eec29a3733069229a090ef6f98a7a928f |
| public/data/from-evidence-to-typed-questions/variant-escape.json | d29bb4e5d3151f2ba81fc61ad61d35e9ec5d02b9b60287ee4cdadbd978452c9f |
| public/data/from-evidence-to-typed-questions/variant-order.json | 69f4f4b073d906307286fd248eb61f3ebc988ff84f7e8ceda8aa51715e43bf7e |
| public/data/from-evidence-to-typed-questions/variant-wording.json | fc107326e237d2ad30a920b80b4b19f42ce0751de5645c0ce8ff06040c283d0a |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report
- Report (Markdown): https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/facts.json
- Table 1, figure "engine-scores" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json
- Table 2, figure "engine-cost-speed" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json
- Table 3, figure "star-score-error" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json
- Table 4, figure "labels-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json
- Table 5, figure "question-changes" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# From probabilities to actions: report
Chapter 1.3 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/from-probabilities-to-actions. Table n of this report is figure n of the chapter.
## Identification
- Title: From probabilities to actions
- Chapter: 1.3 (Part I, From judgment to action; Case studies)
- Question: Given the cost of a contact and the value of a recovered customer, which dissatisfied reviewers should a coffee roaster contact, and what does that rule assume?
- Dek: At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating; weights: sample_weight restores the star mix of the coffee reviews
- Canonical URL: https://vishalsingh.org/decisionai/from-probabilities-to-actions
- Report: https://vishalsingh.org/decisionai/from-probabilities-to-actions/report
- Markdown: https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md
- BibTeX key: singh2026fromprobabilitiestoactions
```bibtex
@misc{singh2026fromprobabilitiestoactions,
author = {Singh, Vishal},
title = {From probabilities to actions},
year = {2026},
url = {https://vishalsingh.org/decisionai/from-probabilities-to-actions},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $6 | | | not applicable; assumed inputs | Amazon Reviews 2023 | contact_cost |
| r times V | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $48 | | | not applicable; assumed inputs | Amazon Reviews 2023 | expected_recovery |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,076 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,015 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_tfidf |
| net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,335 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev_recal |
| assumed value | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $120 | | | not applicable; assumed inputs | Amazon Reviews 2023 | customer_value |
| assumed rate | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | 0.4 | | | not applicable; assumed inputs | Amazon Reviews 2023 | recovery_rate |
| sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 5,000 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_reviews |
| weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 14.2% | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | share_dissatisfied |
| difference in net value, jev_recal minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | +$320 | 95% CI $228 to $411 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_diff_jev_recal_tfidf |
## Findings
Bean & Basket, a coffee roaster that sells bagged coffee online, wants its care team to contact unhappy reviewers with a replacement bag or a voucher. The firm is illustrative, and the reviews come from many coffee products sold on Amazon. Suppose a contact costs $6 in staff time and voucher value, a retained customer is worth $120, and a contact wins back a dissatisfied customer with probability 0.4. A contact with a dissatisfied reviewer is then worth $48 on average, and a contact with a satisfied reviewer is wasted. Each of 5,000 sampled coffee reviews has a model's probability that the reviewer is dissatisfied, and weighted to all coffee reviews, 14.2% of reviewers gave one or two stars. Which reviewers should the team contact?
At these assumed costs, Jev's raw probabilities and a trained classifier earn about the same, $5,100 and $5,000 per thousand reviews, because Jev's probabilities run too high near the cutoff. Recalibrated on labeled reviews, Jev earns about $5,300, ahead of the classifier by +$320 (95% CI $228 to $411). So for this policy and these costs, the ranking edge paid off once labels went into the probabilities. Stars stand in for dissatisfaction, the costs are assumptions, and dissatisfaction alone does not tell you who will respond to a voucher.
The contact cost, the customer value and the recovery rate are assumptions written for this chapter. Finance would estimate the first two from voucher costs and repeat-purchase margins. The recovery rate needs a randomized contact experiment, which the last section of this chapter simulates.
### The cutoff comes from the costs
Contacting a reviewer pays when the expected recovery exceeds the cost: when the probability of dissatisfaction p satisfies p times rV greater than c. The cutoff is
$$
p^{*} = \frac{c}{rV}
$$
which is 0.125 at these costs. A cutoff this low means the team should contact many reviewers who are probably satisfied, because a contact is cheap relative to what a recovered customer is worth.
Many textbooks give the cutoff as the cost of a false alarm divided by the sum of that cost and the cost of a miss. With a false alarm costing c and a miss costing rV, what does that formula give here, and is it the same as p*?
The shortcut gives 0.111, a little lower than 0.125. It assumes that a correct decision costs nothing, and here a correct contact still costs c. The difference is small at these costs and grows as c approaches rV. When c exceeds rV, p* is above one and the right policy is to contact nobody, whatever the model says.
Now score each policy. The chart below sets every way of choosing whom to contact against the simplest one, contacting nobody, at the default costs and the cutoff 0.125.
Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier
Jev's probabilities earn about $5,100 per thousand reviews and the classifier's about $5,000, both well above the keyword rule and contacting everyone. Perfect foresight, a policy that knew every rating in advance, sets the ceiling at about $6,000.
The difference between Jev's raw probabilities and the classifier is +$61 (95% CI −$47 to $159), an interval that includes zero, so this comparison shows no clear advantage for either. Luna and the Haiku adapter do better against the classifier, by +$180 (95% CI $87 to $277) and +$128 (95% CI $31 to $226). The classifier was trained on reviews of other coffee products; a firm would train its own on its own rated reviews, and collecting those labels and maintaining the model are costs these figures leave out.
Raise the cost per contact to $25. The cutoff moves to 0.521. Which reviews drop off the contact list first, and does the ranking of engines change?
Set the contact cost to $60, more than the $48 a contact is expected to recover. No cutoff below one makes contacting pay, so the best policy contacts nobody.
### The rule assumes calibrated probabilities
The formula for p* takes a probability at face value: a review given 0.125 should come from a group in which about that share of reviewers are unhappy. The engines' probabilities do not mean that here. Among reviews to which Jev gives a probability between 0.125 and 0.3, just above the cutoff, only 4.2% have one or two stars, and 85% have four or five. Jev reads doubt into reviews whose writers still rated the coffee well.
Calibration measures this gap. The engines start well behind the classifier, and recalibrating them on part of the sample, then scoring on the rest, closes the gap. Here the ranking edge turns into money: at the same cutoff, recalibrated Jev earns +$259 (95% CI $189 to $333) more than raw Jev.
Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated
In practice the gap means extra contacts. At p*, Jev contacts 287 reviewers per thousand reviews and the classifier 222, and most of the difference is satisfied reviewers.
Switch on recalibration. The points move toward the diagonal, and the probabilities near p* fall.
A second fix leaves the probabilities alone and moves the cutoff instead. Choose the cutoff that earns the most on four folds of the sample, apply it to the fifth, and repeat. For Jev the data push the cutoff from 0.125 up to 0.38.
Table 3. Letting the data choose the cutoff moves it above p* for the engines
Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier
How many labels does that fix take, compared with training a classifier on the same labels? Each draw below gives both arms identical labeled reviews.
Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried
With two hundred labels, recalibrated Jev earns $5,300 and the classifier $3,300. With a thousand, $5,300 and $4,500. At the full sample the gap is still $5,300 against $5,000. On this reading task, a few hundred labels spent checking the endpoint went further than the same labels spent fitting this classifier. These figures leave out what each call and each model costs to run, and they describe one simple classifier and one task. The money-back question in a later chapter goes the other way.
The lesson for a manager: a model that ranks reviews well can still be a poor input to a cost formula. Check its probabilities against the population it will be used on, or let the data set the cutoff. Either fix needs labeled reviews from that population, and these figures do not charge for them.
The core lesson ends here. The next two sections are optional extensions: adding a person as a third action, and asking who actually responds to a contact.
Run the cheap-voucher preset, a contact cost of $3 with a cutoff of 0.062, and the expensive one, and ask why the same model produces different contact lists. Then ask students to derive the gap between the shortcut and p* themselves, and to say what happens to p* as the contact cost approaches the expected recovery.
### Going further: send the review to a person
Between contacting and ignoring there is a third option. A person on the care team can read the review for $1.50 and decide, and the panel assumes that person gets it right with probability 0.9, a number no one measured here. With two cutoffs, the policy contacts reviewers above the upper one, ignores reviewers below the lower one and sends the rest to a person. Reviewer capacity limits how many reviews can go to people, and once it binds, the reviews worth reading are the ones whose expected loss falls most from a person's judgment.
For one review at a time, the comparison is a table of expected losses, one row per action. The action with the lowest expected loss is the one to take.
### Going further: dissatisfied is not the same as responsive
The recovery rate r describes how much a contact changes a customer's chance of staying, and star ratings cannot measure it. An unhappy customer may leave whatever the firm does, and another may return without any voucher. Contacting the most dissatisfied reviewers is the right ranking when a contact helps them at least as much as it helps anyone else at the same cost, an assumption a review label cannot check. A randomized contact experiment would provide a direct way to estimate that incremental effect.
The simulator below is a seeded simulation, not data from any firm. In it, a contact changes retention most for moderately dissatisfied customers who buy often and close to zero for the most dissatisfied. At the same contact budget, it compares targeting by the probability of dissatisfaction with targeting by the response estimated from a simulated experiment.
Cut the contact budget. Which customers does each targeting rule keep, and which one loses more of its retained customers?
A cutoff applied once to each review also ignores time. A contact changes what the firm learns next, because a reply or a repeat purchase is new evidence about that customer. Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) returns to decisions where each action changes the evidence for the next one.
Finance owns the contact cost, the customer value and the recovery rate. What would make you recheck the cutoff: a change in voucher cost, a new month of reviews whose calibration has drifted, or an experiment showing that the most dissatisfied customers do not respond?
### What these data cannot show
The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.
## Tables
### Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier
Net value of contacts per thousand reviews against contacting nobody, at the cost cutoff p*, by policy, weighted
| conf_high | conf_low | estimate | group | n | policy |
| ---: | ---: | ---: | --- | ---: | --- |
| n/a | n/a | 5978 | Ceiling | 5,000 | Perfect foresight |
| 5253 | 5135 | 5195 | Decision engine | 5,000 | OpenAI Luna |
| 5202 | 5083 | 5144 | General language model | 5,000 | Claude Haiku adapter |
| 5144 | 5004 | 5076 | Decision engine | 5,000 | Jev 1.14.0 |
| 5104 | 4928 | 5015 | Trained classifier | 5,000 | TF-IDF + logistic |
| 2309 | 1978 | 2148 | Simple rule | 5,000 | Keyword rule |
| n/a | n/a | 832 | Simple rule | 5,000 | Contact every reviewer |
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json
### Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated
Expected calibration error of the probability of dissatisfaction, as returned and after recalibration, by engine, weighted
| conf_high | conf_low | engine | estimate | n | version |
| ---: | ---: | --- | ---: | ---: | --- |
| 0.079 | 0.071 | Jev 1.14.0 | 0.075 | 5,000 | As returned |
| 0.006 | 0.001 | Jev 1.14.0 | 0.002 | 5,000 | Recalibrated |
| 0.064 | 0.056 | OpenAI Luna | 0.060 | 5,000 | As returned |
| 0.006 | 0.001 | OpenAI Luna | 0.002 | 5,000 | Recalibrated |
| 0.073 | 0.066 | Claude Haiku adapter | 0.069 | 5,000 | As returned |
| 0.016 | 0.007 | TF-IDF + logistic | 0.011 | 5,000 | As returned |
Note: The figure recalibrates only the two engines that return a probability distribution.
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json
### Table 3. Letting the data choose the cutoff moves it above p* for the engines
Cutoff that maximizes net value, chosen on four folds of the sample and applied to the fifth, median of five folds, by model
| Model | Median cutoff | Lowest fold | Highest fold |
| --- | ---: | ---: | ---: |
| Jev 1.14.0 | 0.38 | 0.35 | 0.38 |
| OpenAI Luna | 0.22 | 0.20 | 0.29 |
| TF-IDF + logistic | 0.11 | 0.09 | 0.16 |
Note: Each dot is the median of the five fold cutoffs; the table gives the lowest and highest. No bootstrap interval was computed.
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json
### Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier
Net value per thousand reviews at the cost cutoff p* and at the data-chosen cutoff, by model, weighted
| at_data | at_formula | engine | formula_hi | formula_lo | n |
| ---: | ---: | --- | ---: | ---: | ---: |
| 5335 | 5076 | Jev 1.14.0 | 5144 | 5004 | 5,000 |
| 5273 | 5195 | OpenAI Luna | 5253 | 5135 | 5,000 |
| 4985 | 5015 | TF-IDF + logistic | 5104 | 4928 | 5,000 |
Note: The data-chosen value is scored out of fold and carries no bootstrap interval.
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json
### Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried
Net value of the contact policy per thousand reviews against local labels used, weighted; whiskers run from the lowest to the highest draw
| conf_high | conf_low | labels | model | n_labels | net |
| ---: | ---: | --- | --- | ---: | ---: |
| 2475 | 0 | 50 labels | Classifier, fit and calibrated on these labels | 50 | 1502 |
| 5337 | 3387 | 50 labels | Jev, recalibrated on these labels | 50 | 5058 |
| 3709 | 1570 | 100 labels | Classifier, fit and calibrated on these labels | 100 | 2981 |
| 5329 | 5029 | 100 labels | Jev, recalibrated on these labels | 100 | 5188 |
| 3885 | 1858 | 200 labels | Classifier, fit and calibrated on these labels | 200 | 3257 |
| 5341 | 5072 | 200 labels | Jev, recalibrated on these labels | 200 | 5257 |
| 4387 | 3338 | 500 labels | Classifier, fit and calibrated on these labels | 500 | 4073 |
| 5336 | 5185 | 500 labels | Jev, recalibrated on these labels | 500 | 5285 |
| 4628 | 4386 | 1,000 labels | Classifier, fit and calibrated on these labels | 1,000 | 4503 |
| 5335 | 5248 | 1,000 labels | Jev, recalibrated on these labels | 1,000 | 5300 |
| 4683 | 4490 | 2,000 labels | Classifier, fit and calibrated on these labels | 2,000 | 4581 |
| 5337 | 5225 | 2,000 labels | Jev, recalibrated on these labels | 2,000 | 5307 |
| 4872 | 4749 | 5,000 labels | Classifier, fit and calibrated on these labels | 5,000 | 4816 |
| 5344 | 5270 | 5,000 labels | Jev, recalibrated on these labels | 5,000 | 5320 |
| 4974 | 4897 | 10,000 labels | Classifier, fit and calibrated on these labels | 10,000 | 4939 |
| 5342 | 5304 | 10,000 labels | Jev, recalibrated on these labels | 10,000 | 5333 |
| 5045 | 5008 | 20,000 labels | Classifier, fit and calibrated on these labels | 20,000 | 5032 |
| n/a | n/a | 20,000 labels | Jev, recalibrated on these labels | 20,000 | 5342 |
Note: Platt scaling in place of isotonic calibration gives the same reading.
Source: Amazon Reviews 2023, coffee products; labels from reviews of other products; policy scored on the sampled evaluation reviews, weighted; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json
## All estimates
Every fact in facts.json (36), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| band_hi | 0.3 | probability of dissatisfaction | upper edge of the band just above the cutoff | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| band_rate_jev | 4.2% | share of reviews with one or two stars | among reviews given P from p* up to 0.3 | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 427 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| band_share_high_stars | 84.8% | percent of reviews in the band | four- and five-star reviews among those given P from p* up to 0.3 by Jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 427 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| contact_cost | $6 | US dollars per contact | assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| contact_cost_cheap | $3 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| contact_cost_expensive | $25 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| contact_cost_prohibitive | $60 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| contacts_jev | 287 | contacts per 1,000 reviews | weighted contacts at p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| contacts_tfidf | 222 | contacts per 1,000 reviews | weighted contacts at p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| customer_value | $120 | US dollars per retained customer | assumed value | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| cut_emp_jev | 0.38 | probability of dissatisfaction | median of five cross-fit cost-minimizing cutoffs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Range across folds 0.35 to 0.38. |
| expected_recovery | $48 | US dollars per contact with a dissatisfied customer | r times V | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| lv_clf_1000 | $4,503 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_200 | $3,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_full | $5,032 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_1000 | $5,300 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_200 | $5,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_full | $5,342 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| n_reviews | 5,000 | reviews | sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_diff_adapter_tfidf | +$128 | US dollars per 1,000 reviews | difference in net value, adapter minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $31 to $226 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_diff_jev_recal_jev | +$259 | US dollars per 1,000 reviews | difference in net value, jev_recal minus jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $189 to $333 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_diff_jev_recal_tfidf | +$320 | US dollars per 1,000 reviews | difference in net value, jev_recal minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $228 to $411 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_diff_jev_tfidf | +$61 | US dollars per 1,000 reviews | difference in net value, jev minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI −$47 to $159 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_diff_luna_tfidf | +$180 | US dollars per 1,000 reviews | difference in net value, luna minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $87 to $277 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| net_jev | $5,076 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_jev_recal | $5,335 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_oracle | $5,978 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_tfidf | $5,015 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| p_shortcut | 0.111 | probability of dissatisfaction | shortcut cutoff C_FP / (C_FP + C_FN) with C_FN = rV | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| p_star | 0.125 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| p_star_cheap | 0.062 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| p_star_expensive | 0.521 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| recovery_rate | 0.4 | share of dissatisfied customers a contact retains | assumed rate | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| review_accuracy | 0.9 | share of reviews a person judges correctly | assumed reviewer accuracy | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| review_cost | $1.50 | US dollars per human review | assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | |
| share_dissatisfied | 14.2% | share of reviews with one or two stars | weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
## Definitions
- Calibration: Whether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive.
## Methods
Two decision engines, a general language model and a trained classifier gave each sampled review a probability of dissatisfaction. Policies contact a reviewer when that probability reaches a cutoff, and each policy is scored on the star labels with assumed costs.
### What counts as dissatisfied
A review counts as dissatisfied when its star rating is one or two. The rating is the reviewer's own, so it is a proxy that exists without any rater. A few low ratings complain about delivery rather than the coffee, and a few high ratings contain complaints.
### Engines and the classifier
Jev 1.14.0 and OpenAI Luna return a probability for a yes-or-no question. The Claude Haiku adapter returns a prompted estimate of that probability. The TF-IDF classifier was trained on reviews of other coffee products and calibrated on a held-out part of them.
Each engine was called once per review in October 2026 and the answers were stored.
### Weights, folds and intervals
The sample holds the same number of reviews for each star rating, so every share, cost and calibration curve is weighted back to the star mix of the coffee reviews. Recalibrated probabilities and data-chosen cutoffs are cross-fit over five folds that keep each product's reviews together. Intervals come from a bootstrap that resamples products within star strata, with the same resamples for every policy.
## What these data cannot show
The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/from-probabilities-to-actions/build.py (sha256 58dca1335409b82c81c737aef7f9af14d02d25c1e5e6b9be77888ea624e2ecb3)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 069f32998b43fab552b79437f4492e543998297912ab27ae03e3fa276ff03457 |
| figures/calibration-error.json | 8006eba4847d93ba8a4f148eda7dae82b7b4a3a61016a0fe2a07d35796655bbc |
| figures/data-chosen-cutoffs.json | a6cb47cab02f1bf4081668fe1e971c9faa0b6e4f01c3da0d37e0ee0b36285e6a |
| figures/net-at-data-cutoffs.json | a9a0d2a997bf6d8a12836ab74821ebf5b9e7b0cf2dc70504d2cfa0c044e68f6f |
| figures/net-value-by-policy.json | 4f68a040c412cecaa041de32fd28cf85298f3413336cf4f64a6d8eb9acd6cc33 |
| figures/value-by-labels.json | 072e9978e513582a94013393a3d5a7717008bdd756bff21168e87e86c23f2e59 |
| meta.json | 08b7e478643cd97028ee57ddf47f2196dfb9ee8af303f5c8d85caca83910ad8b |
| public/data/from-probabilities-to-actions/items.json | 508445f13140a53a1db0af4f7562cf285dcda90e0a2f0ec9c5a38fef7edebbf9 |
| public/data/from-probabilities-to-actions/uplift.json | bd0966a9319584eb784730b70bc7056f7d91e54b3cc0cf4f18c0da4077d29454 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/from-probabilities-to-actions/report
- Report (Markdown): https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/facts.json
- Table 1, figure "net-value-by-policy" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json
- Table 2, figure "calibration-error" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json
- Table 3, figure "data-chosen-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json
- Table 4, figure "net-at-data-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json
- Table 5, figure "value-by-labels" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# Resolving and routing customer complaints: report
Chapter 2.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints. Table n of this report is figure n of the chapter.
## Identification
- Title: Resolving and routing customer complaints
- Chapter: 2.1 (Part II, Scarce attention; Case studies)
- Question: Which product team should receive each complaint, when should a person check it first, and can a model tell which complaints will end in money back?
- Dek: A classifier trained on earlier complaints matches consumers' product choices about as often as Jev once it is fit to the actual mix, and it finds more of the complaints that end in money back, where it passes Jev with far fewer labels than the review task needs.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- CFPB Consumer Complaint Database. 10,000 complaints received in 2025 with a narrative of at most 3,000 characters, stratified by product; complaints from August 2023 to 2024 for the classifier; weights: sample_weight restores the product mix of the 2025 complaints within the length limit
- Canonical URL: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints
- Report: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report
- Markdown: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md
- BibTeX key: singh2026resolvingandroutingcustomercomplaints
```bibtex
@misc{singh2026resolvingandroutingcustomercomplaints,
author = {Singh, Vishal},
title = {Resolving and routing customer complaints},
year = {2026},
url = {https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| sample size | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 10,000 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | n_complaints |
| 2025 complaints within the length limit that the sample stands for | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 519,841 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | n_population |
| top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 81.9% | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | agree_jev |
| top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 82.4% | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | agree_tfidf |
## Findings
A bank's complaint desk gets a steady stream of written complaints, and each one needs to reach the right team. Send a mortgage complaint to the credit card team and it sits for days before someone sends it back. This chapter asks a decision engine to read each complaint and pick the product it is about, then asks the question every desk manager asks next: when should a person check the engine's pick before it goes out?
The complaints are 10,000 sampled from distinct narratives filed with the Consumer Financial Protection Bureau in 2025, standing for 519,841 narratives; repeated narratives were removed, which matters because many credit-reporting complaints use the same text. No one rated them by hand. Instead, each complaint carries two answers that already exist: the product the consumer selected when filing, and what the company eventually did about it.
Jev and a classifier fit to the actual complaint mix match the consumer's product choice about equally often, 82% and 82%, and once both are recalibrated the cost comparison shows no clear winner at the assumed costs. So the choice between them rests on what a desk can run and check, and either one still needs people for uncertain complaints. Consumer choices are sometimes wrong, and the costs and review times are assumptions. The chapter follows the desk from automatic routing to human review, a limit on review time and how to spend it, then takes up one companion question: which complaints end in money back.
### Picking the product
Each engine saw the narrative and a list of eleven products with a short definition of each, and picked one. Jev and Luna also returned a probability for every product; the Haiku adapter returned only its pick. The chart compares how often the top pick matches what the consumer chose.
Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev
Jev agrees on 82% of complaints and the classifier on 82%, a gap of −0.6 (95% CI −1.9 to 0.7) points. How the classifier is trained decides this comparison. The training sample holds equal numbers of each product, and fitting it without weights drops the classifier to 78%, while the weighted fit does poorly on rare products: counted equally across products, Jev scores 75% and the classifier 64%. Jev costs about $0.05 per thousand complaints.
Agreement is a soft score here. Consumers choose their own product from a form, and some choose badly: a complaint about a debt collector chasing a credit card balance can honestly belong to either team. The flows below show where Jev and the consumers part ways. The largest disagreements run between credit reporting and debt collection, two products that often appear in the same complaint.
### When to ask a person first
Suppose a misrouted complaint costs $30 in delay and sending a complaint to a person for triage costs $6 of staff time. Routing pays when the chance the pick is right is high enough that the expected misroute cost falls below the triage cost. At these costs the cutoff is 0.8: route when the model is more sure than that, and send everything else to a person.
Both costs are assumptions for this chapter. A desk would measure the first from how long misrouted complaints wait and the second from triage handling time. The calculation also assumes that a person always finds the right team, at a fixed cost, with no limit on how many complaints a person can read. It leaves out engine charges and the cost of building the classifier.
Switch to the classifier. Its stated probabilities are lower, so far more of its complaints fall below the cutoff and go to a person, more than this panel's triage capacity can take. A desk then has three moves: add reviewers, let a queue build, or lower the confidence it requires for automatic routing, which cuts the workload and accepts more routing risk. Raising that bar would send even more complaints to people.
The next chart shows what each policy does with the whole stream of complaints.
Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high
With the cutoff taken at face value, Jev sends 16% of complaints to a person and misroutes 10%, and the classifier sends 43% and misroutes 4%. Priced at the assumed costs, that is $4,054 per thousand complaints for Jev and $3,685 for the classifier. Routing everything by Jev with no check costs $5,442, and sending everything to a person costs $6,000.
The gap of +$368 (95% CI $97 to $655) is about calibration more than about reading. Jev's stated probabilities run high, so it routes complaints it should send to a person. Recalibrate both models on other folds of the sample and Jev costs $3,471, sending 35% to a person, against $3,666 for the classifier.
With both recalibrated, the gap between Jev and the classifier is −$195 (95% CI −$419 to $27) per thousand complaints, and that range includes zero, so at these costs the data show no clear advantage for either.
That comparison gives both models plenty of labels. A desk starting out would have few, so the next chart gives both the same labeled complaints from earlier years.
Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried
With five hundred labels, recalibrated Jev costs $3,600 per thousand complaints, against $4,100 for raw Jev and $4,800 for a classifier fit on those labels. With the full sample the classifier comes closer, at $3,700 against $3,500. Labels from earlier years are what a desk would really have, and these figures leave out what each model costs to run.
### When reviewer time is fixed
So far a person has been free to read as many complaints as the cutoff sends. Suppose the desk instead has 400 reviewer minutes for every thousand complaints, and everything outside the queue goes where Jev's pick sends it. Which complaints should a person read? The complaints and the costs stay as they were; what changes is that the desk can no longer send a person everything below the cutoff.
Uncertainty is one reason to look at a complaint. Consequence is another, so the queue below also uses a second model. Its estimate of money back, from the companion section that follows, stands in for a misroute that matters more, under an assumed extra cost. It is a proxy and not a measured loss from a misroute. Read the columns separately, because review workload, misroutes, misrouted money-back complaints and assumed dollars can move in different directions.
The review budget, the review time, which grows with the length of the complaint, the extra $120 for misrouting a complaint that ends in money back, and a reviewer who always finds the right team are assumptions. Misroutes are counted against the consumer's product choice. The ranking uses recalibrated Jev for the risk of a misroute and the classifier's probability of money back for the consequence.
Reviewing the least certain picks is the natural rule, and it leaves 110.4 misroutes per thousand complaints, down from 181.5 with no review. Adding the consequence moves attention toward complaints that may involve money, leaving 1.14 misrouted money-back complaints against 1.57, a difference of −0.43 (95% CI −0.91 to 0.09), whose range includes zero. Adding review time changes more: because short complaints take less time, the same minutes cover 20% of complaints instead of 14%, and the expected cost changes by −425 (95% CI −639 to −198) dollars per thousand. That gain rests on the assumed link between length and review time, and on a short complaint being reviewed as accurately as a long one. A desk would need to measure both before running this queue.
### A companion question: which complaints end in money back
This is a different job from routing: it predicts what a company will do later, not what a complaint is about. A desk might fast-track complaints that are likely to end with the company paying the consumer back. Only 3% of these complaints closed with monetary relief, so finding them early matters. Each engine was asked whether the consumer describes money the company could return, and the classifier was trained directly on which earlier complaints ended in relief.
Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint
Fast-tracking the tenth of complaints the classifier ranks highest would catch 73% of the money-back cases, against 58% for Jev, a gap of +14.9 (95% CI 8.9 to 20.4) points. Much of any model's score here comes from knowing the product, because money back is far more common for some products than others: ranking complaints by their product's past rate alone gives an AUC of 0.906, above Jev's 0.895. Within a product, the classifier's lead narrows to 0.769 against 0.742, a difference of +0.027 (95% CI −0.009 to 0.065). The engine answers the question it was asked, whether the consumer describes a refundable loss. The classifier learned from history which complaints companies actually paid. One reading is that history carries information a single complaint does not, though this comparison cannot separate that from the difference in the question asked, the number of labels or calibration.
Could a better prompt close the gap? On a random subset of complaints, the engines were asked a sharper question, whether the company will likely give money back, in three ways: on its own, with ten earlier complaints labeled paid or not paid, and with a short description of what tended to get paid in earlier years.
Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier
The sharper question and the description together help a little, and rewording alone accounts for much of that. The labeled examples add little. None of the versions brings the engines level with the classifier. How much history does the classifier need? Refit on smaller samples of past complaints, it matches Jev at about a thousand.
Table 6. For money back, the classifier matches Jev with about a thousand past complaints
The classifier catches 49% with three hundred past complaints, 59% with a thousand and 66% with three thousand. That is far fewer labels than the review task needed, and a desk already has these records, because its own case system stores how each complaint closed.
These counts include calibration labels held out from each sample, and the complaints being scored are not used. They describe this classifier on these records, so treat them as a guide for a pilot. Whether a desk should build this model also depends on whether outcomes arrive in time, whether old records resemble today's complaints, and whether money back is what the desk wants to prioritize.
On reading what a complaint is about, the classifier needs a large labeled sample to match the engines, and on predicting what a company will do it passes them with a modest one. Which of those two jobs should a desk buy a model for, and which should it learn from its own records?
Ask the class to set the misroute cost from a regulator's point of view and then from a call-center manager's, and to watch how the triage share changes in the triage panel. Then ask why a classifier trained on past outcomes can beat a model that reads each complaint more closely on the money-back question.
### What these data cannot show
The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.
## Tables
### Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev
Complaints whose predicted product matches the one the consumer selected, percent, by model, with 95% intervals
| agree | conf_high | conf_low | kind | model | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 81.90 | 83.20 | 80.50 | Decision engine | Jev 1.14.0 | 10,000 |
| 84.20 | 85.30 | 82.90 | Decision engine | OpenAI Luna | 10,000 |
| 84.40 | 85.50 | 83.30 | General language model | Claude Haiku adapter | 10,000 |
| 82.40 | 83.70 | 81.20 | Trained classifier | TF-IDF + logistic | 10,000 |
Note: Jev minus the classifier: −0.6 percentage points, from a bootstrap that resamples complaints within products.
Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json
### Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high
Where each routing policy sends complaints, percent of complaints, 2025, weighted
| outcome | policy | share |
| --- | --- | ---: |
| Routed to the consumer's product | Jev, triage below the cutoff | 73.40 |
| Routed to another product | Jev, triage below the cutoff | 10.20 |
| Sent to a person for triage | Jev, triage below the cutoff | 16.40 |
| Routed to the consumer's product | Luna, triage below the cutoff | 81.60 |
| Routed to another product | Luna, triage below the cutoff | 12.60 |
| Sent to a person for triage | Luna, triage below the cutoff | 5.90 |
| Routed to the consumer's product | TF-IDF classifier, triage below the cutoff | 53.70 |
| Routed to another product | TF-IDF classifier, triage below the cutoff | 3.80 |
| Sent to a person for triage | TF-IDF classifier, triage below the cutoff | 42.50 |
| Routed to the consumer's product | Jev recalibrated, triage below the cutoff | 60.40 |
| Routed to another product | Jev recalibrated, triage below the cutoff | 4.60 |
| Sent to a person for triage | Jev recalibrated, triage below the cutoff | 35.10 |
| Routed to the consumer's product | TF-IDF recalibrated, triage below the cutoff | 55 |
| Routed to another product | TF-IDF recalibrated, triage below the cutoff | 4 |
| Sent to a person for triage | TF-IDF recalibrated, triage below the cutoff | 41 |
| Routed to the consumer's product | Jev, route everything | 81.80 |
| Routed to another product | Jev, route everything | 18.10 |
| Sent to a person for triage | Jev, route everything | 0 |
| Routed to the consumer's product | Haiku adapter, route everything (no probabilities) | 84.40 |
| Routed to another product | Haiku adapter, route everything (no probabilities) | 15.60 |
| Sent to a person for triage | Haiku adapter, route everything (no probabilities) | 0 |
| Routed to the consumer's product | Send everything to triage | 0 |
| Routed to another product | Send everything to triage | 0 |
| Sent to a person for triage | Send everything to triage | 100 |
Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json
### Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried
Expected routing cost per thousand complaints against local labels used, weighted; whiskers run from the lowest to the highest draw
| conf_high | conf_low | cost | labels | model | n_labels |
| ---: | ---: | ---: | --- | --- | ---: |
| 11720 | 4815 | 6289 | 50 labels | Classifier, fit and calibrated on these labels | 50 |
| 4395 | 3471 | 3899 | 50 labels | Jev, recalibrated on these labels | 50 |
| 6643 | 4417 | 5303 | 100 labels | Classifier, fit and calibrated on these labels | 100 |
| 5308 | 3594 | 4128 | 100 labels | Jev, recalibrated on these labels | 100 |
| 6000 | 4733 | 5125 | 200 labels | Classifier, fit and calibrated on these labels | 200 |
| 4556 | 3471 | 3868 | 200 labels | Jev, recalibrated on these labels | 200 |
| 5851 | 4468 | 4792 | 500 labels | Classifier, fit and calibrated on these labels | 500 |
| 3769 | 3532 | 3613 | 500 labels | Jev, recalibrated on these labels | 500 |
| 5187 | 4272 | 4500 | 1,000 labels | Classifier, fit and calibrated on these labels | 1,000 |
| 4054 | 3471 | 3596 | 1,000 labels | Jev, recalibrated on these labels | 1,000 |
| 4497 | 4215 | 4326 | 2,000 labels | Classifier, fit and calibrated on these labels | 2,000 |
| 3839 | 3471 | 3587 | 2,000 labels | Jev, recalibrated on these labels | 2,000 |
| 4208 | 4026 | 4119 | 5,000 labels | Classifier, fit and calibrated on these labels | 5,000 |
| 3594 | 3471 | 3528 | 5,000 labels | Jev, recalibrated on these labels | 5,000 |
| 4145 | 3889 | 3993 | 10,000 labels | Classifier, fit and calibrated on these labels | 10,000 |
| 3594 | 3471 | 3546 | 10,000 labels | Jev, recalibrated on these labels | 10,000 |
| 3837 | 3683 | 3742 | 30,000 labels | Classifier, fit and calibrated on these labels | 30,000 |
| n/a | n/a | 3542 | 30,000 labels | Jev, recalibrated on these labels | 30,000 |
Note: Platt scaling in place of isotonic calibration gives the same reading.
Source: CFPB Consumer Complaint Database; labels from August 2023 to 2024; policy scored on 2025 complaints, weighted; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json
### Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint
Share of complaints closed with monetary relief that fall in each model's top tenth by P(money back), percent
| caught | kind | model |
| ---: | --- | --- |
| 58.20 | Decision engine | Jev 1.14.0 |
| 63.10 | Decision engine | OpenAI Luna |
| 51.80 | General language model | Claude Haiku adapter |
| 73.10 | Trained classifier | TF-IDF + logistic |
Note: Ranked over all complaints, the classifier's AUC is 0.936 and Jev's is 0.895; Jev minus the classifier is −0.041.
Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json
### Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier
AUC for complaints closed with monetary relief, by model and how the question was posed, with 95% intervals; 2,000 sampled 2025 complaints
| auc | conf_high | conf_low | group | label | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 0.947 | 0.959 | 0.932 | Classifier trained on past outcomes | Classifier | 2,000 |
| 0.881 | 0.912 | 0.847 | Jev | Jev: original question | 2,000 |
| 0.899 | 0.922 | 0.873 | Jev | Jev: reworded to predict payment | 2,000 |
| 0.903 | 0.926 | 0.878 | Jev | Jev: reworded + ten labeled examples | 2,000 |
| 0.913 | 0.931 | 0.891 | Jev | Jev: reworded + outcome criteria | 2,000 |
| 0.887 | 0.920 | 0.845 | Luna | Luna: original question | 2,000 |
| 0.903 | 0.929 | 0.875 | Luna | Luna: reworded to predict payment | 2,000 |
| 0.889 | 0.911 | 0.866 | Luna | Luna: reworded + ten labeled examples | 2,000 |
| 0.915 | 0.934 | 0.891 | Luna | Luna: reworded + outcome criteria | 2,000 |
Note: Only about one complaint in forty ended in money back, so the intervals are wide.
Source: CFPB Consumer Complaint Database, 2025 complaints; 2,000 drawn at random from the sample, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json
### Table 6. For money back, the classifier matches Jev with about a thousand past complaints
Money-back complaints caught in the top tenth, percent, against past complaints used to train the classifier, weighted
| caught | conf_high | conf_low | labels |
| ---: | ---: | ---: | --- |
| 34 | 51.90 | 11.20 | 100 complaints |
| 49.30 | 59.50 | 30.50 | 300 complaints |
| 58.80 | 65.30 | 53.10 | 1,000 complaints |
| 66.40 | 69.10 | 63.60 | 3,000 complaints |
| 69.90 | 72.90 | 66.60 | 10,000 complaints |
| 74.90 | 75.90 | 74.20 | 30,000 complaints |
Source: CFPB Consumer Complaint Database, 2025 distinct narratives with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026; ten refits per size, three at the largest
Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json
## All estimates
Every fact in facts.json (73), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| agree_diff_jev_tfidf | −0.6 | percentage points | jev minus tfidf | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −1.9 to 0.7 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| agree_jev | 81.9% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| agree_jev_equal_products | 75.2% | percent of complaints, each product counted equally | mean agreement across the eleven products | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| agree_tfidf | 82.4% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| agree_tfidf_equal_products | 63.6% | percent of complaints, each product counted equally | mean agreement across the eleven products | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| agree_tfidf_unweighted_fit | 78.2% | percent of complaints | agreement for a classifier fit without weights | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_money_diff_jev_tfidf | −0.041 | difference in AUC | Jev minus TF-IDF | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.054 to −0.029 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_money_jev | 0.895 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.881 to 0.908 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_money_product_only | 0.906 | area under the ROC curve | ranking complaints by their product's relief rate in the training years | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_money_tfidf | 0.936 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.928 to 0.945 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_within_adapter | 0.730 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.704 to 0.756 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_within_diff_tfidf_jev | +0.027 | difference in AUC | classifier minus Jev, within product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.009 to 0.065 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_within_jev | 0.742 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.709 to 0.770 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_within_luna | 0.743 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.706 to 0.772 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| auc_within_tfidf | 0.769 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.736 to 0.802 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| capture10_diff_tfidf_jev | +14.9 | percentage points | classifier minus Jev | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 8.9 to 20.4 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| capture10_jev | 58.2% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 53.0% to 63.4% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| capture10_tfidf | 73.1% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 68.4% to 77.9% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| cost1k_jev | $0.05 | US dollars per 1,000 complaints | mean billed cost per call, list price | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| lc_capture_1000 | 58.8% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| lc_capture_300 | 49.3% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 300 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| lc_capture_3000 | 66.4% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 3,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| loss_diff_jev_cal | −$195 | US dollars per 1,000 complaints | jev_cal minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −$419 to $27 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| loss_diff_jev_cut | +$368 | US dollars per 1,000 complaints | jev_cut minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $97 to $655 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| loss_jev_all | $5,442 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $5,041 to $5,861 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_jev_cal | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_jev_cut | $4,054 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,740 to $4,384 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cal | $3,666 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,466 to $3,885 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cut | $3,685 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,482 to $3,893 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_triage_all | $6,000 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $6,000 to $6,000 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| lv_clf_500 | $4,792 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_full | $3,742 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_500 | $3,613 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_full | $3,542 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_raw_jev | $4,054 | US dollars per thousand complaints | raw Jev, no local fitting | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| mis_jev_cal | 4.6% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| mis_jev_cut | 10.2% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| mis_tfidf_cal | 4.0% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| mis_tfidf_cut | 3.8% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| misroute_cost | $30 | US dollars per misrouted complaint | assumed cost: three days of delay at $10 | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | CFPB Consumer Complaint Database | |
| money_extra | $120 | US dollars per misrouted complaint that ends in money back | assumed extra cost | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | CFPB Consumer Complaint Database | |
| n_complaints | 10,000 | complaints | sample size | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| n_population | 519,841 | complaints | 2025 complaints within the length limit that the sample stands for | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| narrative_cap | 3,000 | characters | length cap of the stored narratives | CFPB narratives in the July 2026 copy | 2025 | | | not applicable | CFPB Consumer Complaint Database | |
| queue_diff_value_cost | +79 | per 1,000 complaints | value minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −56 to 233 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| queue_diff_value_left_money | −0.43 | per 1,000 complaints | value minus reviewing the least certain, left_money | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.91 to 0.09 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| queue_diff_value_per_minute_cost | −425 | per 1,000 complaints | value_per_minute minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −639 to −198 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| queue_diff_value_per_minute_left | −12.15 | per 1,000 complaints | value_per_minute minus reviewing the least certain, left | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −18.81 to −4.95 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| queue_minutes | 400 | reviewer minutes per 1,000 complaints | assumed review budget | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | CFPB Consumer Complaint Database | |
| queue_none_cost | $5,827 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $5,402 to $6,277 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_left | 181.5 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 168.2 to 195.6 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_left_money | 3.18 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 2.26 to 4.20 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_reviewed | 0.0% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_cost | $3,499 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,144 to $3,852 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_left | 110.4 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 99.4 to 121.6 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_left_money | 1.57 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.89 to 2.31 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_reviewed | 14.3% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 13.1% to 15.6% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_cost | $3,579 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,229 to $3,944 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_left | 114.7 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 103.6 to 126.1 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_left_money | 1.14 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.54 to 1.88 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_cost | $3,074 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $2,732 to $3,429 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_left | 98.2 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 87.0 to 109.5 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_left_money | 1.06 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.50 to 1.73 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_reviewed | 20.2% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 18.8% to 21.6% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_reviewed | 14.1% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 13.0% to 15.3% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| route_cutoff | 0.8 | probability that the top product is right | 1 minus triage cost over misroute cost | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | CFPB Consumer Complaint Database | |
| share_relief | 2.6% | percent of complaints | company response is closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| share_too_long | 7.7% | percent of deduplicated 2025 complaints with a narrative | narratives over 3,000 characters, excluded | deduplicated 2025 CFPB complaints with a narrative | 2025 | | | full count | CFPB Consumer Complaint Database | |
| tri_jev_cal | 35.1% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| tri_jev_cut | 16.4% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| tri_tfidf_cal | 41.0% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| tri_tfidf_cut | 42.5% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | |
| triage_cost | $6 | US dollars per complaint sent to triage | assumed cost: four minutes of a triage handler | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | CFPB Consumer Complaint Database | |
## Definitions
The chapter marks no glossary term.
## Methods
Two decision engines and a general language model read each sampled complaint narrative and answered two typed questions: which product it is about, and whether the consumer describes money the company could return. A classifier trained on earlier complaints answered the same two. Nothing was rated by hand.
### Which complaints
CFPB withdrew complaint narratives from public access in August 2026, so the only copy available keeps the first 3,000 characters of each narrative. The sample keeps only complaints whose whole narrative fits, which excludes 7.7% of 2025 complaints with a narrative. Every result describes complaints within that length.
The sample holds the same number of complaints for each of the eleven products, and weights restore each product's share of 2025 complaints.
### Labels
Routing is scored against the product the consumer selected when filing. Consumers sometimes pick the wrong product, so a disagreement can be the consumer's mistake rather than the engine's.
The money-back question is scored against the company's recorded response, closed with monetary relief. That outcome depends on the company as well as the complaint.
### Intervals
Intervals are 95% bootstrap intervals that resample complaints within product strata, keep each product's share fixed and use the same resamples for every policy. They describe sampling error for 2025 complaints and say nothing about other years or other companies.
## What these data cannot show
The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/resolving-and-routing-customer-complaints/build.py (sha256 ad8930e6ff310f8e48d53f32dc53dc37c1156cc14259c5adbe4b587e8d5a291f)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 20c650ec38ef9810222843d6e40613ef2d6545f645f749ed57035e7fe6d7ea92 |
| figures/agree-by-model.json | 1b7f77c56eee87f3469be5ac6cf9affb228c657e53801d78d2e3366e9a7fc1f8 |
| figures/cost-by-labels.json | 34360a16ffe4d9b7133cacf6e24209e9e0410bcb594a1d42581a2fab26e5a5ce |
| figures/money-back-caught.json | 03c660401c1437e49780d3a291312f0aa46f0a7debdc69855fd6a151018ea1e8 |
| figures/money-back-prompts.json | 0d8296a19c6854ea1f9c9ef1851b6e05b607916c3ef3ad6e632a3a2c0e89d754 |
| figures/outcomes-to-catch-up.json | 3a66877ee2c772f4622dda2511a750923589ebfad0c9b74ee85deca531a3aa95 |
| figures/where-complaints-go.json | 43adef141cfb6ed2a770238c094ba0436f6ea8ca0c0f591d0e1d38a6b6d305b8 |
| meta.json | e7892b08742ced88d370d9d8660f15d3481e33f08529e79bc4f5923f2673ef90 |
| public/data/resolving-and-routing-customer-complaints/flow.json | 85255672e8768ac873e89cd12604fb38c55cd4c0f5042805034be293c2a785ef |
| public/data/resolving-and-routing-customer-complaints/items.json | 27123950e0178d495717b23b6f360caf78b7babd3f5f88f989218637fe0dd855 |
| public/data/resolving-and-routing-customer-complaints/policy.json | c2c0475becd99606c6f49869aeeaece194b993c52165a170c0114b6bec2ca6b5 |
| public/data/resolving-and-routing-customer-complaints/queue.json | 98f9a4750c6905d6d2c87450977d629f97d0a14659f845a043eca707a0b9bc4c |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report
- Report (Markdown): https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/facts.json
- Table 1, figure "agree-by-model" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json
- Table 2, figure "where-complaints-go" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json
- Table 3, figure "cost-by-labels" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json
- Table 4, figure "money-back-caught" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json
- Table 5, figure "money-back-prompts" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json
- Table 6, figure "outcomes-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# Allocating review attention to code changes: report
Chapter 2.2 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes. Table n of this report is figure n of the chapter.
## Identification
- Title: Allocating review attention to code changes
- Chapter: 2.2 (Part II, Scarce attention; Case studies)
- Question: Which agent-generated code changes should get a specialist's limited review time, and how much does a good score on a test set say about real work?
- Dek: On patches written so that every one passes the required tests, the models rank defective changes well above acceptable ones while diff size does no better than chance, but on real agent patches the same models do only modestly better than chance.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
- Constructed patches. Patches on four small projects written for this booklet, with hidden issue tests; weights: unweighted; evenly split by construction
- SWE-rebench. Issues in sampled repositories; agent patches from the Nebius OpenHands trajectories; weights: sample_weight restores the outcome mix of the patches
- SWE-rebench OpenHands trajectories. Agent patches and the resolved outcome; weights: sample_weight restores the outcome mix of the patches
- Canonical URL: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes
- Report: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report
- Markdown: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report.md
- BibTeX key: singh2026allocatingreviewattentiontocodechanges
```bibtex
@misc{singh2026allocatingreviewattentiontocodechanges,
author = {Singh, Vishal},
title = {Allocating review attention to code changes},
year = {2026},
url = {https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes},
note = {In Decision AI. Published 30 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 10 | | | unweighted; evenly split by construction | Constructed patches | found10_luna |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 9 | | | unweighted; evenly split by construction | Constructed patches | found10_jev |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 5 | | | unweighted; evenly split by construction | Constructed patches | found10_size |
## Findings
An AI agent opens more code changes in a week than a specialist can read. In the workflow this chapter studies, every change has passed the required tests, which always run, and no model's opinion replaces them. That is true of the patches written for this booklet. The real benchmark patches later in the chapter were not filtered this way. The question is which changes get the specialist's limited time. This chapter ranks changes by model judgment and by simple rules, then asks how many problems each ranking finds.
On patches written for this booklet, where the flaw is there to be read in the diff, the models find defects far faster than sorting by size: Luna's ten riskiest hold 10 defects, Jev's 9 and the ten largest 5. On real agent patches the same models show no clear advantage over size. So a good score on a tidy test set is weak evidence about your own queue. The written set's mix is unlike any real queue, and the real set's label is a benchmark outcome, not a count of bugs that would ship.
### A test set where the answer is known
Four small projects were written for this booklet, with 20 issues and two patches for each: one acceptable and one with a real flaw, such as a removed permission check or an off-by-one error. All 40 patches pass the required tests. A hidden test, never shown to any model, settles which is which.
These patches are written for teaching and split evenly, so far more of them are defective than in a real queue. Results from this set are not pooled with real patches.
### Models find the flaws sooner than size does
Each model read the issue, the diff and the guidance, and answered: would this patch fail tests written to check that this issue is fixed? Sort by that answer and review from the top.
Table 1. Ranking by the models finds the defects much sooner than ranking by diff size
Sorting by size finds defects no faster than chance, because a small change can remove a permission check. A keyword rule looking for words like "auth" or "security" in added lines matches 0 of these patches, a quirk of how the set was written and not a finding about real code. Within each issue, Jev rates the defective patch riskier in 18.5 of 20 pairs and Luna in 18, so these rankings do more than react to which issue is harder.
Table 2. One question does most of the work, and equal weights dilute it
Splitting the judgment into smaller questions is a popular idea, and research on simple scoring rules gives it real support [1]. Here the result depends on the weights. For Jev, the single question "does it address the issue?" scores 0.88, while an equal-weight score across all the smaller questions scores 0.66, because questions about security or public interfaces say little about these particular flaws. Smaller questions still help a reviewer see why a patch looks risky, and a different mix of defects could favor them.
### Spending a limited review budget
The panel below turns a ranking into a week's budget. Suppose a defect that ships costs $2,000, a reviewer costs $150 an hour and finds a defect with probability 0.7. All three are assumptions, and a team would estimate them from incident records and review logs.
Double the hours. Which policy gains most from the extra time, and what happens to the gap between the models and random order?
The panel also includes a ranking by risk per review minute. It barely differs here because these patches take similar time to review; it would matter more in a queue with some very large changes.
### Real agent patches are much harder
The written patches have flaws a careful reader can see. Real failures are often subtler. To test that, the same models read 1,941 patches that one coding agent wrote for 259 real open-source issues. 44% of them failed the benchmark's tests for their issue. Another 59 runs ended with no change at all, usually when the agent ran out of steps. Those are left out, because there is nothing to review, and a check for an empty diff catches every one of them.
Table 3. On real agent patches, neither the models nor the rules clearly beat sorting by size
Across all patches, Jev scores 0.60, Luna 0.62 and largest diff first scores 0.60, so Jev's lead over size is +0.01 (95% CI −0.03 to 0.05). Comparing only patches for the same issue, in the 215 issues that have both outcomes, Jev scores 0.58, Luna 0.56 and size 0.52, and Jev's lead is +0.06 (95% CI −0.04 to 0.16). Both ranges include zero. That drop is the lesson to keep: a score on a tidy test set says little about performance on your real work.
This label cannot price a review budget. "Unresolved" also includes patches that existing tests would have caught and patches that do not apply, so it is not the population a specialist would see.
All the visible tests pass, but the issue's acceptance criteria look incomplete. What further evidence should the system ask for? Then look at the two sets of results: what would you test on your own code before giving a model's ranking any say over a specialist's time?
Show the budget panel first and ask students which policy they would buy. Then show the real-patch chart and ask what they would now want to see from a vendor. A semantic judgment never justifies skipping a required test, and the class should be able to say why.
### What these data cannot show
The written patches were made for this booklet with a few kinds of defect, so their scores are not an estimate for any real repository. The real patches come from one agent on one set of issues, their label is not the same as a defect that would ship, and the file-ownership map is an unreviewed draft. These open-source repositories may appear in the models' training data, which comparing patches for the same issue reduces and does not remove. No ranking here was used by real reviewers, whose detection rate and time are assumed, so nothing shows that a team would catch more defects in practice.
## Tables
### Table 1. Ranking by the models finds the defects much sooner than ranking by diff size
Defective patches among the first reviews in each order, constructed patches, if each review reveals the defect
| found | policy | reviews |
| ---: | --- | ---: |
| 1 | Jev 1.14.0 | 1 |
| 2 | Jev 1.14.0 | 2 |
| 3 | Jev 1.14.0 | 3 |
| 4 | Jev 1.14.0 | 4 |
| 5 | Jev 1.14.0 | 5 |
| 5 | Jev 1.14.0 | 6 |
| 6 | Jev 1.14.0 | 7 |
| 7 | Jev 1.14.0 | 8 |
| 8 | Jev 1.14.0 | 9 |
| 9 | Jev 1.14.0 | 10 |
| 10 | Jev 1.14.0 | 11 |
| 11 | Jev 1.14.0 | 12 |
| 12 | Jev 1.14.0 | 13 |
| 12 | Jev 1.14.0 | 14 |
| 12 | Jev 1.14.0 | 15 |
| 12 | Jev 1.14.0 | 16 |
| 13 | Jev 1.14.0 | 17 |
| 13 | Jev 1.14.0 | 18 |
| 14 | Jev 1.14.0 | 19 |
| 14 | Jev 1.14.0 | 20 |
| 14 | Jev 1.14.0 | 21 |
| 14 | Jev 1.14.0 | 22 |
| 14 | Jev 1.14.0 | 23 |
| 15 | Jev 1.14.0 | 24 |
| 16 | Jev 1.14.0 | 25 |
| 16 | Jev 1.14.0 | 26 |
| 17 | Jev 1.14.0 | 27 |
| 17 | Jev 1.14.0 | 28 |
| 17 | Jev 1.14.0 | 29 |
| 17 | Jev 1.14.0 | 30 |
| 17 | Jev 1.14.0 | 31 |
| 18 | Jev 1.14.0 | 32 |
| 18 | Jev 1.14.0 | 33 |
| 19 | Jev 1.14.0 | 34 |
| 19 | Jev 1.14.0 | 35 |
| 19 | Jev 1.14.0 | 36 |
| 19 | Jev 1.14.0 | 37 |
| 20 | Jev 1.14.0 | 38 |
| 20 | Jev 1.14.0 | 39 |
| 20 | Jev 1.14.0 | 40 |
| 1 | OpenAI Luna | 1 |
| 2 | OpenAI Luna | 2 |
| 3 | OpenAI Luna | 3 |
| 4 | OpenAI Luna | 4 |
| 5 | OpenAI Luna | 5 |
| 6 | OpenAI Luna | 6 |
| 7 | OpenAI Luna | 7 |
| 8 | OpenAI Luna | 8 |
| 9 | OpenAI Luna | 9 |
| 10 | OpenAI Luna | 10 |
| 11 | OpenAI Luna | 11 |
| 12 | OpenAI Luna | 12 |
| 13 | OpenAI Luna | 13 |
| 14 | OpenAI Luna | 14 |
| 14 | OpenAI Luna | 15 |
| 15 | OpenAI Luna | 16 |
| 16 | OpenAI Luna | 17 |
| 16 | OpenAI Luna | 18 |
| 16 | OpenAI Luna | 19 |
| 17 | OpenAI Luna | 20 |
| 17 | OpenAI Luna | 21 |
| 17 | OpenAI Luna | 22 |
| 17 | OpenAI Luna | 23 |
| 18 | OpenAI Luna | 24 |
| 18 | OpenAI Luna | 25 |
| 18 | OpenAI Luna | 26 |
| 18 | OpenAI Luna | 27 |
| 18 | OpenAI Luna | 28 |
| 18 | OpenAI Luna | 29 |
| 18 | OpenAI Luna | 30 |
| 18 | OpenAI Luna | 31 |
| 18 | OpenAI Luna | 32 |
| 18 | OpenAI Luna | 33 |
| 18 | OpenAI Luna | 34 |
| 18 | OpenAI Luna | 35 |
| 19 | OpenAI Luna | 36 |
| 20 | OpenAI Luna | 37 |
| 20 | OpenAI Luna | 38 |
| 20 | OpenAI Luna | 39 |
| 20 | OpenAI Luna | 40 |
| 1 | Claude Haiku adapter | 1 |
| 2 | Claude Haiku adapter | 2 |
| 3 | Claude Haiku adapter | 3 |
| 4 | Claude Haiku adapter | 4 |
| 5 | Claude Haiku adapter | 5 |
| 6 | Claude Haiku adapter | 6 |
| 7 | Claude Haiku adapter | 7 |
| 8 | Claude Haiku adapter | 8 |
| 9 | Claude Haiku adapter | 9 |
| 10 | Claude Haiku adapter | 10 |
| 10 | Claude Haiku adapter | 11 |
| 10 | Claude Haiku adapter | 12 |
| 11 | Claude Haiku adapter | 13 |
| 12 | Claude Haiku adapter | 14 |
| 12 | Claude Haiku adapter | 15 |
| 13 | Claude Haiku adapter | 16 |
| 14 | Claude Haiku adapter | 17 |
| 14 | Claude Haiku adapter | 18 |
| 14 | Claude Haiku adapter | 19 |
| 15 | Claude Haiku adapter | 20 |
| 15 | Claude Haiku adapter | 21 |
| 15 | Claude Haiku adapter | 22 |
| 15 | Claude Haiku adapter | 23 |
| 15 | Claude Haiku adapter | 24 |
| 15 | Claude Haiku adapter | 25 |
| 16 | Claude Haiku adapter | 26 |
| 16 | Claude Haiku adapter | 27 |
| 17 | Claude Haiku adapter | 28 |
| 18 | Claude Haiku adapter | 29 |
| 18 | Claude Haiku adapter | 30 |
| 18 | Claude Haiku adapter | 31 |
| 18 | Claude Haiku adapter | 32 |
| 18 | Claude Haiku adapter | 33 |
| 18 | Claude Haiku adapter | 34 |
| 18 | Claude Haiku adapter | 35 |
| 18 | Claude Haiku adapter | 36 |
| 19 | Claude Haiku adapter | 37 |
| 19 | Claude Haiku adapter | 38 |
| 20 | Claude Haiku adapter | 39 |
| 20 | Claude Haiku adapter | 40 |
| 0 | Largest diff first | 1 |
| 0 | Largest diff first | 2 |
| 1 | Largest diff first | 3 |
| 2 | Largest diff first | 4 |
| 2 | Largest diff first | 5 |
| 2 | Largest diff first | 6 |
| 3 | Largest diff first | 7 |
| 3 | Largest diff first | 8 |
| 4 | Largest diff first | 9 |
| 5 | Largest diff first | 10 |
| 5 | Largest diff first | 11 |
| 6 | Largest diff first | 12 |
| 7 | Largest diff first | 13 |
| 7 | Largest diff first | 14 |
| 8 | Largest diff first | 15 |
| 8 | Largest diff first | 16 |
| 8 | Largest diff first | 17 |
| 8 | Largest diff first | 18 |
| 9 | Largest diff first | 19 |
| 9 | Largest diff first | 20 |
| 10 | Largest diff first | 21 |
| 10 | Largest diff first | 22 |
| 10 | Largest diff first | 23 |
| 11 | Largest diff first | 24 |
| 12 | Largest diff first | 25 |
| 12 | Largest diff first | 26 |
| 12 | Largest diff first | 27 |
| 13 | Largest diff first | 28 |
| 13 | Largest diff first | 29 |
| 14 | Largest diff first | 30 |
| 15 | Largest diff first | 31 |
| 15 | Largest diff first | 32 |
| 16 | Largest diff first | 33 |
| 17 | Largest diff first | 34 |
| 17 | Largest diff first | 35 |
| 18 | Largest diff first | 36 |
| 19 | Largest diff first | 37 |
| 19 | Largest diff first | 38 |
| 20 | Largest diff first | 39 |
| 20 | Largest diff first | 40 |
| 0.500 | Random order (average) | 1 |
| 1 | Random order (average) | 2 |
| 1.50 | Random order (average) | 3 |
| 2 | Random order (average) | 4 |
| 2.50 | Random order (average) | 5 |
| 3 | Random order (average) | 6 |
| 3.50 | Random order (average) | 7 |
| 4 | Random order (average) | 8 |
| 4.50 | Random order (average) | 9 |
| 5 | Random order (average) | 10 |
| 5.50 | Random order (average) | 11 |
| 6 | Random order (average) | 12 |
| 6.50 | Random order (average) | 13 |
| 7 | Random order (average) | 14 |
| 7.50 | Random order (average) | 15 |
| 8 | Random order (average) | 16 |
| 8.50 | Random order (average) | 17 |
| 9 | Random order (average) | 18 |
| 9.50 | Random order (average) | 19 |
| 10 | Random order (average) | 20 |
| 10.50 | Random order (average) | 21 |
| 11 | Random order (average) | 22 |
| 11.50 | Random order (average) | 23 |
| 12 | Random order (average) | 24 |
| 12.50 | Random order (average) | 25 |
| 13 | Random order (average) | 26 |
| 13.50 | Random order (average) | 27 |
| 14 | Random order (average) | 28 |
| 14.50 | Random order (average) | 29 |
| 15 | Random order (average) | 30 |
| 15.50 | Random order (average) | 31 |
| 16 | Random order (average) | 32 |
| 16.50 | Random order (average) | 33 |
| 17 | Random order (average) | 34 |
| 17.50 | Random order (average) | 35 |
| 18 | Random order (average) | 36 |
| 18.50 | Random order (average) | 37 |
| 19 | Random order (average) | 38 |
| 19.50 | Random order (average) | 39 |
| 20 | Random order (average) | 40 |
Note: AUC on this set: Luna 0.89, Jev 0.79, Haiku adapter 0.78, largest diff first 0.45.
Source: Constructed patches written for this booklet, October 2026; 40 patches on 20 issues; unweighted
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/defects-found-by-review.json
### Table 2. One question does most of the work, and equal weights dilute it
AUC of each typed question for finding defective patches, constructed patches, by engine
| auc | model | signal |
| ---: | --- | --- |
| 0.785 | Jev 1.14.0 | Whole question: would it fail? |
| 0.662 | Jev 1.14.0 | Equal weight on the six questions |
| 0.877 | Jev 1.14.0 | Addresses the issue |
| 0.535 | Jev 1.14.0 | Changes a public interface |
| 0.499 | Jev 1.14.0 | Security sensitive |
| 0.666 | Jev 1.14.0 | Unrelated changes |
| 0.581 | Jev 1.14.0 | Tests added |
| 0.797 | Jev 1.14.0 | Too little context to judge |
| 0.887 | OpenAI Luna | Whole question: would it fail? |
| 0.681 | OpenAI Luna | Equal weight on the six questions |
| 0.874 | OpenAI Luna | Addresses the issue |
| 0.583 | OpenAI Luna | Changes a public interface |
| 0.502 | OpenAI Luna | Security sensitive |
| 0.586 | OpenAI Luna | Unrelated changes |
| 0.512 | OpenAI Luna | Tests added |
| 0.764 | OpenAI Luna | Too little context to judge |
Source: Constructed patches written for this booklet, October 2026; 40 patches on 20 issues; unweighted
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/which-question-helps.json
### Table 3. On real agent patches, neither the models nor the rules clearly beat sorting by size
AUC for the benchmark's unresolved label, across all patches and within the same issue, with 95% intervals
| across | hi1 | hi2 | lo1 | lo2 | n1 | n2 | policy | within |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | ---: |
| 0.604 | 0.642 | 0.646 | 0.559 | 0.506 | 1941 | 1941 | Jev 1.14.0, whole question | 0.580 |
| 0.618 | 0.665 | 0.642 | 0.574 | 0.490 | 1941 | 1941 | OpenAI Luna, whole question | 0.557 |
| 0.608 | 0.651 | 0.624 | 0.567 | 0.502 | 1941 | 1941 | Claude Haiku adapter, whole question | 0.562 |
| 0.572 | 0.614 | 0.678 | 0.530 | 0.524 | 1941 | 1941 | Jev 1.14.0, equal weight | 0.595 |
| 0.542 | 0.580 | 0.667 | 0.498 | 0.503 | 1941 | 1941 | Jev 1.14.0, fitted weights | 0.578 |
| 0.597 | 0.648 | 0.608 | 0.554 | 0.442 | 1941 | 1941 | Largest diff first | 0.517 |
| 0.543 | 0.594 | 0.601 | 0.494 | 0.444 | 1941 | 1941 | File-ownership rule | 0.512 |
| 0.575 | 0.623 | 0.608 | 0.531 | 0.438 | 1941 | 1941 | Keyword rule | 0.515 |
Source: SWE-rebench and Nebius OpenHands trajectories, issues created by 2025; 2,000 patches, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/same-issue-is-harder.json
## All estimates
Every fact in facts.json (33), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| auc_across_diff_jev_size | +0.01 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.03 to 0.05 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_diff_luna_size | +0.02 | difference in AUC | luna minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.01 to 0.06 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_jev | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.56 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_luna | 0.62 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.57 to 0.66 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_size | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.55 to 0.65 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_addr_jev | 0.88 | area under the ROC curve | the addresses-the-issue question alone | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_adapter | 0.78 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.64 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_jev | 0.79 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.69 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_luna | 0.89 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.77 to 0.99 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_size | 0.45 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.33 to 0.55 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_dawes_jev | 0.66 | area under the ROC curve | equal weight on six direction-fixed questions | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_within_diff_jev_size | +0.06 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.04 to 0.16 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_diff_luna_size | +0.04 | difference in AUC | luna minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.05 to 0.13 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_jev | 0.58 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.51 to 0.65 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_luna | 0.56 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.49 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_size | 0.52 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.44 to 0.61 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| detect_rate | 0.7 | share of defects a reviewer finds | assumed reviewer detection rate | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| found10_jev | 9 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| found10_luna | 10 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| found10_size | 5 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| hour_cost | $150 | US dollars per reviewer hour | assumed cost | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| keyword_hits | 0 | patches of 40 | patches whose added lines match auth, security or api | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| miss_cost | $2,000 | US dollars per defect that ships | assumed cost | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| n_con | 40 | patches | constructed patches | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| n_empty | 59 | agent runs | runs that ended with no diff, excluded | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_hist | 1,941 | patches | sampled agent patches | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_hist_issues | 259 | issues | issues in the sample | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_issues | 20 | issues | issues, each with one acceptable and one defective patch | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| n_mixed | 215 | issues | issues with both resolved and unresolved patches | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| pairs_jev | 18.5 | issues of 20 | defective patch rated riskier than the acceptable patch for the same issue; a tie is split evenly | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| pairs_luna | 18 | issues of 20 | defective patch rated riskier than the acceptable patch for the same issue; a tie is split evenly | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| share_empty | 2.6% | percent of agent runs | runs that ended with no diff, weighted | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| share_unresolved | 43.9% | percent of patches | benchmark outcome is unresolved | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
## Definitions
The chapter marks no glossary term.
## Methods
Two decision engines and a general language model read each patch with its issue and the repository's guidance and answered seven typed questions. Constructed patches and historical patches are reported separately and never pooled.
### The constructed patches
Four small projects were written for this booklet, with an issue, an acceptable patch and a defective patch for each issue. Every patch passes the project's required tests, and a hidden test settles whether it fixes the issue. The patches are evenly split by construction, and the keyword rule is a poor fit by construction, so neither says anything about real repositories.
The budget view uses the assumed reviewer costs and allows part of a patch at the budget's edge, which is an approximation to choosing whole patches.
### The historical patches
One agent's patches for real open-source issues. The label is the benchmark's unresolved outcome, which also counts patches that existing tests would have caught and patches that do not apply. So this set shows how well an outcome can be predicted, and it does not count defects that would reach production. These repositories and issues may appear in the models' training data; comparing patches for the same issue reduces that problem and does not remove it.
Models saw diffs without the agent's leftover scripts. The file-ownership map is a draft that was not reviewed by the repositories' maintainers.
### Intervals
Constructed intervals resample issues, keeping both patches together. Historical intervals resample issues within repositories, with fewer replicates because same-issue AUC is slow to compute. A missing answer is ranked at the sample mean, and refusals are counted in the chapter.
## What these data cannot show
The written patches were made for this booklet with a few kinds of defect, so their scores are not an estimate for any real repository. The real patches come from one agent on one set of issues, their label is not the same as a defect that would ship, and the file-ownership map is an unreviewed draft. These open-source repositories may appear in the models' training data, which comparing patches for the same issue reduces and does not remove. No ranking here was used by real reviewers, whose detection rate and time are assumed, so nothing shows that a team would catch more defects in practice.
## References
1. Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571-582. https://doi.org/10.1037/0003-066X.34.7.571
## Provenance
- Build script: decision_ai/chapters/allocating-review-attention-to-code-changes/build.py (sha256 9a26845e7af2cd3fee289e9630841a19c3c5f921d9c41cba96e0adc5f9873221)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 101cdf86f86fe57545d87f5eac1eaeb4f36c590aa3360dc1876ba4f0f007dc4c |
| figures/defects-found-by-review.json | dfd3ed5cba0f31b34d0407d9fd00486cb9798136c11671a679cb4e6f1acc2c80 |
| figures/same-issue-is-harder.json | ab9e25f79f5342897a30c92d8619bf051fa69489d0efbd7e18cb576c81fdec62 |
| figures/which-question-helps.json | e2c337d5d02164f2a7c72749427dbae8ebe71e4a3ad4b564f85a225764fbad2c |
| meta.json | efa66d99b4dc878a54b8c4f5a3459e084b947f9039bf31a7f2183e2cfe670670 |
| public/data/allocating-review-attention-to-code-changes/budget.json | b7794c5b863c43e51f8978175b4680051349409ca49b5a67d57e4f36476253c9 |
| public/data/allocating-review-attention-to-code-changes/reveal.json | 8782bc8d18cf279067f365dd981c7681a78c4b6722b4f6f92e75f2d5c56053ea |
| refs.json | f02a9f2b53002b0cc1ba8e133d2caf0bd4e7b7f4593ca1534ead259450db78d3 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report
- Report (Markdown): https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/facts.json
- Table 1, figure "defects-found-by-review" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/defects-found-by-review.json
- Table 2, figure "which-question-helps" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/which-question-helps.json
- Table 3, figure "same-issue-is-harder" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/same-issue-is-harder.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# Deciding when an agent is finished: report
Chapter 3.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished. Table n of this report is figure n of the chapter.
## Identification
- Title: Deciding when an agent is finished
- Chapter: 3.1 (Part III, Acting and checking; Case studies)
- Question: When an agent says it is done, should a controller accept that, check further or call a person, and what can a model tell from reading the trace?
- Dek: On tasks written for this booklet, an agent's first finish met every checked requirement in 87.5% of runs, sending it back on a model's judgment raised that to 90.0%, about what sending every finish back achieves, and telling it which requirement failed raised that to 94.0%.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
- Constructed agent tasks. Tasks written for this booklet, each run five times with injected faults by one agent; weights: unweighted; the mix describes the test harness
- Canonical URL: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished
- Report: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report
- Markdown: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report.md
- BibTeX key: singh2026decidingwhenanagentisfinished
```bibtex
@misc{singh2026decidingwhenanagentisfinished,
author = {Singh, Vishal},
title = {Deciding when an agent is finished},
year = {2026},
url = {https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished},
note = {In Decision AI. Published 30 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 87.5% | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_accept |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 90.0% | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_jev |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 94.0% | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_check_named |
| agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 12.5% | | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | share_premature |
## Findings
An AI agent finishes a task and says it is done. Someone, or something, now has to decide whether to believe it. Accept a wrong "done" and a person finds the mistake later, when it is expensive to fix. Check every "done" and the agent saves far less time than it promised. This chapter asks whether a model reading the agent's record can make that call.
The agent in this test said it was done too early on 13% of its first finishes. Sending work back on a model's judgment raised the share of runs with every checked requirement met from 88% to 90%, about the same as sending every finish back, while a check that named the failed requirement reached 94%. So telling the agent what failed mattered more than asking it to look again. The tasks were written for this booklet and that check is the grader itself, so this shows the pattern in one test and not what a real test suite would deliver.
### The test
One agent worked through 40 short tasks: update a document, answer from a set of records, repair a small program. Each task ran several times, and the harness injected a different problem each time, such as a failed tool call or a required input held back. That produced 200 moments when the agent said it was done, and at 25 of them a rule-checked requirement was still unmet.
These tasks and faults were written for this booklet, so their mix describes the test and not any real workload. The agent is a generative OpenAI Luna model, the same model family as the Luna decision engine judging it.
Table 1. The agent's "done" is usually right and wrong often enough to matter
Answering from records is where the agent stopped too early most often, 22% against 8% elsewhere, though a handful of tasks hold many of these finishes, so that gap is not firmly established. A wrong number looks just like a right one, and the record shows no failed test to give it away.
### Reading the record tells a model something about whether the work is done
Each model read the record at the moment of the finish attempt and gave a probability that all the criteria had been met.
Table 2. Reading the trace tells the models something about whether the work is done, with wide intervals on this few tasks
The scores are 0.71 for Jev and 0.76 for the general language model, well short of perfect, with wide intervals. They also depend on how finishes with an unscored rater-only criterion are treated, since there are 34 of them: Jev's score is 0.72 leaving them out and 0.56 counting them as unmet. Jev's choice of next action points the same way. On the premature finishes, it still says to finish 96% of the time. A record can show that an agent did the steps and sounded sure. It often cannot show that the answer is right, because the model judging it has no more access to the truth than the agent did.
### Priced on recorded finishes, a model saves a little over a person
Suppose accepting a premature finish costs $50 in rework and sending a finish to a person costs $5. Then a controller should accept only when the model puts the chance of being done above 0.9, because a mistake costs far more than a check.
Both costs are assumptions, and a platform team would measure them from rework records and review time. The comparison also assumes that a person catches every premature finish and that a check adds no delay beyond its cost.
Table 3. At high cutoffs the models land near the cost of sending every finish to a person
Accepting everything costs $625 per hundred finish attempts, and sending everything to a person costs $500. Models with a cutoff land around that second number: $413 for Jev and $388 for the language model. To get there, Jev sends 84% of the premature finishes to a person and also 59% of the finished work. A simple set of checks on the record, such as whether the last test run passed, stops almost nothing here: it catches 0% of premature finishes and costs $625, because in this test the agent's premature finishes look clean.
Compared with sending every finish to a person, Jev's gap is −$88 (95% CI −$173 to $0), the language model's is −$113 (95% CI −$215 to −$3) and the simple checks' is +$125 (95% CI −$150 to $451), per hundred finish attempts. Those are small savings at the edge of what these tests can detect, with Jev's range reaching zero and the language model's just short of it. The simple checks show no advantage.
The comparison stops at the finish attempt, and it assumes a person catches every premature finish. It cannot say what would happen if the work went back to the agent.
### Running the automated policies to the end
So the five automated policies were run; a person was not, and that comparison stays a calculation. When a controller sends work back, the agent receives a message and continues, and its final state is graded by the same rules. Every controller accepts after two send-backs. The tasks, faults and agent are the same, and each run repeats the recorded run exactly until its first send-back. Sharing that history makes the arms easier to compare, though it does not show how much a fresh agent run would vary at each step.
One run shows why the message matters. A task asked the agent to fix a billing bug: invoices within a five-day grace period were being charged a late fee, and fees after the grace period had to stay as they were. The agent changed the grace period, ran the tests, saw them pass and said it was done. But it had also shifted the formula for later fees, and no test it could see caught that. From there, the same first finish went four ways.
| Policy | What the agent was told | What it did | Result |
| --- | --- | --- | --- |
| Accept | Nothing | Stopped | The later-fee requirement stays unmet |
| Jev, below the cutoff | Re-check against each criterion, each time it was sent back | Re-read its code, re-ran the tests and said everything was fine each time | Still unmet |
| Independent check, silent | The same general message | The same re-reading | Still unmet |
| Independent check, named | That the third criterion is not met | Read the older test file, restored the original formula and re-ran the tests | Every checked requirement met |
This run shows the common pattern and not a guarantee: the named message rescued more runs than the general one, and it left others unrepaired, as the counts below show.
The agent's first finish meets every checked requirement in 87.5% of runs. If it is sent back whenever Jev's probability falls at or below 0.9, what share of runs would end with every requirement met?
Sending back every finish, with no judgment at all, reaches 90% at 9.7 steps per run, against 5.0 for accepting. Using Jev to choose reaches 90% at 8.1 steps. Jev does carry signal: it sends back 88% of first finishes that missed a requirement and 62% of those that met them all, which changes the steps per run by −1.7 (95% CI −2.1 to −1.3). It does not produce more finished work, with a gain over sending everything back of +0.5 (95% CI 0.0 to 1.5) points, because a general request to look again rarely fixes anything.
The independent check is different in kind. It is the grader itself, a test the agent never saw, so it sends back only work that missed a checked requirement. It is an oracle-style comparator for the checked requirements, which no real test suite would match, and it does not guarantee a repair after the send-back. Comparing its silent and named rows separates the value of detecting a failure from the value of saying what failed. Told only to re-check, the agent reaches 91%, and fixes the same 6 of 25 premature finishes as Jev's and the blanket send-backs do. Told which requirement failed, it reaches 94%, a gain of +6.5 (95% CI 2.5 to 11.5) points over accepting at 5.5 steps, and fixes 13. Naming the failure alone adds +3.5 (95% CI 1.0 to 6.5) points. Sending work back also carries a small risk: finished work later failed in 2 runs under sending everything back and 1 under Jev.
Against accepting, sending everything back gains +2.0 (95% CI −1.5 to 6.5) points and Jev gains +2.5 (95% CI −0.5 to 7.0); both ranges include zero. The silent check's range reaches zero, and only the named check's clearly excludes it.
An action changes what the system knows next. The useful part of a send-back is the evidence it carries, and here the agent could act on a specific failure and could do little with a general request to look again.
### Where the models do help
A stopping decision is not the only one a controller makes. When a tool reports a failure and says to try again, the right move is clear, and Jev picks retry 97% of the time. A missing input is harder. Across the 62 checkpoints with an input still withheld, Jev chooses to ask the user only 19% of the time. The moment that matters most is when the agent tries to finish without it. That happened 11 times, and Jev agreed with finishing in 11.
Table 4. The models handle a failed tool well and rarely ask for a missing input
What evidence would let you trust an agent's "done" without a person: a test it did not write, a second system that recomputes the answer, a history of being right on this kind of task? Which of those could you build for your own workflow?
Run the prediction before showing the table. Then ask why a send-back chosen by a model did about as well as sending everything back, and what a team would have to build to get the named check's result on its own work.
### What these data cannot show
The executed runs are real runs of one agent on tasks written for this booklet, so they show what happened in this test and not what would happen on real work; the tasks are few and intervals are wide. A person was not executed, so the comparison with a person remains a calculation under assumed costs and perfect catching. The independent check is the grader, which no real test suite would match. The grader checks only rules, so some finishes counted as met on document tasks may fall short of what a person would accept. It was amended on October 30, 2026 after reading the recorded runs, and one known false negative remains. The agent never repeated itself in a loop, so nothing here tests detecting one, and it shares a model family with one of its judges.
## Tables
### Table 1. The agent's "done" is usually right and wrong often enough to matter
Finish attempts by task kind and the grader's verdict, percent of attempts
| kind | share | verdict |
| --- | ---: | --- |
| Answer from records | 78.50 | Criteria met |
| Answer from records | 21.50 | Said done too early |
| Repair a program | 86.20 | Criteria met |
| Repair a program | 13.80 | Said done too early |
| Update a document | 97.10 | Criteria met |
| Update a document | 2.90 | Said done too early |
Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted
Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/finish-by-kind.json
### Table 2. Reading the trace tells the models something about whether the work is done, with wide intervals on this few tasks
AUC of each model's P(done) against the grader's verdict, with 95% intervals, finish attempts
| auc | conf_high | conf_low | kind | model | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 0.713 | 0.830 | 0.551 | Decision engine | Jev 1.14.0 | 200 |
| 0.760 | 0.862 | 0.624 | Decision engine | OpenAI Luna | 200 |
| 0.765 | 0.873 | 0.621 | General language model | Claude Haiku adapter | 200 |
Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted
Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/judging-done.json
### Table 3. At high cutoffs the models land near the cost of sending every finish to a person
Expected cost per 100 finish attempts by the cutoff for accepting, at assumed costs
| cost | cutoff | policy |
| ---: | ---: | --- |
| 525 | 0.500 | Jev 1.14.0 |
| 525 | 0.510 | Jev 1.14.0 |
| 502.5 | 0.520 | Jev 1.14.0 |
| 505 | 0.530 | Jev 1.14.0 |
| 460 | 0.540 | Jev 1.14.0 |
| 460 | 0.550 | Jev 1.14.0 |
| 442.5 | 0.560 | Jev 1.14.0 |
| 445 | 0.570 | Jev 1.14.0 |
| 445 | 0.580 | Jev 1.14.0 |
| 450 | 0.590 | Jev 1.14.0 |
| 452.5 | 0.600 | Jev 1.14.0 |
| 452.5 | 0.610 | Jev 1.14.0 |
| 455 | 0.620 | Jev 1.14.0 |
| 460 | 0.630 | Jev 1.14.0 |
| 462.5 | 0.640 | Jev 1.14.0 |
| 462.5 | 0.650 | Jev 1.14.0 |
| 462.5 | 0.660 | Jev 1.14.0 |
| 465 | 0.670 | Jev 1.14.0 |
| 450 | 0.680 | Jev 1.14.0 |
| 450 | 0.690 | Jev 1.14.0 |
| 430 | 0.700 | Jev 1.14.0 |
| 430 | 0.710 | Jev 1.14.0 |
| 432.5 | 0.720 | Jev 1.14.0 |
| 435 | 0.730 | Jev 1.14.0 |
| 415 | 0.740 | Jev 1.14.0 |
| 420 | 0.750 | Jev 1.14.0 |
| 425 | 0.760 | Jev 1.14.0 |
| 415 | 0.770 | Jev 1.14.0 |
| 422.5 | 0.780 | Jev 1.14.0 |
| 382.5 | 0.790 | Jev 1.14.0 |
| 395 | 0.800 | Jev 1.14.0 |
| 407.5 | 0.810 | Jev 1.14.0 |
| 422.5 | 0.820 | Jev 1.14.0 |
| 440 | 0.830 | Jev 1.14.0 |
| 420 | 0.840 | Jev 1.14.0 |
| 427.5 | 0.850 | Jev 1.14.0 |
| 442.5 | 0.860 | Jev 1.14.0 |
| 415 | 0.870 | Jev 1.14.0 |
| 382.5 | 0.880 | Jev 1.14.0 |
| 392.5 | 0.890 | Jev 1.14.0 |
| 412.5 | 0.900 | Jev 1.14.0 |
| 397.5 | 0.910 | Jev 1.14.0 |
| 415 | 0.920 | Jev 1.14.0 |
| 435 | 0.930 | Jev 1.14.0 |
| 462.5 | 0.940 | Jev 1.14.0 |
| 460 | 0.950 | Jev 1.14.0 |
| 467.5 | 0.960 | Jev 1.14.0 |
| 495 | 0.970 | Jev 1.14.0 |
| 500 | 0.980 | Jev 1.14.0 |
| 500 | 0.990 | Jev 1.14.0 |
| 502.5 | 0.500 | OpenAI Luna |
| 502.5 | 0.510 | OpenAI Luna |
| 502.5 | 0.520 | OpenAI Luna |
| 502.5 | 0.530 | OpenAI Luna |
| 505 | 0.540 | OpenAI Luna |
| 505 | 0.550 | OpenAI Luna |
| 505 | 0.560 | OpenAI Luna |
| 505 | 0.570 | OpenAI Luna |
| 505 | 0.580 | OpenAI Luna |
| 505 | 0.590 | OpenAI Luna |
| 505 | 0.600 | OpenAI Luna |
| 505 | 0.610 | OpenAI Luna |
| 507.5 | 0.620 | OpenAI Luna |
| 507.5 | 0.630 | OpenAI Luna |
| 510 | 0.640 | OpenAI Luna |
| 510 | 0.650 | OpenAI Luna |
| 510 | 0.660 | OpenAI Luna |
| 510 | 0.670 | OpenAI Luna |
| 490 | 0.680 | OpenAI Luna |
| 472.5 | 0.690 | OpenAI Luna |
| 472.5 | 0.700 | OpenAI Luna |
| 472.5 | 0.710 | OpenAI Luna |
| 472.5 | 0.720 | OpenAI Luna |
| 477.5 | 0.730 | OpenAI Luna |
| 477.5 | 0.740 | OpenAI Luna |
| 477.5 | 0.750 | OpenAI Luna |
| 482.5 | 0.760 | OpenAI Luna |
| 482.5 | 0.770 | OpenAI Luna |
| 460 | 0.780 | OpenAI Luna |
| 460 | 0.790 | OpenAI Luna |
| 437.5 | 0.800 | OpenAI Luna |
| 437.5 | 0.810 | OpenAI Luna |
| 417.5 | 0.820 | OpenAI Luna |
| 420 | 0.830 | OpenAI Luna |
| 422.5 | 0.840 | OpenAI Luna |
| 425 | 0.850 | OpenAI Luna |
| 427.5 | 0.860 | OpenAI Luna |
| 410 | 0.870 | OpenAI Luna |
| 395 | 0.880 | OpenAI Luna |
| 400 | 0.890 | OpenAI Luna |
| 405 | 0.900 | OpenAI Luna |
| 407.5 | 0.910 | OpenAI Luna |
| 365 | 0.920 | OpenAI Luna |
| 372.5 | 0.930 | OpenAI Luna |
| 360 | 0.940 | OpenAI Luna |
| 310 | 0.950 | OpenAI Luna |
| 315 | 0.960 | OpenAI Luna |
| 327.5 | 0.970 | OpenAI Luna |
| 337.5 | 0.980 | OpenAI Luna |
| 357.5 | 0.990 | OpenAI Luna |
| 412.5 | 0.500 | Claude Haiku adapter |
| 412.5 | 0.510 | Claude Haiku adapter |
| 412.5 | 0.520 | Claude Haiku adapter |
| 412.5 | 0.530 | Claude Haiku adapter |
| 412.5 | 0.540 | Claude Haiku adapter |
| 415 | 0.550 | Claude Haiku adapter |
| 415 | 0.560 | Claude Haiku adapter |
| 415 | 0.570 | Claude Haiku adapter |
| 415 | 0.580 | Claude Haiku adapter |
| 415 | 0.590 | Claude Haiku adapter |
| 395 | 0.600 | Claude Haiku adapter |
| 395 | 0.610 | Claude Haiku adapter |
| 395 | 0.620 | Claude Haiku adapter |
| 395 | 0.630 | Claude Haiku adapter |
| 395 | 0.640 | Claude Haiku adapter |
| 395 | 0.650 | Claude Haiku adapter |
| 395 | 0.660 | Claude Haiku adapter |
| 395 | 0.670 | Claude Haiku adapter |
| 395 | 0.680 | Claude Haiku adapter |
| 395 | 0.690 | Claude Haiku adapter |
| 372.5 | 0.700 | Claude Haiku adapter |
| 372.5 | 0.710 | Claude Haiku adapter |
| 372.5 | 0.720 | Claude Haiku adapter |
| 372.5 | 0.730 | Claude Haiku adapter |
| 372.5 | 0.740 | Claude Haiku adapter |
| 372.5 | 0.750 | Claude Haiku adapter |
| 372.5 | 0.760 | Claude Haiku adapter |
| 372.5 | 0.770 | Claude Haiku adapter |
| 372.5 | 0.780 | Claude Haiku adapter |
| 372.5 | 0.790 | Claude Haiku adapter |
| 300 | 0.800 | Claude Haiku adapter |
| 300 | 0.810 | Claude Haiku adapter |
| 302.5 | 0.820 | Claude Haiku adapter |
| 302.5 | 0.830 | Claude Haiku adapter |
| 302.5 | 0.840 | Claude Haiku adapter |
| 362.5 | 0.850 | Claude Haiku adapter |
| 362.5 | 0.860 | Claude Haiku adapter |
| 362.5 | 0.870 | Claude Haiku adapter |
| 362.5 | 0.880 | Claude Haiku adapter |
| 362.5 | 0.890 | Claude Haiku adapter |
| 387.5 | 0.900 | Claude Haiku adapter |
| 387.5 | 0.910 | Claude Haiku adapter |
| 392.5 | 0.920 | Claude Haiku adapter |
| 495 | 0.930 | Claude Haiku adapter |
| 495 | 0.940 | Claude Haiku adapter |
| 487.5 | 0.950 | Claude Haiku adapter |
| 492.5 | 0.960 | Claude Haiku adapter |
| 500 | 0.970 | Claude Haiku adapter |
| 500 | 0.980 | Claude Haiku adapter |
| 500 | 0.990 | Claude Haiku adapter |
| 625 | 0.500 | Accept every finish |
| 500 | 0.500 | Send every finish to a person |
| 625 | 0.510 | Accept every finish |
| 500 | 0.510 | Send every finish to a person |
| 625 | 0.520 | Accept every finish |
| 500 | 0.520 | Send every finish to a person |
| 625 | 0.530 | Accept every finish |
| 500 | 0.530 | Send every finish to a person |
| 625 | 0.540 | Accept every finish |
| 500 | 0.540 | Send every finish to a person |
| 625 | 0.550 | Accept every finish |
| 500 | 0.550 | Send every finish to a person |
| 625 | 0.560 | Accept every finish |
| 500 | 0.560 | Send every finish to a person |
| 625 | 0.570 | Accept every finish |
| 500 | 0.570 | Send every finish to a person |
| 625 | 0.580 | Accept every finish |
| 500 | 0.580 | Send every finish to a person |
| 625 | 0.590 | Accept every finish |
| 500 | 0.590 | Send every finish to a person |
| 625 | 0.600 | Accept every finish |
| 500 | 0.600 | Send every finish to a person |
| 625 | 0.610 | Accept every finish |
| 500 | 0.610 | Send every finish to a person |
| 625 | 0.620 | Accept every finish |
| 500 | 0.620 | Send every finish to a person |
| 625 | 0.630 | Accept every finish |
| 500 | 0.630 | Send every finish to a person |
| 625 | 0.640 | Accept every finish |
| 500 | 0.640 | Send every finish to a person |
| 625 | 0.650 | Accept every finish |
| 500 | 0.650 | Send every finish to a person |
| 625 | 0.660 | Accept every finish |
| 500 | 0.660 | Send every finish to a person |
| 625 | 0.670 | Accept every finish |
| 500 | 0.670 | Send every finish to a person |
| 625 | 0.680 | Accept every finish |
| 500 | 0.680 | Send every finish to a person |
| 625 | 0.690 | Accept every finish |
| 500 | 0.690 | Send every finish to a person |
| 625 | 0.700 | Accept every finish |
| 500 | 0.700 | Send every finish to a person |
| 625 | 0.710 | Accept every finish |
| 500 | 0.710 | Send every finish to a person |
| 625 | 0.720 | Accept every finish |
| 500 | 0.720 | Send every finish to a person |
| 625 | 0.730 | Accept every finish |
| 500 | 0.730 | Send every finish to a person |
| 625 | 0.740 | Accept every finish |
| 500 | 0.740 | Send every finish to a person |
| 625 | 0.750 | Accept every finish |
| 500 | 0.750 | Send every finish to a person |
| 625 | 0.760 | Accept every finish |
| 500 | 0.760 | Send every finish to a person |
| 625 | 0.770 | Accept every finish |
| 500 | 0.770 | Send every finish to a person |
| 625 | 0.780 | Accept every finish |
| 500 | 0.780 | Send every finish to a person |
| 625 | 0.790 | Accept every finish |
| 500 | 0.790 | Send every finish to a person |
| 625 | 0.800 | Accept every finish |
| 500 | 0.800 | Send every finish to a person |
| 625 | 0.810 | Accept every finish |
| 500 | 0.810 | Send every finish to a person |
| 625 | 0.820 | Accept every finish |
| 500 | 0.820 | Send every finish to a person |
| 625 | 0.830 | Accept every finish |
| 500 | 0.830 | Send every finish to a person |
| 625 | 0.840 | Accept every finish |
| 500 | 0.840 | Send every finish to a person |
| 625 | 0.850 | Accept every finish |
| 500 | 0.850 | Send every finish to a person |
| 625 | 0.860 | Accept every finish |
| 500 | 0.860 | Send every finish to a person |
| 625 | 0.870 | Accept every finish |
| 500 | 0.870 | Send every finish to a person |
| 625 | 0.880 | Accept every finish |
| 500 | 0.880 | Send every finish to a person |
| 625 | 0.890 | Accept every finish |
| 500 | 0.890 | Send every finish to a person |
| 625 | 0.900 | Accept every finish |
| 500 | 0.900 | Send every finish to a person |
| 625 | 0.910 | Accept every finish |
| 500 | 0.910 | Send every finish to a person |
| 625 | 0.920 | Accept every finish |
| 500 | 0.920 | Send every finish to a person |
| 625 | 0.930 | Accept every finish |
| 500 | 0.930 | Send every finish to a person |
| 625 | 0.940 | Accept every finish |
| 500 | 0.940 | Send every finish to a person |
| 625 | 0.950 | Accept every finish |
| 500 | 0.950 | Send every finish to a person |
| 625 | 0.960 | Accept every finish |
| 500 | 0.960 | Send every finish to a person |
| 625 | 0.970 | Accept every finish |
| 500 | 0.970 | Send every finish to a person |
| 625 | 0.980 | Accept every finish |
| 500 | 0.980 | Send every finish to a person |
| 625 | 0.990 | Accept every finish |
| 500 | 0.990 | Send every finish to a person |
Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted
Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/cost-by-cutoff.json
### Table 4. The models handle a failed tool well and rarely ask for a missing input
Share of checkpoints where each model's next action fits the situation, percent
| model | share | situation |
| --- | ---: | --- |
| Jev 1.14.0 | 97.40 | A tool failed and said to try again: chose retry |
| Jev 1.14.0 | 80.60 | A required input is still missing: did not finish |
| Jev 1.14.0 | 19.40 | A required input is still missing: asked the user |
| OpenAI Luna | 92.10 | A tool failed and said to try again: chose retry |
| OpenAI Luna | 77.40 | A required input is still missing: did not finish |
| OpenAI Luna | 12.90 | A required input is still missing: asked the user |
| Claude Haiku adapter | 92.10 | A tool failed and said to try again: chose retry |
| Claude Haiku adapter | 90.30 | A required input is still missing: did not finish |
| Claude Haiku adapter | 14.50 | A required input is still missing: asked the user |
Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted
Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/handling-faults.json
## All estimates
Every fact in facts.json (63), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| accept_cutoff | 0.9 | probability that the work is done | 1 minus person cost over premature cost | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed agent tasks | |
| ask_jev | 19.4% | percent of checkpoints | next action is ask the user while an input is withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 | | 62 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| auc_adapter | 0.76 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.62 to 0.87 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| auc_jev | 0.71 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.55 to 0.83 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| auc_jev_no_rubric | 0.72 | area under the ROC curve | Jev, leaving those finishes out | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| auc_jev_rubric_unmet | 0.56 | area under the ROC curve | Jev, counting those finishes as unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| caught_checks | 0.0% | percent of premature finishes | premature finishes sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| caught_jev | 84.0% | percent of premature finishes | premature finishes sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| cost_accept | $625 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $350 to $951 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_adapter | $388 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $285 to $497 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_checks | $625 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $350 to $951 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_diff_adapter | −$113 | US dollars per 100 finish attempts | adapter minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$215 to −$3 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| cost_diff_checks | +$125 | US dollars per 100 finish attempts | checks minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$150 to $451 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| cost_diff_jev | −$88 | US dollars per 100 finish attempts | jev minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$173 to $0 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| cost_jev | $413 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $327 to $500 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_person | $500 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| diff_met_check_named_accept | +6.5 | percentage points | check_named minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 2.5 to 11.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_met_check_named_check_silent | +3.5 | percentage points | check_named minus check_silent | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 1.0 to 6.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_met_check_silent_accept | +3.0 | percentage points | check_silent minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 0.0 to 7.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_met_jev_accept | +2.5 | percentage points | jev minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −0.5 to 7.0 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_met_jev_recheck | +0.5 | percentage points | jev minus recheck | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 0.0 to 1.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_met_recheck_accept | +2.0 | percentage points | recheck minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −1.5 to 6.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| diff_steps_jev_recheck | −1.7 | steps per run | jev minus recheck | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −2.1 to −1.3 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| jev_sent_done | 61.7% | percent of first finishes | Jev sends back a first finish that met every rule | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | 175 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| jev_sent_premature | 88.0% | percent of first finishes | Jev sends back a first finish that missed a rule | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | 25 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| max_sendbacks | two | send-backs per run | limit in every controller | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| met_accept | 87.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| met_check_named | 94.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| met_check_silent | 90.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 86.0% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| met_jev | 90.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| met_recheck | 89.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.0% to 94.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_finish | 200 | finish attempts | times the agent said it was done | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_premature | 25 | finish attempts | agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_premature_exec | 25 | runs | first finish missed a rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_rubric | 34 | finish attempts | counted as met with a rater-only criterion unscored | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_tasks | 40 | tasks | constructed tasks | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_withheld | 62 | checkpoints | a required input is still withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| n_withheld_finish | 11 | finish attempts | the agent says it is done while a required input is withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| nextaction_finish_premature_jev | 96.0% | percent of premature finishes | Jev's next action is still finish | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | 25 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| person_cost | $5 | US dollars per finish attempt sent to a person | assumed: five minutes at a dollar a minute | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed agent tasks | |
| premature_cost | $50 | US dollars per premature finish accepted | assumed rework cost | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed agent tasks | |
| regress_accept | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| regress_check_named | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| regress_check_silent | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| regress_jev | 1 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| regress_recheck | 2 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| rescued_accept | 0 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| rescued_check_named | 13 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| rescued_check_silent | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| rescued_jev | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| rescued_recheck | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| retry_jev | 97.4% | percent of checkpoints | next action is retry after a transient failure | 38 checkpoints after an injected tool failure | agent runs and engine calls, October 2026 | | 38 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| sent_checks | 0.0% | percent of finished work | finished work sent to a person anyway | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| sent_jev | 59.4% | percent of finished work | finished work sent to a person anyway | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| share_premature | 12.5% | percent of finish attempts | agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| share_premature_other | 8.1% | percent of finish attempts | premature, documents and program repair | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| share_premature_records | 21.5% | percent of finish attempts | premature, record lookups | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| steps_accept | 5.0 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 4.6 to 5.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| steps_check_named | 5.5 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 4.9 to 6.1 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| steps_check_silent | 5.5 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 5.0 to 6.2 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| steps_jev | 8.1 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 7.0 to 9.1 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| steps_recheck | 9.7 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 8.8 to 10.7 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
| withheld_finish_jev | 11 | finish attempts | Jev's next action is finish | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 | | | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | |
## Definitions
The chapter marks no glossary term.
## Methods
One agent worked through tasks written for this booklet while the harness injected faults. Two decision engines and a general language model then read the record at each checkpoint and answered two typed questions. A grader with rules written before any run supplies the verdict.
### The agent and the tasks
The tasks are short: update a document, answer from records, repair a small program. Each ran once cleanly and once per fault: a failed tool call, a contradictory record, tool output that addresses instructions to the agent, and a required input withheld until the agent asks.
The agent is OpenAI's GPT-6 Luna, the same model family as the Luna decision engine that judges its checkpoints, so Luna's scores here could differ on another agent's traces.
### Where the verdicts come from
A finish attempt counts as premature when any rule-checked criterion is unmet at that step. Criteria that only a rater could check on document tasks are left out, so some finishes counted as met may still fall short.
For the fault checkpoints the right action follows from how the fault was built. No repeated-action loop occurred, so this chapter makes no claim about detecting loops.
### Scoring and intervals
The cost comparison scores each controller on the same recorded finish attempts. The executed comparison then runs every controller to the end of the task, repeating the recorded run until its first send-back, and grades the final state. Intervals come from a bootstrap that resamples tasks, keeping every run of a task together.
## What these data cannot show
The executed runs are real runs of one agent on tasks written for this booklet, so they show what happened in this test and not what would happen on real work; the tasks are few and intervals are wide. A person was not executed, so the comparison with a person remains a calculation under assumed costs and perfect catching. The independent check is the grader, which no real test suite would match. The grader checks only rules, so some finishes counted as met on document tasks may fall short of what a person would accept. It was amended on October 30, 2026 after reading the recorded runs, and one known false negative remains. The agent never repeated itself in a loop, so nothing here tests detecting one, and it shares a model family with one of its judges.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/deciding-when-an-agent-is-finished/build.py (sha256 81973105bd0fc1c9838b0022e4ea02c7923af012f16095a4548470efe3f5a6be)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 2f1ad12ffd4a1c771761c48355d0a63c7a692e8c4267ba3f6e5450434d7d6231 |
| figures/cost-by-cutoff.json | 02e55c84bd3559f1903d3873ad5d64584bf974d88d78e10e876fef7e237d6467 |
| figures/finish-by-kind.json | 7231d602976bbc0d1f1c2014b7a43219e8b29835d9e9b8ec111706d1f880b5d9 |
| figures/handling-faults.json | 645ff1496889ca19992d40c4f701fd240e950556220b7af0b1b92be312b02af1 |
| figures/judging-done.json | 8ffbbf124909b48910350cd376c0c4f1715e2336529245cb021da73fd72d69fb |
| meta.json | ebd35c8478214420d4b0a1bd57760bdc18c9ea2f1521fdac72aba445ac8f5b87 |
| public/data/deciding-when-an-agent-is-finished/executed.json | 80929511e1de850d4ff10f0d84026e6f03d1bf88005e6229956639660e31451c |
| public/data/deciding-when-an-agent-is-finished/policy.json | ee8fb54577ee509d81bab2853945295bde0e25f8bf8510c626e7db64fb05ce8c |
| public/data/deciding-when-an-agent-is-finished/reveal.json | 5400af5bcc3818f0e8e76caca38d0935ec386690c645b2fa76f84f28b1699516 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report
- Report (Markdown): https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/facts.json
- Table 1, figure "finish-by-kind" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/finish-by-kind.json
- Table 2, figure "judging-done" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/judging-done.json
- Table 3, figure "cost-by-cutoff" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/cost-by-cutoff.json
- Table 4, figure "handling-faults" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/handling-faults.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt
---
# Comparing decision systems: report
Chapter 4.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems. Table n of this report is figure n of the chapter.
## Identification
- Title: Comparing decision systems
- Chapter: 4.1 (Part IV, Deciding whether to adopt; Case studies)
- Question: Across the cases, when should a team write a rule, train a classifier, ask a decision endpoint or keep a person, and what should it ask before adopting a system?
- Dek: Across reviews, complaints and agent work, the choice between an endpoint and a trained classifier turns on how many labels a task needs and whether the probabilities are calibrated, and an agent's work improved most when a check it could not influence named what had failed.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
- Amazon Reviews 2023. Results reused from the two review chapters; weights: sample_weight restores the star mix
- CFPB Consumer Complaint Database. Results reused from the complaint chapter; weights: sample_weight restores the product mix
- SWE-rebench. Results reused from the code-review chapter; weights: sample_weight restores the outcome mix
- Constructed patches. Results reused from the code-review chapter; weights: unweighted; evenly split by construction
- Constructed agent tasks. Results reused from the agent chapter; weights: unweighted; the mix describes the test harness
- Canonical URL: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems
- Report: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report
- Markdown: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md
- BibTeX key: singh2026comparingandauditingdecisionsystems
```bibtex
@misc{singh2026comparingandauditingdecisionsystems,
author = {Singh, Vishal},
title = {Comparing decision systems},
year = {2026},
url = {https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems},
note = {In Decision AI. Published 30 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $4,466 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_net_1000 |
| mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,047 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_net_20000 |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,076 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev |
| mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 58.8% | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | lc_capture_1000 |
| share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 58.2% | 95% CI 53.0% to 63.4% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | capture_jev |
| share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 73.1% | 95% CI 68.4% to 77.9% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | capture_tfidf |
## Findings
Trained on a thousand labeled reviews, a classifier's contact policy earns $4,500 per thousand reviews at the assumed costs, and it takes the full labeled sample to reach $5,000, about what raw Jev earns with no labels ($5,100). On money back the order reverses: a classifier with about a thousand past complaints catches 59% of those cases in its top tenth, level with Jev's 58%, and with the full sample it reaches 73%.
So how much local data a task needs differs widely, and it is an important practical difference between an endpoint and a classifier. A good ranking is not enough when the action rule takes probabilities at face value, and a score on a tidy test set says little about real work. These results come from a handful of cases, with stand-in labels and assumed costs, so treat them as questions to ask of your own data.
### What local labels change
On reading a review, the endpoint needs no training and the classifier needs a large labeled sample to catch up. On predicting what a company will do, the classifier learns from records a desk already holds and passes the endpoint with a modest sample. The reason is a matter of interpretation: reading can draw on what a language model already knows, while a company's habits sit in its own records. These two cases are consistent with that reading and cannot test it.
When both uses get the same labels, the comparison sharpens. On the contact decision, two hundred labels give $5,300 spent recalibrating Jev and $3,300 spent fitting a classifier. On complaint routing, five hundred labels give expected costs of $3,600 and $4,800. So for reading tasks, a modest budget of labels went further checking an endpoint, in these two tests and with this one classifier. For forecasting what a company will do, it went further training on outcomes.
### A good ranking is not enough when the action rule takes probabilities at face value
The next chart puts each priced decision on one scale, comparing Jev's policy with the simple alternative in that case.
Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities
In the review case, costs alone put the contact cutoff at 0.125, while the data put Jev's at 0.38 because its probabilities run high. Raw Jev earns $5,100 and the classifier $5,000, with no clear gap; recalibrated Jev earns $5,300. On complaints the raw comparison favors the classifier, and with both recalibrated the gap is −$195 (95% CI −$419 to $27) dollars per thousand. So a firm that applies a cost cutoff to an endpoint's raw probabilities may get less than its ranking score suggests, and a modest labeled sample is worth budgeting even if it never trains a model. A policy that only fills a fixed budget from the top of a ranking leans less on the probabilities themselves.
Call charges did not decide any of this. Jev costs about $0.05 per thousand complaints, against expected handling costs of $3,471 per thousand in the same assumed scenario. A full comparison would also count integration, checking and upkeep, which these figures leave out.
### A tidy test set says little about real work
Table 2. Jev's ranking score differs widely from one question to the next
Jev scores 0.79 on patches written with a visible flaw and 0.60 on real agent patches, where its lead over sorting by size is +0.01 (95% CI −0.03 to 0.05). Money back shows a related trap: a high score can come from something other than reading. Ranking complaints by their product's past rate alone gives 0.906, and within a product Jev scores 0.742 and the classifier 0.769.
### Evidence that names the problem beats a request to look again
An agent's first finish met every checked requirement in 88% of runs. Sending work back on Jev's judgment reached 90% and sending everything back 90%. A check the agent could not influence, naming the failed requirement, reached 94%. That check is the grader itself, an oracle-style comparator for the checked requirements, and the tasks were written for this booklet. The five automated policies were run; a person was not. The pattern matches what the same chapter found on simpler ground: after a tool reports a failure, a model chooses to retry 97% of the time. When the evidence says what happened, the next step is easy.
### Where to start
(Interactive guide: yes-or-no questions lead to a starting recommendation. The same routes are in the table that follows.)
The guide follows the routes in this table, each drawn from one case.
| Your situation | Where to start | Where it comes from |
| --- | --- | --- |
| An exact test can check the answer | The rule or the test, and report what failed | Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) |
| Predicting a later outcome, with many past examples | Evaluate a classifier trained on your outcomes | Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) |
| Predicting a later outcome, with few past examples | Record outcomes first; use an endpoint only as a stopgap | Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) |
| A reader could not tell from the evidence whether the answer is right | A person, or a check the system cannot influence | Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) |
| Reading the text, with thousands of labels of your own | Compare an endpoint and a classifier as complete policies | Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) |
| Reading the text, with few labels | Try an endpoint, and spend a few hundred labels calibrating it before acting | Chapter 1.2, From evidence to typed questions (https://vishalsingh.org/decisionai/from-evidence-to-typed-questions) |
These are starting points drawn from a few cases and not rules. A team's own costs and records can reverse any of them.
### Would you deploy this? The complaint desk
Take the desk's routing decision and put the evidence in one place. The choices are to deploy, run a limited pilot, measure further or keep the current process.
| Evidence | What the case shows | What is still missing |
| --- | --- | --- |
| Judgment | Agreement with consumers' choices of 82% for Jev and 82% for the classifier | How often the consumer's choice is itself wrong |
| Probabilities | Neither model's stated probabilities can be taken at face value | Calibration on the desk's own complaints |
| Policy | Recalibrated, $3,471 for Jev and $3,666 for the classifier per thousand complaints, with no clear gap | The desk's real costs of delay and triage |
| Attention | Ranking review by expected loss per minute changes cost by −425 (95% CI −639 to −198) dollars per thousand, under assumed review times | Measured handling time, and reviewer accuracy on short complaints |
| Failures | Largest disagreements run between credit reporting and debt collection | What a misroute costs in each direction |
| Observed effect | None; no policy here ran on a desk | A randomized pilot |
Given only this evidence, what would you recommend to the desk? Choose before reading on.
This evidence does not justify full deployment. It justifies a local evaluation of whichever model the desk can run most cheaply, with its probabilities recalibrated on a labeled sample of the desk's own complaints, because neither shows a clear lead. Start in shadow, recording the model's picks while the current process keeps routing. Changing live routing also needs a measured baseline, a limited set of complaints and a way back. For the money-back question, closed-case records make a classifier a strong baseline to evaluate, once the desk has checked that outcomes arrive in time, that old records resemble today's complaints and that money back is what it wants to prioritize.
Adoption also changes what the desk can see. Misroutes that bounce back get recorded, and ones that get handled without complaint do not, so a small random sample of automatically routed complaints, reviewed on a regular schedule, is what would show the policy going stale.
### Questions to ask before adopting a system
Each case suggests a question for a vendor or an internal team.
| Question | Why it matters here |
| --- | --- |
| Was it tested on work like yours, or on a tidy test set? | The same engine scored far lower on real code changes than on written ones |
| What is the simplest alternative, and has it been run on the same records? | A classifier or sorting by size was hard to beat, and the classifier's result depended on how it was fit |
| What does the label actually measure? | Stars, consumer choices and company responses are all stand-ins |
| Do the probabilities mean what they say, near the cutoff you will use? | Recalibrating changed the answer in two of the priced decisions |
| What happens to the answers if the question changes? | Rewording one question changed 2.5% of Jev's answers |
| Who owns the question, the costs and the cutoff, and what triggers a recheck? | The engine, the costs and the population can each change on their own |
| Has anyone run this policy, or only scored it? | Only the agent case ran its policies, on tasks written for this booklet |
Pick one decision your organization might hand to a model. Walk it through the guide. Where does it land, how many labels would it take to find out, and what would you need to measure before you trusted that answer?
Ask groups to choose a decision, walk it through the guide and defend the result against another group playing the vendor. Then ask each group to name the cheapest experiment that would change its mind.
### What these data cannot show
This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.
## Tables
### Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities
Jev's policy against the simple alternative, percent of that alternative's value or cost, with 95% intervals
| conf_high | conf_low | decision | gap |
| ---: | ---: | --- | ---: |
| 3.20 | −0.900 | Contacting reviewers, raw probabilities (against a classifier) | 1.20 |
| 8.20 | 4.50 | Contacting reviewers, Jev recalibrated (against a classifier) | 6.40 |
| −2.60 | −17.80 | Routing complaints, raw probabilities (against a classifier) | −10 |
| 11.40 | −0.700 | Routing complaints, both recalibrated (against a classifier) | 5.30 |
| 34.60 | 0 | Accepting an agent's finish (against a person checking all) | 17.50 |
Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed
Data: https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json
### Table 2. Jev's ranking score differs widely from one question to the next
AUC by question, Jev and the trained classifier, with 95% intervals; no classifier was trained for the patch and agent cases
| auc | conf_high | conf_low | model | task |
| ---: | ---: | ---: | --- | --- |
| 0.980 | 0.982 | 0.978 | Jev | Is this reviewer unhappy? |
| 0.968 | 0.972 | 0.964 | Trained classifier | Is this reviewer unhappy? |
| 0.742 | 0.770 | 0.709 | Jev | Will this complaint end in money back, within a product? |
| 0.769 | 0.802 | 0.736 | Trained classifier | Will this complaint end in money back, within a product? |
| 0.785 | 0.890 | 0.690 | Jev | Would this written patch fail its hidden test? |
| 0.604 | 0.640 | 0.560 | Jev | Did this real agent patch fail its benchmark? |
| 0.713 | 0.830 | 0.550 | Jev | Is the agent's work really done? |
Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed
Data: https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json
## All estimates
Every fact in facts.json (38), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| agree_jev | 81.9% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| agree_tfidf | 82.4% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_across_diff_jev_size | +0.01 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.03 to 0.05 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | From allocating-review-attention-to-code-changes. |
| auc_done_jev | 0.71 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.55 to 0.83 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| auc_money_jev | 0.895 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.881 to 0.908 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_money_product_only | 0.906 | area under the ROC curve | ranking complaints by their product's relief rate in the training years | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_money_tfidf | 0.936 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.928 to 0.945 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_patch_real | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.56 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | From allocating-review-attention-to-code-changes. |
| auc_patch_written | 0.79 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.69 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | From allocating-review-attention-to-code-changes. |
| auc_reviews_jev | 0.980 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| auc_within_jev | 0.742 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.709 to 0.770 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_within_tfidf | 0.769 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.736 to 0.802 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| capture_jev | 58.2% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 53.0% to 63.4% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| capture_tfidf | 73.1% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 68.4% to 77.9% | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| cost_complaints | $0.05 | US dollars per 1,000 complaints | mean billed cost per call, list price | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| cut_data | 0.38 | probability of dissatisfaction | median of five cross-fit cost-minimizing cutoffs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Range across folds 0.35 to 0.38. |
| cut_theory | 0.125 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Amazon Reviews 2023 | From from-probabilities-to-actions. |
| lc_capture_1000 | 58.8% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| lc_net_1000 | $4,466 | US dollars per 1,000 reviews | mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| lc_net_20000 | $5,047 | US dollars per 1,000 reviews | mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| loss_complaints | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| loss_diff_jev_cal | −$195 | US dollars per 1,000 complaints | jev_cal minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −$419 to $27 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| loss_jev_cal | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cal | $3,666 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,466 to $3,885 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| lv_clf_200 | $3,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws. |
| lv_cost_clf_500 | $4,792 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws. |
| lv_cost_jev_500 | $3,613 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 | | | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws. |
| lv_jev_200 | $5,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws. |
| met_accept | 87.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_check_named | 94.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_jev | 90.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_recheck | 89.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.0% to 94.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| net_jev | $5,076 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| net_jev_recal | $5,335 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| net_tfidf | $5,015 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| queue_diff_value_per_minute_cost | −425 | per 1,000 complaints | value_per_minute minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −639 to −198 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| retry_jev | 97.4% | percent of checkpoints | next action is retry after a transient failure | 38 checkpoints after an injected tool failure | agent runs and engine calls, October 2026 | | 38 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| wording_change | 2.5% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 1.4% to 3.9% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
## Definitions
The chapter marks no glossary term.
## Methods
This chapter runs nothing new. Its numbers are copied by script from the other chapters' data files.
### Putting the priced decisions on one scale
The three decisions use different assumed costs, so each gap and its interval is divided by the simple alternative's dollar figure in the same case, with the sign set so that right means better for Jev. The denominators are treated as fixed, so these ranges are approximate. The percent scale helps compare direction and spread; it does not make the three assumed cost sets comparable.
## What these data cannot show
This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/comparing-and-auditing-decision-systems/build.py (sha256 756143a43f0dee2e10eb7f037f68ef8dd2e978818bf3dda7f2366edd2325f058)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 2cfb55a8f2401a7aa72efc6a2b6d1640fa7810ec3fd37190c7dbc4f286c114cc |
| figures/policy-gaps.json | 03bcbc36dd93675885cb097ae587331a5f2b307df3898b86bebb39006076a018 |
| figures/same-model-three-questions.json | 0afa4141d1b4fb93f0fc5ddfe99aeace040d31f3dbcda3a53ed87a31a4babc34 |
| meta.json | 1d4dc1729ddbe586dc4909fd14343f12f3319e01cca1b54b0c52cd4824847050 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report
- Report (Markdown): https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/facts.json
- Table 1, figure "policy-gaps" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json
- Table 2, figure "same-model-three-questions" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt