Comparing decision systems: report

Identification

Title

Comparing decision systems

Chapter

4.1 (Part IV, Deciding whether to adopt; Case studies)

Question

Across the cases, when should a team write a rule, train a classifier, ask a decision endpoint or keep a person, and what should it ask before adopting a system?

Dek

Across reviews, complaints and agent work, the choice between an endpoint and a trained classifier turns on how many labels a task needs and whether the probabilities are calibrated, and an agent's work improved most when a check it could not influence named what had failed.

Grain

records

Status

draft

Published

30 October 2026

Updated

not revised

Data sources

Amazon Reviews 2023. Results reused from the two review chapters; weights: sample_weight restores the star mix
CFPB Consumer Complaint Database. Results reused from the complaint chapter; weights: sample_weight restores the product mix
SWE-rebench. Results reused from the code-review chapter; weights: sample_weight restores the outcome mix
Constructed patches. Results reused from the code-review chapter; weights: unweighted; evenly split by construction
Constructed agent tasks. Results reused from the agent chapter; weights: unweighted; the mix describes the test harness

Canonical URL

https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems

Markdown

https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md

BibTeX

@misc{singh2026comparingandauditingdecisionsystems,
  author = {Singh, Vishal},
  title = {Comparing decision systems},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems},
  note = {In Decision AI. Published 30 October 2026.}
}

Principal estimates

The facts the dek and the opening prose cite, in order of appearance. Every fact is in the table of all estimates below.

Estimand	Population	Period	Value	Interval	n	Weights	Source	Fact id
mean net value over refits on 1,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$4,466		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	lc_net_1000
mean net value over refits on 20,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$5,047		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	lc_net_20000
net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$5,076	95% CI $5,004 to $5,144	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	net_jev
mean top-tenth capture over refits on 1,000 labeled complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	58.8%		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	lc_capture_1000
share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	58.2%	95% CI 53.0% to 63.4%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	capture_jev
share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	73.1%	95% CI 68.4% to 77.9%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	capture_tfidf

Findings

Trained on a thousand labeled reviews, a classifier's contact policy earns $4,500 per thousand reviews at the assumed costs, and it takes the full labeled sample to reach $5,000, about what raw Jev earns with no labels ($5,100). On money back the order reverses: a classifier with about a thousand past complaints catches 59% of those cases in its top tenth, level with Jev's 58%, and with the full sample it reaches 73%.

So how much local data a task needs differs widely, and it is an important practical difference between an endpoint and a classifier. A good ranking is not enough when the action rule takes probabilities at face value, and a score on a tidy test set says little about real work. These results come from a handful of cases, with stand-in labels and assumed costs, so treat them as questions to ask of your own data.

What local labels change

On reading a review, the endpoint needs no training and the classifier needs a large labeled sample to catch up. On predicting what a company will do, the classifier learns from records a desk already holds and passes the endpoint with a modest sample. The reason is a matter of interpretation: reading can draw on what a language model already knows, while a company's habits sit in its own records. These two cases are consistent with that reading and cannot test it.

When both uses get the same labels, the comparison sharpens. On the contact decision, two hundred labels give $5,300 spent recalibrating Jev and $3,300 spent fitting a classifier. On complaint routing, five hundred labels give expected costs of $3,600 and $4,800. So for reading tasks, a modest budget of labels went further checking an endpoint, in these two tests and with this one classifier. For forecasting what a company will do, it went further training on outcomes.

A good ranking is not enough when the action rule takes probabilities at face value

The next chart puts each priced decision on one scale, comparing Jev's policy with the simple alternative in that case.

Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities

In the review case, costs alone put the contact cutoff at 0.125, while the data put Jev's at 0.38 because its probabilities run high. Raw Jev earns $5,100 and the classifier $5,000, with no clear gap; recalibrated Jev earns $5,300. On complaints the raw comparison favors the classifier, and with both recalibrated the gap is −$195 (95% CI −$419 to $27) dollars per thousand. So a firm that applies a cost cutoff to an endpoint's raw probabilities may get less than its ranking score suggests, and a modest labeled sample is worth budgeting even if it never trains a model. A policy that only fills a fixed budget from the top of a ranking leans less on the probabilities themselves.

Call charges did not decide any of this. Jev costs about $0.05 per thousand complaints, against expected handling costs of $3,471 per thousand in the same assumed scenario. A full comparison would also count integration, checking and upkeep, which these figures leave out.

A tidy test set says little about real work

Table 2. Jev's ranking score differs widely from one question to the next

Jev scores 0.79 on patches written with a visible flaw and 0.60 on real agent patches, where its lead over sorting by size is +0.01 (95% CI −0.03 to 0.05). Money back shows a related trap: a high score can come from something other than reading. Ranking complaints by their product's past rate alone gives 0.906, and within a product Jev scores 0.742 and the classifier 0.769.

Evidence that names the problem beats a request to look again

An agent's first finish met every checked requirement in 88% of runs. Sending work back on Jev's judgment reached 90% and sending everything back 90%. A check the agent could not influence, naming the failed requirement, reached 94%. That check is the grader itself, an oracle-style comparator for the checked requirements, and the tasks were written for this booklet. The five automated policies were run; a person was not. The pattern matches what the same chapter found on simpler ground: after a tool reports a failure, a model chooses to retry 97% of the time. When the evidence says what happened, the next step is easy.

Where to start

Decision guide

Where should this decision start?

Can you write an exact test for the right answer?

Do the required tests pass? Did the tool report an error?

The guide follows the routes in this table, each drawn from one case.

Your situation	Where to start	Where it comes from
An exact test can check the answer	The rule or the test, and report what failed	Chapter 3.1, Deciding when an agent is finished
Predicting a later outcome, with many past examples	Evaluate a classifier trained on your outcomes	Chapter 2.1, Resolving and routing customer complaints
Predicting a later outcome, with few past examples	Record outcomes first; use an endpoint only as a stopgap	Chapter 2.1, Resolving and routing customer complaints
A reader could not tell from the evidence whether the answer is right	A person, or a check the system cannot influence	Chapter 3.1, Deciding when an agent is finished
Reading the text, with thousands of labels of your own	Compare an endpoint and a classifier as complete policies	Chapter 1.3, From probabilities to actions
Reading the text, with few labels	Try an endpoint, and spend a few hundred labels calibrating it before acting	Chapter 1.2, From evidence to typed questions

These are starting points drawn from a few cases and not rules. A team's own costs and records can reverse any of them.

Would you deploy this? The complaint desk

Take the desk's routing decision and put the evidence in one place. The choices are to deploy, run a limited pilot, measure further or keep the current process.

Evidence	What the case shows	What is still missing
Judgment	Agreement with consumers' choices of 82% for Jev and 82% for the classifier	How often the consumer's choice is itself wrong
Probabilities	Neither model's stated probabilities can be taken at face value	Calibration on the desk's own complaints
Policy	Recalibrated, $3,471 for Jev and $3,666 for the classifier per thousand complaints, with no clear gap	The desk's real costs of delay and triage
Attention	Ranking review by expected loss per minute changes cost by −425 (95% CI −639 to −198) dollars per thousand, under assumed review times	Measured handling time, and reviewer accuracy on short complaints
Failures	Largest disagreements run between credit reporting and debt collection	What a misroute costs in each direction
Observed effect	None; no policy here ran on a desk	A randomized pilot

This evidence does not justify full deployment. It justifies a local evaluation of whichever model the desk can run most cheaply, with its probabilities recalibrated on a labeled sample of the desk's own complaints, because neither shows a clear lead. Start in shadow, recording the model's picks while the current process keeps routing. Changing live routing also needs a measured baseline, a limited set of complaints and a way back. For the money-back question, closed-case records make a classifier a strong baseline to evaluate, once the desk has checked that outcomes arrive in time, that old records resemble today's complaints and that money back is what it wants to prioritize.

Adoption also changes what the desk can see. Misroutes that bounce back get recorded, and ones that get handled without complaint do not, so a small random sample of automatically routed complaints, reviewed on a regular schedule, is what would show the policy going stale.

Questions to ask before adopting a system

Each case suggests a question for a vendor or an internal team.

Question	Why it matters here
Was it tested on work like yours, or on a tidy test set?	The same engine scored far lower on real code changes than on written ones
What is the simplest alternative, and has it been run on the same records?	A classifier or sorting by size was hard to beat, and the classifier's result depended on how it was fit
What does the label actually measure?	Stars, consumer choices and company responses are all stand-ins
Do the probabilities mean what they say, near the cutoff you will use?	Recalibrating changed the answer in two of the priced decisions
What happens to the answers if the question changes?	Rewording one question changed 2.5% of Jev's answers
Who owns the question, the costs and the cutoff, and what triggers a recheck?	The engine, the costs and the population can each change on their own
Has anyone run this policy, or only scored it?	Only the agent case ran its policies, on tasks written for this booklet

What these data cannot show

This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.

Tables

Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities

Jev's policy against the simple alternative, percent of that alternative's value or cost, with 95% intervals

conf_high	conf_low	decision	gap
3.20	−0.900	Contacting reviewers, raw probabilities (against a classifier)	1.20
8.20	4.50	Contacting reviewers, Jev recalibrated (against a classifier)	6.40
−2.60	−17.80	Routing complaints, raw probabilities (against a classifier)	−10
11.40	−0.700	Routing complaints, both recalibrated (against a classifier)	5.30
34.60	0	Accepting an agent's finish (against a person checking all)	17.50

Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

Data: vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json

Table 2. Jev's ranking score differs widely from one question to the next

AUC by question, Jev and the trained classifier, with 95% intervals; no classifier was trained for the patch and agent cases

auc	conf_high	conf_low	model	task
0.980	0.982	0.978	Jev	Is this reviewer unhappy?
0.968	0.972	0.964	Trained classifier	Is this reviewer unhappy?
0.742	0.770	0.709	Jev	Will this complaint end in money back, within a product?
0.769	0.802	0.736	Trained classifier	Will this complaint end in money back, within a product?
0.785	0.890	0.690	Jev	Would this written patch fail its hidden test?
0.604	0.640	0.560	Jev	Did this real agent patch fail its benchmark?
0.713	0.830	0.550	Jev	Is the agent's work really done?

Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

Data: vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json

All estimates

Every fact in facts.json (38), sorted by id. The same values as JSON: facts.json.

Fact id	Value	Unit	Estimand	Population	Period	Interval	n	Weights	Source	Note
agree_jev	81.9%	percent of complaints	top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 80.5% to 83.2%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
agree_tfidf	82.4%	percent of complaints	top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 81.2% to 83.7%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
auc_across_diff_jev_size	+0.01	difference in AUC	jev minus largest diff first	1,941 agent patches for 259 issues in 20 repositories	issues created by March 2025; engines called October 2026	95% CI −0.03 to 0.05	1,941 patches	sample_weight restores the outcome mix of the patches	SWE-rebench	From allocating-review-attention-to-code-changes.
auc_done_jev	0.71	area under the ROC curve	P(done) against the grader's verdict on finish attempts	200 finish attempts by one agent on 40 tasks written for this booklet	agent runs and engine calls, October 2026	95% CI 0.55 to 0.83	200 finish attempts	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
auc_money_jev	0.895	area under the ROC curve	P(money back) against closed with monetary relief	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.881 to 0.908	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
auc_money_product_only	0.906	area under the ROC curve	ranking complaints by their product's relief rate in the training years	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
auc_money_tfidf	0.936	area under the ROC curve	P(money back) against closed with monetary relief	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.928 to 0.945	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
auc_patch_real	0.60	area under the ROC curve	unresolved ranked above resolved, patches for different issues	1,941 agent patches for 259 issues in 20 repositories	issues created by March 2025; engines called October 2026	95% CI 0.56 to 0.64	1,941 patches	sample_weight restores the outcome mix of the patches	SWE-rebench	From allocating-review-attention-to-code-changes.
auc_patch_written	0.79	area under the ROC curve	ranks defective patches above acceptable ones	40 patches written for this booklet	patches written October 2026; engines called October 2026	95% CI 0.69 to 0.89	40 patches	unweighted; evenly split by construction	Constructed patches	From allocating-review-attention-to-code-changes.
auc_reviews_jev	0.980	area under the ROC curve	weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.978 to 0.982	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-evidence-to-typed-questions.
auc_within_jev	0.742	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.709 to 0.770	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
auc_within_tfidf	0.769	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.736 to 0.802	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
capture_jev	58.2%	percent of complaints closed with monetary relief	share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 53.0% to 63.4%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
capture_tfidf	73.1%	percent of complaints closed with monetary relief	share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 68.4% to 77.9%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
cost_complaints	$0.05	US dollars per 1,000 complaints	mean billed cost per call, list price	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
cut_data	0.38	probability of dissatisfaction	median of five cross-fit cost-minimizing cutoffs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Range across folds 0.35 to 0.38.
cut_theory	0.125	probability of dissatisfaction	cutoff c / (rV)	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023	From from-probabilities-to-actions.
lc_capture_1000	58.8%	percent of complaints closed with monetary relief	mean top-tenth capture over refits on 1,000 labeled complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
lc_net_1000	$4,466	US dollars per 1,000 reviews	mean net value over refits on 1,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-evidence-to-typed-questions.
lc_net_20000	$5,047	US dollars per 1,000 reviews	mean net value over refits on 20,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-evidence-to-typed-questions.
loss_complaints	$3,471	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,243 to $3,709	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product.
loss_diff_jev_cal	−$195	US dollars per 1,000 complaints	jev_cal minus the classifier with a cutoff	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −$419 to $27	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
loss_jev_cal	$3,471	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,243 to $3,709	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product.
loss_tfidf_cal	$3,666	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,466 to $3,885	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product.
lv_clf_200	$3,257	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws.
lv_cost_clf_500	$4,792	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws.
lv_cost_jev_500	$3,613	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws.
lv_jev_200	$5,257	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws.
met_accept	87.5%	percent of runs	accepted final state meets every rule-checked criterion	200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task	agent runs and engine calls, October 2026	95% CI 81.0% to 93.0%	200 runs	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
met_check_named	94.0%	percent of runs	accepted final state meets every rule-checked criterion	200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task	agent runs and engine calls, October 2026	95% CI 90.5% to 97.0%	200 runs	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
met_jev	90.0%	percent of runs	accepted final state meets every rule-checked criterion	200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task	agent runs and engine calls, October 2026	95% CI 85.5% to 94.5%	200 runs	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
met_recheck	89.5%	percent of runs	accepted final state meets every rule-checked criterion	200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task	agent runs and engine calls, October 2026	95% CI 85.0% to 94.0%	200 runs	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
net_jev	$5,076	US dollars per 1,000 reviews	net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $5,004 to $5,144	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Assumed costs; measured error mix.
net_jev_recal	$5,335	US dollars per 1,000 reviews	net value per 1,000 reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $5,273 to $5,393	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Assumed costs; measured error mix.
net_tfidf	$5,015	US dollars per 1,000 reviews	net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $4,928 to $5,104	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-probabilities-to-actions. Assumed costs; measured error mix.
queue_diff_value_per_minute_cost	−425	per 1,000 complaints	value_per_minute minus reviewing the least certain, cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −639 to −198	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	From resolving-and-routing-customer-complaints.
retry_jev	97.4%	percent of checkpoints	next action is retry after a transient failure	38 checkpoints after an injected tool failure	agent runs and engine calls, October 2026		38 checkpoints	unweighted; the mix of tasks and faults describes the test harness	Constructed agent tasks	From deciding-when-an-agent-is-finished.
wording_change	2.5%	percent of reviews	answer differs between the two versions of the question	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 1.4% to 3.9%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	From from-evidence-to-typed-questions.

Definitions

The chapter marks no glossary term.

Methods

This chapter runs nothing new. Its numbers are copied by script from the other chapters' data files.

Putting the priced decisions on one scale

The three decisions use different assumed costs, so each gap and its interval is divided by the simple alternative's dollar figure in the same case, with the sign set so that right means better for Jev. The denominators are treated as fixed, so these ranges are approximate. The percent scale helps compare direction and spread; it does not make the three assumed cost sets comparable.

What these data cannot show

This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.

References

The chapter cites no reference.

Provenance

Build script: decision_ai/chapters/comparing-and-auditing-decision-systems/build.py (sha256 756143a43f0dee2e10eb7f037f68ef8dd2e978818bf3dda7f2366edd2325f058)
Built on: 2026-10-30
Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs

File	SHA-256
facts.json	2cfb55a8f2401a7aa72efc6a2b6d1640fa7810ec3fd37190c7dbc4f286c114cc
figures/policy-gaps.json	03bcbc36dd93675885cb097ae587331a5f2b307df3898b86bebb39006076a018
figures/same-model-three-questions.json	0afa4141d1b4fb93f0fc5ddfe99aeace040d31f3dbcda3a53ed87a31a4babc34
meta.json	1d4dc1729ddbe586dc4909fd14343f12f3319e01cca1b54b0c52cd4824847050
refs.json	ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356

Machine formats

Report (Markdown): https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md
Every estimate (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/facts.json
Table 1, figure "policy-gaps" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json
Table 2, figure "same-model-three-questions" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json
Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt