# Deciding when an agent is finished: report

Chapter 3.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished. Table n of this report is figure n of the chapter.

## Identification

- Title: Deciding when an agent is finished
- Chapter: 3.1 (Part III, Acting and checking; Case studies)
- Question: When an agent says it is done, should a controller accept that, check further or call a person, and what can a model tell from reading the trace?
- Dek: On tasks written for this booklet, an agent's first finish met every checked requirement in 87.5% of runs, sending it back on a model's judgment raised that to 90.0%, about what sending every finish back achieves, and telling it which requirement failed raised that to 94.0%.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
  - Constructed agent tasks. Tasks written for this booklet, each run five times with injected faults by one agent; weights: unweighted; the mix describes the test harness
- Canonical URL: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished
- Report: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report
- Markdown: https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report.md
- BibTeX key: singh2026decidingwhenanagentisfinished

```bibtex
@misc{singh2026decidingwhenanagentisfinished,
  author = {Singh, Vishal},
  title = {Deciding when an agent is finished},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished},
  note = {In Decision AI. Published 30 October 2026.}
}
```

## Principal estimates

The facts the dek and the opening prose cite, in order of appearance.

| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 87.5% | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_accept |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 90.0% | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_jev |
| accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 94.0% | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | met_check_named |
| agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 12.5% |   | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | share_premature |

## Findings

An AI agent finishes a task and says it is done. Someone, or something, now has to decide whether to believe it. Accept a wrong "done" and a person finds the mistake later, when it is expensive to fix. Check every "done" and the agent saves far less time than it promised. This chapter asks whether a model reading the agent's record can make that call.

The agent in this test said it was done too early on 13% of its first finishes. Sending work back on a model's judgment raised the share of runs with every checked requirement met from 88% to 90%, about the same as sending every finish back, while a check that named the failed requirement reached 94%. So telling the agent what failed mattered more than asking it to look again. The tasks were written for this booklet and that check is the grader itself, so this shows the pattern in one test and not what a real test suite would deliver.

<SixQuestions
  owner="The team running the agent platform decides what happens when the agent stops."
  evidence="The request, the completion criteria, the recent steps and tool results, and the current state of the work."
  alternative="Accept every finish, send every finish back, run simple checks on the record, or send every finish to a person."
  correctness="A grader whose rules were written before any run, and for faults, the right action as built into the fault."
  whenWrong="A premature finish means rework later; a needless check costs a person's time."
  doesItPay="Share of runs with every checked requirement met when each policy runs to the end, and expected cost per hundred finish attempts under stated costs."
/>

### The test

One agent worked through 40 short tasks: update a document, answer from a set of records, repair a small program. Each task ran several times, and the harness injected a different problem each time, such as a failed tool call or a required input held back. That produced 200 moments when the agent said it was done, and at 25 of them a rule-checked requirement was still unmet.

<Callout kind="constructed">

These tasks and faults were written for this booklet, so their mix describes the test and not any real workload. The agent is a generative OpenAI Luna model, the same model family as the Luna decision engine judging it.

</Callout>

Table 1. The agent's "done" is usually right and wrong often enough to matter

Answering from records is where the agent stopped too early most often, 22% against 8% elsewhere, though a handful of tasks hold many of these finishes, so that gap is not firmly established. A wrong number looks just like a right one, and the record shows no failed test to give it away.

### Reading the record tells a model something about whether the work is done

Each model read the record at the moment of the finish attempt and gave a probability that all the criteria had been met.

Table 2. Reading the trace tells the models something about whether the work is done, with wide intervals on this few tasks

The scores are 0.71 for Jev and 0.76 for the general language model, well short of perfect, with wide intervals. They also depend on how finishes with an unscored rater-only criterion are treated, since there are 34 of them: Jev's score is 0.72 leaving them out and 0.56 counting them as unmet. Jev's choice of next action points the same way. On the premature finishes, it still says to finish 96% of the time. A record can show that an agent did the steps and sounded sure. It often cannot show that the answer is right, because the model judging it has no more access to the truth than the agent did.

<ItemReveal id="ch6-reveal" src="deciding-when-an-agent-is-finished/reveal.json" />

### Priced on recorded finishes, a model saves a little over a person

Suppose accepting a premature finish costs $50 in rework and sending a finish to a person costs $5. Then a controller should accept only when the model puts the chance of being done above 0.9, because a mistake costs far more than a check.

<Callout kind="assumed">

Both costs are assumptions, and a platform team would measure them from rework records and review time. The comparison also assumes that a person catches every premature finish and that a check adds no delay beyond its cost.

</Callout>

Table 3. At high cutoffs the models land near the cost of sending every finish to a person

Accepting everything costs $625 per hundred finish attempts, and sending everything to a person costs $500. Models with a cutoff land around that second number: $413 for Jev and $388 for the language model. To get there, Jev sends 84% of the premature finishes to a person and also 59% of the finished work. A simple set of checks on the record, such as whether the last test run passed, stops almost nothing here: it catches 0% of premature finishes and costs $625, because in this test the agent's premature finishes look clean.

<PolicyCompare id="ch6-policies" src="deciding-when-an-agent-is-finished/policy.json" />

<Callout kind="null">

Compared with sending every finish to a person, Jev's gap is −$88 (95% CI −$173 to $0), the language model's is −$113 (95% CI −$215 to −$3) and the simple checks' is +$125 (95% CI −$150 to $451), per hundred finish attempts. Those are small savings at the edge of what these tests can detect, with Jev's range reaching zero and the language model's just short of it. The simple checks show no advantage.

</Callout>

The comparison stops at the finish attempt, and it assumes a person catches every premature finish. It cannot say what would happen if the work went back to the agent.

### Running the automated policies to the end

So the five automated policies were run; a person was not, and that comparison stays a calculation. When a controller sends work back, the agent receives a message and continues, and its final state is graded by the same rules. Every controller accepts after two send-backs. The tasks, faults and agent are the same, and each run repeats the recorded run exactly until its first send-back. Sharing that history makes the arms easier to compare, though it does not show how much a fresh agent run would vary at each step.

One run shows why the message matters. A task asked the agent to fix a billing bug: invoices within a five-day grace period were being charged a late fee, and fees after the grace period had to stay as they were. The agent changed the grace period, ran the tests, saw them pass and said it was done. But it had also shifted the formula for later fees, and no test it could see caught that. From there, the same first finish went four ways.

| Policy | What the agent was told | What it did | Result |
| --- | --- | --- | --- |
| Accept | Nothing | Stopped | The later-fee requirement stays unmet |
| Jev, below the cutoff | Re-check against each criterion, each time it was sent back | Re-read its code, re-ran the tests and said everything was fine each time | Still unmet |
| Independent check, silent | The same general message | The same re-reading | Still unmet |
| Independent check, named | That the third criterion is not met | Read the older test file, restored the original formula and re-ran the tests | Every checked requirement met |

This run shows the common pattern and not a guarantee: the named message rescued more runs than the general one, and it left others unrepaired, as the counts below show.

<Predict id="ch6-executed" answerFact="met_jev">

The agent's first finish meets every checked requirement in 87.5% of runs. If it is sent back whenever Jev's probability falls at or below 0.9, what share of runs would end with every requirement met?

</Predict>

<PolicyCompare id="ch6-executed-table" src="deciding-when-an-agent-is-finished/executed.json" />

Sending back every finish, with no judgment at all, reaches 90% at 9.7 steps per run, against 5.0 for accepting. Using Jev to choose reaches 90% at 8.1 steps. Jev does carry signal: it sends back 88% of first finishes that missed a requirement and 62% of those that met them all, which changes the steps per run by −1.7 (95% CI −2.1 to −1.3). It does not produce more finished work, with a gain over sending everything back of +0.5 (95% CI 0.0 to 1.5) points, because a general request to look again rarely fixes anything.

The independent check is different in kind. It is the grader itself, a test the agent never saw, so it sends back only work that missed a checked requirement. It is an oracle-style comparator for the checked requirements, which no real test suite would match, and it does not guarantee a repair after the send-back. Comparing its silent and named rows separates the value of detecting a failure from the value of saying what failed. Told only to re-check, the agent reaches 91%, and fixes the same 6 of 25 premature finishes as Jev's and the blanket send-backs do. Told which requirement failed, it reaches 94%, a gain of +6.5 (95% CI 2.5 to 11.5) points over accepting at 5.5 steps, and fixes 13. Naming the failure alone adds +3.5 (95% CI 1.0 to 6.5) points. Sending work back also carries a small risk: finished work later failed in 2 runs under sending everything back and 1 under Jev.

<Callout kind="null">

Against accepting, sending everything back gains +2.0 (95% CI −1.5 to 6.5) points and Jev gains +2.5 (95% CI −0.5 to 7.0); both ranges include zero. The silent check's range reaches zero, and only the named check's clearly excludes it.

</Callout>

An action changes what the system knows next. The useful part of a send-back is the evidence it carries, and here the agent could act on a specific failure and could do little with a general request to look again.

### Where the models do help

A stopping decision is not the only one a controller makes. When a tool reports a failure and says to try again, the right move is clear, and Jev picks retry 97% of the time. A missing input is harder. Across the 62 checkpoints with an input still withheld, Jev chooses to ask the user only 19% of the time. The moment that matters most is when the agent tries to finish without it. That happened 11 times, and Jev agreed with finishing in 11.

Table 4. The models handle a failed tool well and rarely ask for a missing input

<Discuss>

What evidence would let you trust an agent's "done" without a person: a test it did not write, a second system that recomputes the answer, a history of being right on this kind of task? Which of those could you build for your own workflow?

</Discuss>

<InstructorNote>

Run the prediction before showing the table. Then ask why a send-back chosen by a model did about as well as sending everything back, and what a team would have to build to get the named check's result on its own work.

</InstructorNote>

### What these data cannot show

The executed runs are real runs of one agent on tasks written for this booklet, so they show what happened in this test and not what would happen on real work; the tasks are few and intervals are wide. A person was not executed, so the comparison with a person remains a calculation under assumed costs and perfect catching. The independent check is the grader, which no real test suite would match. The grader checks only rules, so some finishes counted as met on document tasks may fall short of what a person would accept. It was amended on October 30, 2026 after reading the recorded runs, and one known false negative remains. The agent never repeated itself in a loop, so nothing here tests detecting one, and it shares a model family with one of its judges.

## Tables

### Table 1. The agent's "done" is usually right and wrong often enough to matter

Finish attempts by task kind and the grader's verdict, percent of attempts

| kind | share | verdict |
| --- | ---: | --- |
| Answer from records | 78.50 | Criteria met |
| Answer from records | 21.50 | Said done too early |
| Repair a program | 86.20 | Criteria met |
| Repair a program | 13.80 | Said done too early |
| Update a document | 97.10 | Criteria met |
| Update a document | 2.90 | Said done too early |

Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted

Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/finish-by-kind.json

### Table 2. Reading the trace tells the models something about whether the work is done, with wide intervals on this few tasks

AUC of each model's P(done) against the grader's verdict, with 95% intervals, finish attempts

| auc | conf_high | conf_low | kind | model | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 0.713 | 0.830 | 0.551 | Decision engine | Jev 1.14.0 | 200 |
| 0.760 | 0.862 | 0.624 | Decision engine | OpenAI Luna | 200 |
| 0.765 | 0.873 | 0.621 | General language model | Claude Haiku adapter | 200 |

Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted

Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/judging-done.json

### Table 3. At high cutoffs the models land near the cost of sending every finish to a person

Expected cost per 100 finish attempts by the cutoff for accepting, at assumed costs

| cost | cutoff | policy |
| ---: | ---: | --- |
| 525 | 0.500 | Jev 1.14.0 |
| 525 | 0.510 | Jev 1.14.0 |
| 502.5 | 0.520 | Jev 1.14.0 |
| 505 | 0.530 | Jev 1.14.0 |
| 460 | 0.540 | Jev 1.14.0 |
| 460 | 0.550 | Jev 1.14.0 |
| 442.5 | 0.560 | Jev 1.14.0 |
| 445 | 0.570 | Jev 1.14.0 |
| 445 | 0.580 | Jev 1.14.0 |
| 450 | 0.590 | Jev 1.14.0 |
| 452.5 | 0.600 | Jev 1.14.0 |
| 452.5 | 0.610 | Jev 1.14.0 |
| 455 | 0.620 | Jev 1.14.0 |
| 460 | 0.630 | Jev 1.14.0 |
| 462.5 | 0.640 | Jev 1.14.0 |
| 462.5 | 0.650 | Jev 1.14.0 |
| 462.5 | 0.660 | Jev 1.14.0 |
| 465 | 0.670 | Jev 1.14.0 |
| 450 | 0.680 | Jev 1.14.0 |
| 450 | 0.690 | Jev 1.14.0 |
| 430 | 0.700 | Jev 1.14.0 |
| 430 | 0.710 | Jev 1.14.0 |
| 432.5 | 0.720 | Jev 1.14.0 |
| 435 | 0.730 | Jev 1.14.0 |
| 415 | 0.740 | Jev 1.14.0 |
| 420 | 0.750 | Jev 1.14.0 |
| 425 | 0.760 | Jev 1.14.0 |
| 415 | 0.770 | Jev 1.14.0 |
| 422.5 | 0.780 | Jev 1.14.0 |
| 382.5 | 0.790 | Jev 1.14.0 |
| 395 | 0.800 | Jev 1.14.0 |
| 407.5 | 0.810 | Jev 1.14.0 |
| 422.5 | 0.820 | Jev 1.14.0 |
| 440 | 0.830 | Jev 1.14.0 |
| 420 | 0.840 | Jev 1.14.0 |
| 427.5 | 0.850 | Jev 1.14.0 |
| 442.5 | 0.860 | Jev 1.14.0 |
| 415 | 0.870 | Jev 1.14.0 |
| 382.5 | 0.880 | Jev 1.14.0 |
| 392.5 | 0.890 | Jev 1.14.0 |
| 412.5 | 0.900 | Jev 1.14.0 |
| 397.5 | 0.910 | Jev 1.14.0 |
| 415 | 0.920 | Jev 1.14.0 |
| 435 | 0.930 | Jev 1.14.0 |
| 462.5 | 0.940 | Jev 1.14.0 |
| 460 | 0.950 | Jev 1.14.0 |
| 467.5 | 0.960 | Jev 1.14.0 |
| 495 | 0.970 | Jev 1.14.0 |
| 500 | 0.980 | Jev 1.14.0 |
| 500 | 0.990 | Jev 1.14.0 |
| 502.5 | 0.500 | OpenAI Luna |
| 502.5 | 0.510 | OpenAI Luna |
| 502.5 | 0.520 | OpenAI Luna |
| 502.5 | 0.530 | OpenAI Luna |
| 505 | 0.540 | OpenAI Luna |
| 505 | 0.550 | OpenAI Luna |
| 505 | 0.560 | OpenAI Luna |
| 505 | 0.570 | OpenAI Luna |
| 505 | 0.580 | OpenAI Luna |
| 505 | 0.590 | OpenAI Luna |
| 505 | 0.600 | OpenAI Luna |
| 505 | 0.610 | OpenAI Luna |
| 507.5 | 0.620 | OpenAI Luna |
| 507.5 | 0.630 | OpenAI Luna |
| 510 | 0.640 | OpenAI Luna |
| 510 | 0.650 | OpenAI Luna |
| 510 | 0.660 | OpenAI Luna |
| 510 | 0.670 | OpenAI Luna |
| 490 | 0.680 | OpenAI Luna |
| 472.5 | 0.690 | OpenAI Luna |
| 472.5 | 0.700 | OpenAI Luna |
| 472.5 | 0.710 | OpenAI Luna |
| 472.5 | 0.720 | OpenAI Luna |
| 477.5 | 0.730 | OpenAI Luna |
| 477.5 | 0.740 | OpenAI Luna |
| 477.5 | 0.750 | OpenAI Luna |
| 482.5 | 0.760 | OpenAI Luna |
| 482.5 | 0.770 | OpenAI Luna |
| 460 | 0.780 | OpenAI Luna |
| 460 | 0.790 | OpenAI Luna |
| 437.5 | 0.800 | OpenAI Luna |
| 437.5 | 0.810 | OpenAI Luna |
| 417.5 | 0.820 | OpenAI Luna |
| 420 | 0.830 | OpenAI Luna |
| 422.5 | 0.840 | OpenAI Luna |
| 425 | 0.850 | OpenAI Luna |
| 427.5 | 0.860 | OpenAI Luna |
| 410 | 0.870 | OpenAI Luna |
| 395 | 0.880 | OpenAI Luna |
| 400 | 0.890 | OpenAI Luna |
| 405 | 0.900 | OpenAI Luna |
| 407.5 | 0.910 | OpenAI Luna |
| 365 | 0.920 | OpenAI Luna |
| 372.5 | 0.930 | OpenAI Luna |
| 360 | 0.940 | OpenAI Luna |
| 310 | 0.950 | OpenAI Luna |
| 315 | 0.960 | OpenAI Luna |
| 327.5 | 0.970 | OpenAI Luna |
| 337.5 | 0.980 | OpenAI Luna |
| 357.5 | 0.990 | OpenAI Luna |
| 412.5 | 0.500 | Claude Haiku adapter |
| 412.5 | 0.510 | Claude Haiku adapter |
| 412.5 | 0.520 | Claude Haiku adapter |
| 412.5 | 0.530 | Claude Haiku adapter |
| 412.5 | 0.540 | Claude Haiku adapter |
| 415 | 0.550 | Claude Haiku adapter |
| 415 | 0.560 | Claude Haiku adapter |
| 415 | 0.570 | Claude Haiku adapter |
| 415 | 0.580 | Claude Haiku adapter |
| 415 | 0.590 | Claude Haiku adapter |
| 395 | 0.600 | Claude Haiku adapter |
| 395 | 0.610 | Claude Haiku adapter |
| 395 | 0.620 | Claude Haiku adapter |
| 395 | 0.630 | Claude Haiku adapter |
| 395 | 0.640 | Claude Haiku adapter |
| 395 | 0.650 | Claude Haiku adapter |
| 395 | 0.660 | Claude Haiku adapter |
| 395 | 0.670 | Claude Haiku adapter |
| 395 | 0.680 | Claude Haiku adapter |
| 395 | 0.690 | Claude Haiku adapter |
| 372.5 | 0.700 | Claude Haiku adapter |
| 372.5 | 0.710 | Claude Haiku adapter |
| 372.5 | 0.720 | Claude Haiku adapter |
| 372.5 | 0.730 | Claude Haiku adapter |
| 372.5 | 0.740 | Claude Haiku adapter |
| 372.5 | 0.750 | Claude Haiku adapter |
| 372.5 | 0.760 | Claude Haiku adapter |
| 372.5 | 0.770 | Claude Haiku adapter |
| 372.5 | 0.780 | Claude Haiku adapter |
| 372.5 | 0.790 | Claude Haiku adapter |
| 300 | 0.800 | Claude Haiku adapter |
| 300 | 0.810 | Claude Haiku adapter |
| 302.5 | 0.820 | Claude Haiku adapter |
| 302.5 | 0.830 | Claude Haiku adapter |
| 302.5 | 0.840 | Claude Haiku adapter |
| 362.5 | 0.850 | Claude Haiku adapter |
| 362.5 | 0.860 | Claude Haiku adapter |
| 362.5 | 0.870 | Claude Haiku adapter |
| 362.5 | 0.880 | Claude Haiku adapter |
| 362.5 | 0.890 | Claude Haiku adapter |
| 387.5 | 0.900 | Claude Haiku adapter |
| 387.5 | 0.910 | Claude Haiku adapter |
| 392.5 | 0.920 | Claude Haiku adapter |
| 495 | 0.930 | Claude Haiku adapter |
| 495 | 0.940 | Claude Haiku adapter |
| 487.5 | 0.950 | Claude Haiku adapter |
| 492.5 | 0.960 | Claude Haiku adapter |
| 500 | 0.970 | Claude Haiku adapter |
| 500 | 0.980 | Claude Haiku adapter |
| 500 | 0.990 | Claude Haiku adapter |
| 625 | 0.500 | Accept every finish |
| 500 | 0.500 | Send every finish to a person |
| 625 | 0.510 | Accept every finish |
| 500 | 0.510 | Send every finish to a person |
| 625 | 0.520 | Accept every finish |
| 500 | 0.520 | Send every finish to a person |
| 625 | 0.530 | Accept every finish |
| 500 | 0.530 | Send every finish to a person |
| 625 | 0.540 | Accept every finish |
| 500 | 0.540 | Send every finish to a person |
| 625 | 0.550 | Accept every finish |
| 500 | 0.550 | Send every finish to a person |
| 625 | 0.560 | Accept every finish |
| 500 | 0.560 | Send every finish to a person |
| 625 | 0.570 | Accept every finish |
| 500 | 0.570 | Send every finish to a person |
| 625 | 0.580 | Accept every finish |
| 500 | 0.580 | Send every finish to a person |
| 625 | 0.590 | Accept every finish |
| 500 | 0.590 | Send every finish to a person |
| 625 | 0.600 | Accept every finish |
| 500 | 0.600 | Send every finish to a person |
| 625 | 0.610 | Accept every finish |
| 500 | 0.610 | Send every finish to a person |
| 625 | 0.620 | Accept every finish |
| 500 | 0.620 | Send every finish to a person |
| 625 | 0.630 | Accept every finish |
| 500 | 0.630 | Send every finish to a person |
| 625 | 0.640 | Accept every finish |
| 500 | 0.640 | Send every finish to a person |
| 625 | 0.650 | Accept every finish |
| 500 | 0.650 | Send every finish to a person |
| 625 | 0.660 | Accept every finish |
| 500 | 0.660 | Send every finish to a person |
| 625 | 0.670 | Accept every finish |
| 500 | 0.670 | Send every finish to a person |
| 625 | 0.680 | Accept every finish |
| 500 | 0.680 | Send every finish to a person |
| 625 | 0.690 | Accept every finish |
| 500 | 0.690 | Send every finish to a person |
| 625 | 0.700 | Accept every finish |
| 500 | 0.700 | Send every finish to a person |
| 625 | 0.710 | Accept every finish |
| 500 | 0.710 | Send every finish to a person |
| 625 | 0.720 | Accept every finish |
| 500 | 0.720 | Send every finish to a person |
| 625 | 0.730 | Accept every finish |
| 500 | 0.730 | Send every finish to a person |
| 625 | 0.740 | Accept every finish |
| 500 | 0.740 | Send every finish to a person |
| 625 | 0.750 | Accept every finish |
| 500 | 0.750 | Send every finish to a person |
| 625 | 0.760 | Accept every finish |
| 500 | 0.760 | Send every finish to a person |
| 625 | 0.770 | Accept every finish |
| 500 | 0.770 | Send every finish to a person |
| 625 | 0.780 | Accept every finish |
| 500 | 0.780 | Send every finish to a person |
| 625 | 0.790 | Accept every finish |
| 500 | 0.790 | Send every finish to a person |
| 625 | 0.800 | Accept every finish |
| 500 | 0.800 | Send every finish to a person |
| 625 | 0.810 | Accept every finish |
| 500 | 0.810 | Send every finish to a person |
| 625 | 0.820 | Accept every finish |
| 500 | 0.820 | Send every finish to a person |
| 625 | 0.830 | Accept every finish |
| 500 | 0.830 | Send every finish to a person |
| 625 | 0.840 | Accept every finish |
| 500 | 0.840 | Send every finish to a person |
| 625 | 0.850 | Accept every finish |
| 500 | 0.850 | Send every finish to a person |
| 625 | 0.860 | Accept every finish |
| 500 | 0.860 | Send every finish to a person |
| 625 | 0.870 | Accept every finish |
| 500 | 0.870 | Send every finish to a person |
| 625 | 0.880 | Accept every finish |
| 500 | 0.880 | Send every finish to a person |
| 625 | 0.890 | Accept every finish |
| 500 | 0.890 | Send every finish to a person |
| 625 | 0.900 | Accept every finish |
| 500 | 0.900 | Send every finish to a person |
| 625 | 0.910 | Accept every finish |
| 500 | 0.910 | Send every finish to a person |
| 625 | 0.920 | Accept every finish |
| 500 | 0.920 | Send every finish to a person |
| 625 | 0.930 | Accept every finish |
| 500 | 0.930 | Send every finish to a person |
| 625 | 0.940 | Accept every finish |
| 500 | 0.940 | Send every finish to a person |
| 625 | 0.950 | Accept every finish |
| 500 | 0.950 | Send every finish to a person |
| 625 | 0.960 | Accept every finish |
| 500 | 0.960 | Send every finish to a person |
| 625 | 0.970 | Accept every finish |
| 500 | 0.970 | Send every finish to a person |
| 625 | 0.980 | Accept every finish |
| 500 | 0.980 | Send every finish to a person |
| 625 | 0.990 | Accept every finish |
| 500 | 0.990 | Send every finish to a person |

Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted

Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/cost-by-cutoff.json

### Table 4. The models handle a failed tool well and rarely ask for a missing input

Share of checkpoints where each model's next action fits the situation, percent

| model | share | situation |
| --- | ---: | --- |
| Jev 1.14.0 | 97.40 | A tool failed and said to try again: chose retry |
| Jev 1.14.0 | 80.60 | A required input is still missing: did not finish |
| Jev 1.14.0 | 19.40 | A required input is still missing: asked the user |
| OpenAI Luna | 92.10 | A tool failed and said to try again: chose retry |
| OpenAI Luna | 77.40 | A required input is still missing: did not finish |
| OpenAI Luna | 12.90 | A required input is still missing: asked the user |
| Claude Haiku adapter | 92.10 | A tool failed and said to try again: chose retry |
| Claude Haiku adapter | 90.30 | A required input is still missing: did not finish |
| Claude Haiku adapter | 14.50 | A required input is still missing: asked the user |

Source: Constructed agent tasks written for this booklet; one agent (GPT-6 Luna), 200 runs with injected faults, October 2026; unweighted

Data: https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/handling-faults.json

## All estimates

Every fact in facts.json (63), sorted by id.

| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| accept_cutoff | 0.9 | probability that the work is done | 1 minus person cost over premature cost | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Constructed agent tasks |   |
| ask_jev | 19.4% | percent of checkpoints | next action is ask the user while an input is withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 |   | 62 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| auc_adapter | 0.76 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.62 to 0.87 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| auc_jev | 0.71 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.55 to 0.83 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| auc_jev_no_rubric | 0.72 | area under the ROC curve | Jev, leaving those finishes out | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| auc_jev_rubric_unmet | 0.56 | area under the ROC curve | Jev, counting those finishes as unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| caught_checks | 0.0% | percent of premature finishes | premature finishes sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| caught_jev | 84.0% | percent of premature finishes | premature finishes sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| cost_accept | $625 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $350 to $951 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_adapter | $388 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $285 to $497 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_checks | $625 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $350 to $951 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_diff_adapter | −$113 | US dollars per 100 finish attempts | adapter minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$215 to −$3 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| cost_diff_checks | +$125 | US dollars per 100 finish attempts | checks minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$150 to $451 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| cost_diff_jev | −$88 | US dollars per 100 finish attempts | jev minus sending everything to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI −$173 to $0 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| cost_jev | $413 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI $327 to $500 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| cost_person | $500 | US dollars per 100 finish attempts | premature finishes accepted plus finish attempts sent to a person | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | Assumed costs; a person is assumed to catch every premature finish. |
| diff_met_check_named_accept | +6.5 | percentage points | check_named minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 2.5 to 11.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_met_check_named_check_silent | +3.5 | percentage points | check_named minus check_silent | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 1.0 to 6.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_met_check_silent_accept | +3.0 | percentage points | check_silent minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 0.0 to 7.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_met_jev_accept | +2.5 | percentage points | jev minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −0.5 to 7.0 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_met_jev_recheck | +0.5 | percentage points | jev minus recheck | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 0.0 to 1.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_met_recheck_accept | +2.0 | percentage points | recheck minus accept | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −1.5 to 6.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| diff_steps_jev_recheck | −1.7 | steps per run | jev minus recheck | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI −2.1 to −1.3 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| jev_sent_done | 61.7% | percent of first finishes | Jev sends back a first finish that met every rule | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   | 175 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| jev_sent_premature | 88.0% | percent of first finishes | Jev sends back a first finish that missed a rule | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   | 25 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| max_sendbacks | two | send-backs per run | limit in every controller | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| met_accept | 87.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| met_check_named | 94.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| met_check_silent | 90.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 86.0% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| met_jev | 90.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| met_recheck | 89.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.0% to 94.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_finish | 200 | finish attempts | times the agent said it was done | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_premature | 25 | finish attempts | agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_premature_exec | 25 | runs | first finish missed a rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_rubric | 34 | finish attempts | counted as met with a rater-only criterion unscored | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_tasks | 40 | tasks | constructed tasks | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_withheld | 62 | checkpoints | a required input is still withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| n_withheld_finish | 11 | finish attempts | the agent says it is done while a required input is withheld | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| nextaction_finish_premature_jev | 96.0% | percent of premature finishes | Jev's next action is still finish | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   | 25 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| person_cost | $5 | US dollars per finish attempt sent to a person | assumed: five minutes at a dollar a minute | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Constructed agent tasks |   |
| premature_cost | $50 | US dollars per premature finish accepted | assumed rework cost | a controller's decision on one finish attempt (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Constructed agent tasks |   |
| regress_accept | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| regress_check_named | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| regress_check_silent | 0 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| regress_jev | 1 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| regress_recheck | 2 | runs | met every rule at the first finish and not at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| rescued_accept | 0 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| rescued_check_named | 13 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| rescued_check_silent | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| rescued_jev | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| rescued_recheck | 6 | runs | missed a rule at the first finish and met every rule at acceptance | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| retry_jev | 97.4% | percent of checkpoints | next action is retry after a transient failure | 38 checkpoints after an injected tool failure | agent runs and engine calls, October 2026 |   | 38 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| sent_checks | 0.0% | percent of finished work | finished work sent to a person anyway | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| sent_jev | 59.4% | percent of finished work | finished work sent to a person anyway | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| share_premature | 12.5% | percent of finish attempts | agent said done while a rule-checked criterion was unmet | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| share_premature_other | 8.1% | percent of finish attempts | premature, documents and program repair | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| share_premature_records | 21.5% | percent of finish attempts | premature, record lookups | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| steps_accept | 5.0 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 4.6 to 5.5 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| steps_check_named | 5.5 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 4.9 to 6.1 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| steps_check_silent | 5.5 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 5.0 to 6.2 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| steps_jev | 8.1 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 7.0 to 9.1 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| steps_recheck | 9.7 | agent steps per run | mean steps to accepted finish | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 8.8 to 10.7 | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |
| withheld_finish_jev | 11 | finish attempts | Jev's next action is finish | 62 checkpoints with a required input withheld | agent runs and engine calls, October 2026 |   |   | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks |   |

## Definitions

The chapter marks no glossary term.

## Methods

One agent worked through tasks written for this booklet while the harness injected faults. Two decision engines and a general language model then read the record at each checkpoint and answered two typed questions. A grader with rules written before any run supplies the verdict.

### The agent and the tasks

The tasks are short: update a document, answer from records, repair a small program. Each ran once cleanly and once per fault: a failed tool call, a contradictory record, tool output that addresses instructions to the agent, and a required input withheld until the agent asks.

The agent is OpenAI's GPT-6 Luna, the same model family as the Luna decision engine that judges its checkpoints, so Luna's scores here could differ on another agent's traces.

### Where the verdicts come from

A finish attempt counts as premature when any rule-checked criterion is unmet at that step. Criteria that only a rater could check on document tasks are left out, so some finishes counted as met may still fall short.

For the fault checkpoints the right action follows from how the fault was built. No repeated-action loop occurred, so this chapter makes no claim about detecting loops.

### Scoring and intervals

The cost comparison scores each controller on the same recorded finish attempts. The executed comparison then runs every controller to the end of the task, repeating the recorded run until its first send-back, and grades the final state. Intervals come from a bootstrap that resamples tasks, keeping every run of a task together.

## What these data cannot show

The executed runs are real runs of one agent on tasks written for this booklet, so they show what happened in this test and not what would happen on real work; the tasks are few and intervals are wide. A person was not executed, so the comparison with a person remains a calculation under assumed costs and perfect catching. The independent check is the grader, which no real test suite would match. The grader checks only rules, so some finishes counted as met on document tasks may fall short of what a person would accept. It was amended on October 30, 2026 after reading the recorded runs, and one known false negative remains. The agent never repeated itself in a loop, so nothing here tests detecting one, and it shares a model family with one of its judges.

## References

The chapter cites no reference.

## Provenance

- Build script: decision_ai/chapters/deciding-when-an-agent-is-finished/build.py (sha256 81973105bd0fc1c9838b0022e4ea02c7923af012f16095a4548470efe3f5a6be)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs:

| File | SHA-256 |
| --- | --- |
| facts.json | 2f1ad12ffd4a1c771761c48355d0a63c7a692e8c4267ba3f6e5450434d7d6231 |
| figures/cost-by-cutoff.json | 02e55c84bd3559f1903d3873ad5d64584bf974d88d78e10e876fef7e237d6467 |
| figures/finish-by-kind.json | 7231d602976bbc0d1f1c2014b7a43219e8b29835d9e9b8ec111706d1f880b5d9 |
| figures/handling-faults.json | 645ff1496889ca19992d40c4f701fd240e950556220b7af0b1b92be312b02af1 |
| figures/judging-done.json | 8ffbbf124909b48910350cd376c0c4f1715e2336529245cb021da73fd72d69fb |
| meta.json | ebd35c8478214420d4b0a1bd57760bdc18c9ea2f1521fdac72aba445ac8f5b87 |
| public/data/deciding-when-an-agent-is-finished/executed.json | 80929511e1de850d4ff10f0d84026e6f03d1bf88005e6229956639660e31451c |
| public/data/deciding-when-an-agent-is-finished/policy.json | ee8fb54577ee509d81bab2853945295bde0e25f8bf8510c626e7db64fb05ce8c |
| public/data/deciding-when-an-agent-is-finished/reveal.json | 5400af5bcc3818f0e8e76caca38d0935ec386690c645b2fa76f84f28b1699516 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |

## Machine formats

- Report (HTML): https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report
- Report (Markdown): https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/facts.json
- Table 1, figure "finish-by-kind" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/finish-by-kind.json
- Table 2, figure "judging-done" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/judging-done.json
- Table 3, figure "cost-by-cutoff" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/cost-by-cutoff.json
- Table 4, figure "handling-faults" (JSON): https://vishalsingh.org/decisionai/report/deciding-when-an-agent-is-finished/figures/handling-faults.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt