Skip to content
Contents
Decision AI

    Type to search. Planned chapters match by title only.

    All chapters

    Every volume, part and chapter. A written chapter shows its finding. A planned chapter shows the question it will ask and is marked Planned. The research view adds each chapter’s label, grain and datasets.

    Showing 7 of 7 chapters.

    Introduction

    1. About this bookletDraft

      This booklet explores the decision endpoints that major AI labs have recently released, an interface popularized by TypeSafe's Jev, by posing business and workflow decisions as typed questions and checking whether acting on the answers beats the simplest credible alternative.

      Recordsauthored-examples

    Volume 1

    Case studies

    Part IFrom judgment to action

    1. 1.1

      Decision AI in applications and agentsDraft

      An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.

      Recordsauthored-examples

    2. 1.2

      From evidence to typed questionsDraft

      On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.

      Recordsamazon-reviews-2023

    3. 1.3

      From probabilities to actionsDraft

      At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.

      Recordsamazon-reviews-2023

    Part IIScarce attention

    1. 2.1

      Resolving and routing customer complaintsDraft

      A classifier trained on earlier complaints matches consumers' product choices about as often as Jev once it is fit to the actual mix, and it finds more of the complaints that end in money back, where it passes Jev with far fewer labels than the review task needs.

      Recordscfpb-complaints

    2. 2.2

      Allocating review attention to code changesDraft

      On patches written so that every one passes the required tests, the models rank defective changes well above acceptable ones while diff size does no better than chance, but on real agent patches the same models do only modestly better than chance.

      Recordsconstructed-patchesswe-rebenchnebius-openhands-trajectories

    Part IIIActing and checking

    1. 3.1

      Deciding when an agent is finishedDraft

      On tasks written for this booklet, an agent's first finish met every checked requirement in 87.5% of runs, sending it back on a model's judgment raised that to 90.0%, about what sending every finish back achieves, and telling it which requirement failed raised that to 94.0%.

      Recordsconstructed-agent-tasks

    Part IVDeciding whether to adopt

    1. 4.1

      Comparing decision systemsDraft

      Across reviews, complaints and agent work, the choice between an endpoint and a trained classifier turns on how many labels a task needs and whether the probabilities are calibrated, and an agent's work improved most when a check it could not influence named what had failed.

      Recordsamazon-reviews-2023cfpb-complaintsswe-rebenchconstructed-patchesconstructed-agent-tasks