# From probabilities to actions: report

Chapter 1.3 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/from-probabilities-to-actions. Table n of this report is figure n of the chapter.

## Identification

- Title: From probabilities to actions
- Chapter: 1.3 (Part I, From judgment to action; Case studies)
- Question: Given the cost of a contact and the value of a recovered customer, which dissatisfied reviewers should a coffee roaster contact, and what does that rule assume?
- Dek: At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
  - Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating; weights: sample_weight restores the star mix of the coffee reviews
- Canonical URL: https://vishalsingh.org/decisionai/from-probabilities-to-actions
- Report: https://vishalsingh.org/decisionai/from-probabilities-to-actions/report
- Markdown: https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md
- BibTeX key: singh2026fromprobabilitiestoactions

```bibtex
@misc{singh2026fromprobabilitiestoactions,
  author = {Singh, Vishal},
  title = {From probabilities to actions},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/from-probabilities-to-actions},
  note = {In Decision AI. Published 29 October 2026.}
}
```

## Principal estimates

The facts the dek and the opening prose cite, in order of appearance.

| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $6 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 | contact_cost |
| r times V | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $48 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 | expected_recovery |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,076 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,015 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_tfidf |
| net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,335 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev_recal |
| assumed value | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | $120 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 | customer_value |
| assumed rate | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 | 0.4 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 | recovery_rate |
| sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 5,000 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_reviews |
| weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 14.2% |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | share_dissatisfied |
| difference in net value, jev_recal minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | +$320 | 95% CI $228 to $411 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_diff_jev_recal_tfidf |

## Findings

Bean & Basket, a coffee roaster that sells bagged coffee online, wants its care team to contact unhappy reviewers with a replacement bag or a voucher. The firm is illustrative, and the reviews come from many coffee products sold on Amazon. Suppose a contact costs $6 in staff time and voucher value, a retained customer is worth $120, and a contact wins back a dissatisfied customer with probability 0.4. A contact with a dissatisfied reviewer is then worth $48 on average, and a contact with a satisfied reviewer is wasted. Each of 5,000 sampled coffee reviews has a model's probability that the reviewer is dissatisfied, and weighted to all coffee reviews, 14.2% of reviewers gave one or two stars. Which reviewers should the team contact?

At these assumed costs, Jev's raw probabilities and a trained classifier earn about the same, $5,100 and $5,000 per thousand reviews, because Jev's probabilities run too high near the cutoff. Recalibrated on labeled reviews, Jev earns about $5,300, ahead of the classifier by +$320 (95% CI $228 to $411). So for this policy and these costs, the ranking edge paid off once labels went into the probabilities. Stars stand in for dissatisfaction, the costs are assumptions, and dissatisfaction alone does not tell you who will respond to a voucher.

<Callout kind="assumed">

The contact cost, the customer value and the recovery rate are assumptions written for this chapter. Finance would estimate the first two from voucher costs and repeat-purchase margins. The recovery rate needs a randomized contact experiment, which the last section of this chapter simulates.

</Callout>

<SixQuestions
  owner="The customer-care lead decides whom to contact; finance owns the costs."
  evidence="The review's title and text, available the day it is posted."
  alternative="A keyword rule, or a classifier trained on reviews of other coffee products."
  correctness="The reviewer's own star rating, used as a proxy: one or two stars counts as dissatisfied."
  whenWrong="A wasted contact costs the voucher; a missed unhappy customer may not buy again."
  doesItPay="Net value per thousand reviews under the stated costs, compared across policies."
/>

### The cutoff comes from the costs

Contacting a reviewer pays when the expected recovery exceeds the cost: when the probability of dissatisfaction p satisfies p times rV greater than c. The cutoff is

$$
p^{*} = \frac{c}{rV}
$$

which is 0.125 at these costs. A cutoff this low means the team should contact many reviewers who are probably satisfied, because a contact is cheap relative to what a recovered customer is worth.

<Predict id="ch3-shortcut" answerFact="p_shortcut">

Many textbooks give the cutoff as the cost of a false alarm divided by the sum of that cost and the cost of a miss. With a false alarm costing c and a miss costing rV, what does that formula give here, and is it the same as p*?

</Predict>

The shortcut gives 0.111, a little lower than 0.125. It assumes that a correct decision costs nothing, and here a correct contact still costs c. The difference is small at these costs and grows as c approaches rV. When c exceeds rV, p* is above one and the right policy is to contact nobody, whatever the model says.

<ThresholdExplorer
  id="ch3-threshold"
  items="from-probabilities-to-actions/items"
  question="dissatisfied"
  label="dissatisfied"
  unit="reviews"
  engine="jev"
  costs={{ c: 6, V: 120, r: 0.4 }}
  presets={{ "cheap-vouchers": { c: 3 }, "expensive-vouchers": { c: 25 }, "prohibitive": { c: 60 }, "recalibrated": { recalibrated: true } }}
/>

Now score each policy. The chart below sets every way of choosing whom to contact against the simplest one, contacting nobody, at the default costs and the cutoff 0.125.

Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier

Jev's probabilities earn about $5,100 per thousand reviews and the classifier's about $5,000, both well above the keyword rule and contacting everyone. Perfect foresight, a policy that knew every rating in advance, sets the ceiling at about $6,000.

<Callout kind="null">

The difference between Jev's raw probabilities and the classifier is +$61 (95% CI −$47 to $159), an interval that includes zero, so this comparison shows no clear advantage for either. Luna and the Haiku adapter do better against the classifier, by +$180 (95% CI $87 to $277) and +$128 (95% CI $31 to $226). The classifier was trained on reviews of other coffee products; a firm would train its own on its own rated reviews, and collecting those labels and maintaining the model are costs these figures leave out.

</Callout>

<TryThis preset="expensive-vouchers" target="ch3-threshold">

Raise the cost per contact to $25. The cutoff moves to 0.521. Which reviews drop off the contact list first, and does the ranking of engines change?

</TryThis>

<TryThis preset="prohibitive" target="ch3-threshold">

Set the contact cost to $60, more than the $48 a contact is expected to recover. No cutoff below one makes contacting pay, so the best policy contacts nobody.

</TryThis>

### The rule assumes calibrated probabilities

The formula for p* takes a probability at face value: a review given 0.125 should come from a group in which about that share of reviewers are unhappy. The engines' probabilities do not mean that here. Among reviews to which Jev gives a probability between 0.125 and 0.3, just above the cutoff, only 4.2% have one or two stars, and 85% have four or five. Jev reads doubt into reviews whose writers still rated the coffee well.

Calibration measures this gap. The engines start well behind the classifier, and recalibrating them on part of the sample, then scoring on the rest, closes the gap. Here the ranking edge turns into money: at the same cutoff, recalibrated Jev earns +$259 (95% CI $189 to $333) more than raw Jev.

Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated

In practice the gap means extra contacts. At p*, Jev contacts 287 reviewers per thousand reviews and the classifier 222, and most of the difference is satisfied reviewers.

<ReliabilityDiagram
  id="ch3-reliability"
  items="from-probabilities-to-actions/items"
  label="dissatisfied"
  engine="jev"
  presets={{ "luna": { engine: "luna" }, "jev-recalibrated": { engine: "jev", recalibrated: true } }}
/>

<TryThis preset="jev-recalibrated" target="ch3-reliability">

Switch on recalibration. The points move toward the diagonal, and the probabilities near p* fall.

</TryThis>

A second fix leaves the probabilities alone and moves the cutoff instead. Choose the cutoff that earns the most on four folds of the sample, apply it to the fifth, and repeat. For Jev the data push the cutoff from 0.125 up to 0.38.

Table 3. Letting the data choose the cutoff moves it above p* for the engines

Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier

How many labels does that fix take, compared with training a classifier on the same labels? Each draw below gives both arms identical labeled reviews.

Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried

With two hundred labels, recalibrated Jev earns $5,300 and the classifier $3,300. With a thousand, $5,300 and $4,500. At the full sample the gap is still $5,300 against $5,000. On this reading task, a few hundred labels spent checking the endpoint went further than the same labels spent fitting this classifier. These figures leave out what each call and each model costs to run, and they describe one simple classifier and one task. The money-back question in a later chapter goes the other way.

The lesson for a manager: a model that ranks reviews well can still be a poor input to a cost formula. Check its probabilities against the population it will be used on, or let the data set the cutoff. Either fix needs labeled reviews from that population, and these figures do not charge for them.

The core lesson ends here. The next two sections are optional extensions: adding a person as a third action, and asking who actually responds to a contact.

<InstructorNote>

Run the cheap-voucher preset, a contact cost of $3 with a cutoff of 0.062, and the expensive one, and ask why the same model produces different contact lists. Then ask students to derive the gap between the shortcut and p* themselves, and to say what happens to p* as the contact cost approaches the expected recovery.

</InstructorNote>

### Going further: send the review to a person

Between contacting and ignoring there is a third option. A person on the care team can read the review for $1.50 and decide, and the panel assumes that person gets it right with probability 0.9, a number no one measured here. With two cutoffs, the policy contacts reviewers above the upper one, ignores reviewers below the lower one and sends the rest to a person. Reviewer capacity limits how many reviews can go to people, and once it binds, the reviews worth reading are the ones whose expected loss falls most from a person's judgment.

<TriageBand
  id="ch3-triage"
  mode="two"
  items="from-probabilities-to-actions/items"
  engine="jev"
  label="dissatisfied"
  unit="reviews"
  costs={{ c: 6, V: 120, r: 0.4, h: 1.5, a: 0.9 }}
  tLow={0.1}
  tHigh={0.6}
  capacityPer1000={60}
  presets={{ "tight-capacity": { capacityPer1000: 15 } }}
/>

For one review at a time, the comparison is a table of expected losses, one row per action. The action with the lowest expected loss is the one to take.

<ActionLossTable
  id="ch3-action"
  kind="binary"
  items="from-probabilities-to-actions/items"
  engine="jev"
  label="dissatisfied"
  unit="reviews"
  costs={{ c: 6, V: 120, r: 0.4, h: 1.5, a: 0.9 }}
  presets={{ "costly-review": { costs: { h: 8 } } }}
/>

### Going further: dissatisfied is not the same as responsive

The recovery rate r describes how much a contact changes a customer's chance of staying, and star ratings cannot measure it. An unhappy customer may leave whatever the firm does, and another may return without any voucher. Contacting the most dissatisfied reviewers is the right ranking when a contact helps them at least as much as it helps anyone else at the same cost, an assumption a review label cannot check. A randomized contact experiment would provide a direct way to estimate that incremental effect.

The simulator below is a seeded simulation, not data from any firm. In it, a contact changes retention most for moderately dissatisfied customers who buy often and close to zero for the most dissatisfied. At the same contact budget, it compares targeting by the probability of dissatisfaction with targeting by the response estimated from a simulated experiment.

<UpliftSimulator
  id="ch3-uplift"
  data="from-probabilities-to-actions/uplift"
  budgetShare={0.2}
  bins={5}
  c={6}
  V={120}
  presets={{ "small-budget": { budgetShare: 0.08 } }}
/>

<TryThis preset="small-budget" target="ch3-uplift">

Cut the contact budget. Which customers does each targeting rule keep, and which one loses more of its retained customers?

</TryThis>

A cutoff applied once to each review also ignores time. A contact changes what the firm learns next, because a reply or a repeat purchase is new evidence about that customer. Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) returns to decisions where each action changes the evidence for the next one.

<Discuss>

Finance owns the contact cost, the customer value and the recovery rate. What would make you recheck the cutoff: a change in voucher cost, a new month of reviews whose calibration has drifted, or an experiment showing that the most dissatisfied customers do not respond?

</Discuss>

### What these data cannot show

The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.

## Tables

### Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier

Net value of contacts per thousand reviews against contacting nobody, at the cost cutoff p*, by policy, weighted

| conf_high | conf_low | estimate | group | n | policy |
| ---: | ---: | ---: | --- | ---: | --- |
| n/a | n/a | 5978 | Ceiling | 5,000 | Perfect foresight |
| 5253 | 5135 | 5195 | Decision engine | 5,000 | OpenAI Luna |
| 5202 | 5083 | 5144 | General language model | 5,000 | Claude Haiku adapter |
| 5144 | 5004 | 5076 | Decision engine | 5,000 | Jev 1.14.0 |
| 5104 | 4928 | 5015 | Trained classifier | 5,000 | TF-IDF + logistic |
| 2309 | 1978 | 2148 | Simple rule | 5,000 | Keyword rule |
| n/a | n/a | 832 | Simple rule | 5,000 | Contact every reviewer |

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json

### Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated

Expected calibration error of the probability of dissatisfaction, as returned and after recalibration, by engine, weighted

| conf_high | conf_low | engine | estimate | n | version |
| ---: | ---: | --- | ---: | ---: | --- |
| 0.079 | 0.071 | Jev 1.14.0 | 0.075 | 5,000 | As returned |
| 0.006 | 0.001 | Jev 1.14.0 | 0.002 | 5,000 | Recalibrated |
| 0.064 | 0.056 | OpenAI Luna | 0.060 | 5,000 | As returned |
| 0.006 | 0.001 | OpenAI Luna | 0.002 | 5,000 | Recalibrated |
| 0.073 | 0.066 | Claude Haiku adapter | 0.069 | 5,000 | As returned |
| 0.016 | 0.007 | TF-IDF + logistic | 0.011 | 5,000 | As returned |

Note: The figure recalibrates only the two engines that return a probability distribution.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json

### Table 3. Letting the data choose the cutoff moves it above p* for the engines

Cutoff that maximizes net value, chosen on four folds of the sample and applied to the fifth, median of five folds, by model

| Model | Median cutoff | Lowest fold | Highest fold |
| --- | ---: | ---: | ---: |
| Jev 1.14.0 | 0.38 | 0.35 | 0.38 |
| OpenAI Luna | 0.22 | 0.20 | 0.29 |
| TF-IDF + logistic | 0.11 | 0.09 | 0.16 |

Note: Each dot is the median of the five fold cutoffs; the table gives the lowest and highest. No bootstrap interval was computed.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json

### Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier

Net value per thousand reviews at the cost cutoff p* and at the data-chosen cutoff, by model, weighted

| at_data | at_formula | engine | formula_hi | formula_lo | n |
| ---: | ---: | --- | ---: | ---: | ---: |
| 5335 | 5076 | Jev 1.14.0 | 5144 | 5004 | 5,000 |
| 5273 | 5195 | OpenAI Luna | 5253 | 5135 | 5,000 |
| 4985 | 5015 | TF-IDF + logistic | 5104 | 4928 | 5,000 |

Note: The data-chosen value is scored out of fold and carries no bootstrap interval.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json

### Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried

Net value of the contact policy per thousand reviews against local labels used, weighted; whiskers run from the lowest to the highest draw

| conf_high | conf_low | labels | model | n_labels | net |
| ---: | ---: | --- | --- | ---: | ---: |
| 2475 | 0 | 50 labels | Classifier, fit and calibrated on these labels | 50 | 1502 |
| 5337 | 3387 | 50 labels | Jev, recalibrated on these labels | 50 | 5058 |
| 3709 | 1570 | 100 labels | Classifier, fit and calibrated on these labels | 100 | 2981 |
| 5329 | 5029 | 100 labels | Jev, recalibrated on these labels | 100 | 5188 |
| 3885 | 1858 | 200 labels | Classifier, fit and calibrated on these labels | 200 | 3257 |
| 5341 | 5072 | 200 labels | Jev, recalibrated on these labels | 200 | 5257 |
| 4387 | 3338 | 500 labels | Classifier, fit and calibrated on these labels | 500 | 4073 |
| 5336 | 5185 | 500 labels | Jev, recalibrated on these labels | 500 | 5285 |
| 4628 | 4386 | 1,000 labels | Classifier, fit and calibrated on these labels | 1,000 | 4503 |
| 5335 | 5248 | 1,000 labels | Jev, recalibrated on these labels | 1,000 | 5300 |
| 4683 | 4490 | 2,000 labels | Classifier, fit and calibrated on these labels | 2,000 | 4581 |
| 5337 | 5225 | 2,000 labels | Jev, recalibrated on these labels | 2,000 | 5307 |
| 4872 | 4749 | 5,000 labels | Classifier, fit and calibrated on these labels | 5,000 | 4816 |
| 5344 | 5270 | 5,000 labels | Jev, recalibrated on these labels | 5,000 | 5320 |
| 4974 | 4897 | 10,000 labels | Classifier, fit and calibrated on these labels | 10,000 | 4939 |
| 5342 | 5304 | 10,000 labels | Jev, recalibrated on these labels | 10,000 | 5333 |
| 5045 | 5008 | 20,000 labels | Classifier, fit and calibrated on these labels | 20,000 | 5032 |
| n/a | n/a | 20,000 labels | Jev, recalibrated on these labels | 20,000 | 5342 |

Note: Platt scaling in place of isotonic calibration gives the same reading.

Source: Amazon Reviews 2023, coffee products; labels from reviews of other products; policy scored on the sampled evaluation reviews, weighted; engine calls October 2026; assumed costs

Data: https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json

## All estimates

Every fact in facts.json (36), sorted by id.

| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| band_hi | 0.3 | probability of dissatisfaction | upper edge of the band just above the cutoff | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| band_rate_jev | 4.2% | share of reviews with one or two stars | among reviews given P from p* up to 0.3 | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   | 427 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| band_share_high_stars | 84.8% | percent of reviews in the band | four- and five-star reviews among those given P from p* up to 0.3 by Jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   | 427 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| contact_cost | $6 | US dollars per contact | assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| contact_cost_cheap | $3 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| contact_cost_expensive | $25 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| contact_cost_prohibitive | $60 | US dollars per contact | assumed cost, preset | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| contacts_jev | 287 | contacts per 1,000 reviews | weighted contacts at p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| contacts_tfidf | 222 | contacts per 1,000 reviews | weighted contacts at p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| customer_value | $120 | US dollars per retained customer | assumed value | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| cut_emp_jev | 0.38 | probability of dissatisfaction | median of five cross-fit cost-minimizing cutoffs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Range across folds 0.35 to 0.38. |
| expected_recovery | $48 | US dollars per contact with a dissatisfied customer | r times V | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| lv_clf_1000 | $4,503 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_200 | $3,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_full | $5,032 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_1000 | $5,300 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_200 | $5,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_full | $5,342 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs. Both arms use the same labeled draws. |
| n_reviews | 5,000 | reviews | sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_diff_adapter_tfidf | +$128 | US dollars per 1,000 reviews | difference in net value, adapter minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $31 to $226 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_diff_jev_recal_jev | +$259 | US dollars per 1,000 reviews | difference in net value, jev_recal minus jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $189 to $333 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_diff_jev_recal_tfidf | +$320 | US dollars per 1,000 reviews | difference in net value, jev_recal minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $228 to $411 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_diff_jev_tfidf | +$61 | US dollars per 1,000 reviews | difference in net value, jev minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI −$47 to $159 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_diff_luna_tfidf | +$180 | US dollars per 1,000 reviews | difference in net value, luna minus tfidf | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $87 to $277 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |
| net_jev | $5,076 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_jev_recal | $5,335 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_oracle | $5,978 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| net_tfidf | $5,015 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | Assumed costs; measured error mix. |
| p_shortcut | 0.111 | probability of dissatisfaction | shortcut cutoff C_FP / (C_FP + C_FN) with C_FN = rV | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| p_star | 0.125 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| p_star_cheap | 0.062 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| p_star_expensive | 0.521 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| recovery_rate | 0.4 | share of dissatisfied customers a contact retains | assumed rate | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| review_accuracy | 0.9 | share of reviews a person judges correctly | assumed reviewer accuracy | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| review_cost | $1.50 | US dollars per human review | assumed cost | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 |   |
| share_dissatisfied | 14.2% | share of reviews with one or two stars | weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 |   |

## Definitions

- Calibration: Whether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive.

## Methods

Two decision engines, a general language model and a trained classifier gave each sampled review a probability of dissatisfaction. Policies contact a reviewer when that probability reaches a cutoff, and each policy is scored on the star labels with assumed costs.

### What counts as dissatisfied

A review counts as dissatisfied when its star rating is one or two. The rating is the reviewer's own, so it is a proxy that exists without any rater. A few low ratings complain about delivery rather than the coffee, and a few high ratings contain complaints.

### Engines and the classifier

Jev 1.14.0 and OpenAI Luna return a probability for a yes-or-no question. The Claude Haiku adapter returns a prompted estimate of that probability. The TF-IDF classifier was trained on reviews of other coffee products and calibrated on a held-out part of them.

Each engine was called once per review in October 2026 and the answers were stored.

### Weights, folds and intervals

The sample holds the same number of reviews for each star rating, so every share, cost and calibration curve is weighted back to the star mix of the coffee reviews. Recalibrated probabilities and data-chosen cutoffs are cross-fit over five folds that keep each product's reviews together. Intervals come from a bootstrap that resamples products within star strata, with the same resamples for every policy.

## What these data cannot show

The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.

## References

The chapter cites no reference.

## Provenance

- Build script: decision_ai/chapters/from-probabilities-to-actions/build.py (sha256 58dca1335409b82c81c737aef7f9af14d02d25c1e5e6b9be77888ea624e2ecb3)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs:

| File | SHA-256 |
| --- | --- |
| facts.json | 069f32998b43fab552b79437f4492e543998297912ab27ae03e3fa276ff03457 |
| figures/calibration-error.json | 8006eba4847d93ba8a4f148eda7dae82b7b4a3a61016a0fe2a07d35796655bbc |
| figures/data-chosen-cutoffs.json | a6cb47cab02f1bf4081668fe1e971c9faa0b6e4f01c3da0d37e0ee0b36285e6a |
| figures/net-at-data-cutoffs.json | a9a0d2a997bf6d8a12836ab74821ebf5b9e7b0cf2dc70504d2cfa0c044e68f6f |
| figures/net-value-by-policy.json | 4f68a040c412cecaa041de32fd28cf85298f3413336cf4f64a6d8eb9acd6cc33 |
| figures/value-by-labels.json | 072e9978e513582a94013393a3d5a7717008bdd756bff21168e87e86c23f2e59 |
| meta.json | 08b7e478643cd97028ee57ddf47f2196dfb9ee8af303f5c8d85caca83910ad8b |
| public/data/from-probabilities-to-actions/items.json | 508445f13140a53a1db0af4f7562cf285dcda90e0a2f0ec9c5a38fef7edebbf9 |
| public/data/from-probabilities-to-actions/uplift.json | bd0966a9319584eb784730b70bc7056f7d91e54b3cc0cf4f18c0da4077d29454 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |

## Machine formats

- Report (HTML): https://vishalsingh.org/decisionai/from-probabilities-to-actions/report
- Report (Markdown): https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/facts.json
- Table 1, figure "net-value-by-policy" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json
- Table 2, figure "calibration-error" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json
- Table 3, figure "data-chosen-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json
- Table 4, figure "net-at-data-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json
- Table 5, figure "value-by-labels" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt