From probabilities to actions: report

Identification

Title

From probabilities to actions

Chapter

1.3 (Part I, From judgment to action; Case studies)

Question

Given the cost of a contact and the value of a recovered customer, which dissatisfied reviewers should a coffee roaster contact, and what does that rule assume?

Dek

At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.

Grain

records

Status

draft

Published

29 October 2026

Updated

not revised

Data sources

Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating; weights: sample_weight restores the star mix of the coffee reviews

Canonical URL

https://vishalsingh.org/decisionai/from-probabilities-to-actions

Markdown

https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md

BibTeX

@misc{singh2026fromprobabilitiestoactions,
  author = {Singh, Vishal},
  title = {From probabilities to actions},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/from-probabilities-to-actions},
  note = {In Decision AI. Published 29 October 2026.}
}

Principal estimates

The facts the dek and the opening prose cite, in order of appearance. Every fact is in the table of all estimates below.

Estimand	Population	Period	Value	Interval	n	Weights	Source	Fact id
assumed cost	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026	$6			not applicable; assumed inputs	Amazon Reviews 2023	contact_cost
r times V	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026	$48			not applicable; assumed inputs	Amazon Reviews 2023	expected_recovery
net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$5,076	95% CI $5,004 to $5,144	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	net_jev
net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$5,015	95% CI $4,928 to $5,104	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	net_tfidf
net value per 1,000 reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	$5,335	95% CI $5,273 to $5,393	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	net_jev_recal
assumed value	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026	$120			not applicable; assumed inputs	Amazon Reviews 2023	customer_value
assumed rate	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026	0.4			not applicable; assumed inputs	Amazon Reviews 2023	recovery_rate
sample size	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	5,000			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	n_reviews
weighted share	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	14.2%			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	share_dissatisfied
difference in net value, jev_recal minus tfidf	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	+$320	95% CI $228 to $411	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	net_diff_jev_recal_tfidf

Findings

Bean & Basket, a coffee roaster that sells bagged coffee online, wants its care team to contact unhappy reviewers with a replacement bag or a voucher. The firm is illustrative, and the reviews come from many coffee products sold on Amazon. Suppose a contact costs $6 in staff time and voucher value, a retained customer is worth $120, and a contact wins back a dissatisfied customer with probability 0.4. A contact with a dissatisfied reviewer is then worth $48 on average, and a contact with a satisfied reviewer is wasted. Each of 5,000 sampled coffee reviews has a model's probability that the reviewer is dissatisfied, and weighted to all coffee reviews, 14.2% of reviewers gave one or two stars. Which reviewers should the team contact?

At these assumed costs, Jev's raw probabilities and a trained classifier earn about the same, $5,100 and $5,000 per thousand reviews, because Jev's probabilities run too high near the cutoff. Recalibrated on labeled reviews, Jev earns about $5,300, ahead of the classifier by +$320 (95% CI $228 to $411). So for this policy and these costs, the ranking edge paid off once labels went into the probabilities. Stars stand in for dissatisfaction, the costs are assumptions, and dissatisfaction alone does not tell you who will respond to a voucher.

The decision

Owner and actions: The customer-care lead decides whom to contact; finance owns the costs.
Evidence at decision time: The review's title and text, available the day it is posted.
Simplest alternative: A keyword rule, or a classifier trained on reviews of other coffee products.
What establishes correctness: The reviewer's own star rating, used as a proxy: one or two stars counts as dissatisfied.
When the model is wrong or cannot decide: A wasted contact costs the voucher; a missed unhappy customer may not buy again.
Does the policy pay: Net value per thousand reviews under the stated costs, compared across policies.

The cutoff comes from the costs

Contacting a reviewer pays when the expected recovery exceeds the cost: when the probability of dissatisfaction p satisfies p times rV greater than c. The cutoff is

p^{*} = \frac{c}{rV}

which is 0.125 at these costs. A cutoff this low means the team should contact many reviewers who are probably satisfied, because a contact is cheap relative to what a recovered customer is worth.

The shortcut gives 0.111, a little lower than 0.125. It assumes that a correct decision costs nothing, and here a correct contact still costs c. The difference is small at these costs and grows as c approaches rV. When c exceeds rV, p* is above one and the right policy is to contact nobody, whatever the model says.

Threshold explorer

Expected value of contacting at each cutoff

Loading data

Now score each policy. The chart below sets every way of choosing whom to contact against the simplest one, contacting nobody, at the default costs and the cutoff 0.125.

Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier

Jev's probabilities earn about $5,100 per thousand reviews and the classifier's about $5,000, both well above the keyword rule and contacting everyone. Perfect foresight, a policy that knew every rating in advance, sets the ceiling at about $6,000.

No difference found

The difference between Jev's raw probabilities and the classifier is +$61 (95% CI −$47 to $159), an interval that includes zero, so this comparison shows no clear advantage for either. Luna and the Haiku adapter do better against the classifier, by +$180 (95% CI $87 to $277) and +$128 (95% CI $31 to $226). The classifier was trained on reviews of other coffee products; a firm would train its own on its own rated reviews, and collecting those labels and maintaining the model are costs these figures leave out.

The rule assumes calibrated probabilities

The formula for p* takes a probability at face value: a review given 0.125 should come from a group in which about that share of reviewers are unhappy. The engines' probabilities do not mean that here. Among reviews to which Jev gives a probability between 0.125 and 0.3, just above the cutoff, only 4.2% have one or two stars, and 85% have four or five. Jev reads doubt into reviews whose writers still rated the coffee well.

Calibration measures this gap. The engines start well behind the classifier, and recalibrating them on part of the sample, then scoring on the rest, closes the gap. Here the ranking edge turns into money: at the same cutoff, recalibrated Jev earns +$259 (95% CI $189 to $333) more than raw Jev.

Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated

In practice the gap means extra contacts. At p*, Jev contacts 287 reviewers per thousand reviews and the classifier 222, and most of the difference is satisfied reviewers.

Reliability diagram

Predicted probability against observed share

Loading data

A second fix leaves the probabilities alone and moves the cutoff instead. Choose the cutoff that earns the most on four folds of the sample, apply it to the fifth, and repeat. For Jev the data push the cutoff from 0.125 up to 0.38.

Table 3. Letting the data choose the cutoff moves it above p* for the engines

Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier

How many labels does that fix take, compared with training a classifier on the same labels? Each draw below gives both arms identical labeled reviews.

Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried

With two hundred labels, recalibrated Jev earns $5,300 and the classifier $3,300. With a thousand, $5,300 and $4,500. At the full sample the gap is still $5,300 against $5,000. On this reading task, a few hundred labels spent checking the endpoint went further than the same labels spent fitting this classifier. These figures leave out what each call and each model costs to run, and they describe one simple classifier and one task. The money-back question in a later chapter goes the other way.

The lesson for a manager: a model that ranks reviews well can still be a poor input to a cost formula. Check its probabilities against the population it will be used on, or let the data set the cutoff. Either fix needs labeled reviews from that population, and these figures do not charge for them.

The core lesson ends here. The next two sections are optional extensions: adding a person as a third action, and asking who actually responds to a contact.

Going further: send the review to a person

Between contacting and ignoring there is a third option. A person on the care team can read the review for $1.50 and decide, and the panel assumes that person gets it right with probability 0.9, a number no one measured here. With two cutoffs, the policy contacts reviewers above the upper one, ignores reviewers below the lower one and sends the rest to a person. Reviewer capacity limits how many reviews can go to people, and once it binds, the reviews worth reading are the ones whose expected loss falls most from a person's judgment.

Triage band

Automate, review or triage

Loading data

For one review at a time, the comparison is a table of expected losses, one row per action. The action with the lowest expected loss is the one to take.

Action loss table

Expected loss of each action

Loading data

Going further: dissatisfied is not the same as responsive

The recovery rate r describes how much a contact changes a customer's chance of staying, and star ratings cannot measure it. An unhappy customer may leave whatever the firm does, and another may return without any voucher. Contacting the most dissatisfied reviewers is the right ranking when a contact helps them at least as much as it helps anyone else at the same cost, an assumption a review label cannot check. A randomized contact experiment would provide a direct way to estimate that incremental effect.

The simulator below is a seeded simulation, not data from any firm. In it, a contact changes retention most for moderately dissatisfied customers who buy often and close to zero for the most dissatisfied. At the same contact budget, it compares targeting by the probability of dissatisfaction with targeting by the response estimated from a simulated experiment.

Uplift simulatorsynthetic

Targeting by dissatisfaction or by estimated uplift

Loading data

A cutoff applied once to each review also ignores time. A contact changes what the firm learns next, because a reply or a repeat purchase is new evidence about that customer. Chapter 3.1, Deciding when an agent is finished returns to decisions where each action changes the evidence for the next one.

What these data cannot show

The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.

Tables

Table 1. Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier

Net value of contacts per thousand reviews against contacting nobody, at the cost cutoff p*, by policy, weighted

conf_high	conf_low	estimate	group	n	policy
n/a	n/a	5978	Ceiling	5,000	Perfect foresight
5253	5135	5195	Decision engine	5,000	OpenAI Luna
5202	5083	5144	General language model	5,000	Claude Haiku adapter
5144	5004	5076	Decision engine	5,000	Jev 1.14.0
5104	4928	5015	Trained classifier	5,000	TF-IDF + logistic
2309	1978	2148	Simple rule	5,000	Keyword rule
n/a	n/a	832	Simple rule	5,000	Contact every reviewer

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json

Table 2. The engines' probabilities are off by more than the classifier's until they are recalibrated

Expected calibration error of the probability of dissatisfaction, as returned and after recalibration, by engine, weighted

conf_high	conf_low	engine	estimate	n	version
0.079	0.071	Jev 1.14.0	0.075	5,000	As returned
0.006	0.001	Jev 1.14.0	0.002	5,000	Recalibrated
0.064	0.056	OpenAI Luna	0.060	5,000	As returned
0.006	0.001	OpenAI Luna	0.002	5,000	Recalibrated
0.073	0.066	Claude Haiku adapter	0.069	5,000	As returned
0.016	0.007	TF-IDF + logistic	0.011	5,000	As returned

Note: The figure recalibrates only the two engines that return a probability distribution.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json

Table 3. Letting the data choose the cutoff moves it above p* for the engines

Cutoff that maximizes net value, chosen on four folds of the sample and applied to the fifth, median of five folds, by model

Model	Median cutoff	Lowest fold	Highest fold
Jev 1.14.0	0.38	0.35	0.38
OpenAI Luna	0.22	0.20	0.29
TF-IDF + logistic	0.11	0.09	0.16

Note: Each dot is the median of the five fold cutoffs; the table gives the lowest and highest. No bootstrap interval was computed.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json

Table 4. A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier

Net value per thousand reviews at the cost cutoff p* and at the data-chosen cutoff, by model, weighted

at_data	at_formula	engine	formula_hi	formula_lo	n
5335	5076	Jev 1.14.0	5144	5004	5,000
5273	5195	OpenAI Luna	5253	5135	5,000
4985	5015	TF-IDF + logistic	5104	4928	5,000

Note: The data-chosen value is scored out of fold and carries no bootstrap interval.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json

Table 5. For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried

Net value of the contact policy per thousand reviews against local labels used, weighted; whiskers run from the lowest to the highest draw

conf_high	conf_low	labels	model	n_labels	net
2475	0	50 labels	Classifier, fit and calibrated on these labels	50	1502
5337	3387	50 labels	Jev, recalibrated on these labels	50	5058
3709	1570	100 labels	Classifier, fit and calibrated on these labels	100	2981
5329	5029	100 labels	Jev, recalibrated on these labels	100	5188
3885	1858	200 labels	Classifier, fit and calibrated on these labels	200	3257
5341	5072	200 labels	Jev, recalibrated on these labels	200	5257
4387	3338	500 labels	Classifier, fit and calibrated on these labels	500	4073
5336	5185	500 labels	Jev, recalibrated on these labels	500	5285
4628	4386	1,000 labels	Classifier, fit and calibrated on these labels	1,000	4503
5335	5248	1,000 labels	Jev, recalibrated on these labels	1,000	5300
4683	4490	2,000 labels	Classifier, fit and calibrated on these labels	2,000	4581
5337	5225	2,000 labels	Jev, recalibrated on these labels	2,000	5307
4872	4749	5,000 labels	Classifier, fit and calibrated on these labels	5,000	4816
5344	5270	5,000 labels	Jev, recalibrated on these labels	5,000	5320
4974	4897	10,000 labels	Classifier, fit and calibrated on these labels	10,000	4939
5342	5304	10,000 labels	Jev, recalibrated on these labels	10,000	5333
5045	5008	20,000 labels	Classifier, fit and calibrated on these labels	20,000	5032
n/a	n/a	20,000 labels	Jev, recalibrated on these labels	20,000	5342

Note: Platt scaling in place of isotonic calibration gives the same reading.

Source: Amazon Reviews 2023, coffee products; labels from reviews of other products; policy scored on the sampled evaluation reviews, weighted; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json

All estimates

Every fact in facts.json (36), sorted by id. The same values as JSON: facts.json.

Fact id	Value	Unit	Estimand	Population	Period	Interval	n	Weights	Source	Note
band_hi	0.3	probability of dissatisfaction	upper edge of the band just above the cutoff	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
band_rate_jev	4.2%	share of reviews with one or two stars	among reviews given P from p* up to 0.3	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		427 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
band_share_high_stars	84.8%	percent of reviews in the band	four- and five-star reviews among those given P from p* up to 0.3 by Jev	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		427 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
contact_cost	$6	US dollars per contact	assumed cost	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
contact_cost_cheap	$3	US dollars per contact	assumed cost, preset	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
contact_cost_expensive	$25	US dollars per contact	assumed cost, preset	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
contact_cost_prohibitive	$60	US dollars per contact	assumed cost, preset	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
contacts_jev	287	contacts per 1,000 reviews	weighted contacts at p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
contacts_tfidf	222	contacts per 1,000 reviews	weighted contacts at p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
customer_value	$120	US dollars per retained customer	assumed value	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
cut_emp_jev	0.38	probability of dissatisfaction	median of five cross-fit cost-minimizing cutoffs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Range across folds 0.35 to 0.38.
expected_recovery	$48	US dollars per contact with a dissatisfied customer	r times V	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
lv_clf_1000	$4,503	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
lv_clf_200	$3,257	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
lv_clf_full	$5,032	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
lv_jev_1000	$5,300	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
lv_jev_200	$5,257	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
lv_jev_full	$5,342	US dollars per thousand reviews	net value at p*, mean across label draws	the coffee reviews, labels drawn from reviews of other products	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs. Both arms use the same labeled draws.
n_reviews	5,000	reviews	sample size	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_diff_adapter_tfidf	+$128	US dollars per 1,000 reviews	difference in net value, adapter minus tfidf	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $31 to $226	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_diff_jev_recal_jev	+$259	US dollars per 1,000 reviews	difference in net value, jev_recal minus jev	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $189 to $333	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_diff_jev_recal_tfidf	+$320	US dollars per 1,000 reviews	difference in net value, jev_recal minus tfidf	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $228 to $411	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_diff_jev_tfidf	+$61	US dollars per 1,000 reviews	difference in net value, jev minus tfidf	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI −$47 to $159	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_diff_luna_tfidf	+$180	US dollars per 1,000 reviews	difference in net value, luna minus tfidf	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $87 to $277	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
net_jev	$5,076	US dollars per 1,000 reviews	net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $5,004 to $5,144	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs; measured error mix.
net_jev_recal	$5,335	US dollars per 1,000 reviews	net value per 1,000 reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $5,273 to $5,393	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs; measured error mix.
net_oracle	$5,978	US dollars per 1,000 reviews	net value per 1,000 reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs; measured error mix.
net_tfidf	$5,015	US dollars per 1,000 reviews	net value of contacts against contacting nobody, at the cutoff p*	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI $4,928 to $5,104	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	Assumed costs; measured error mix.
p_shortcut	0.111	probability of dissatisfaction	shortcut cutoff C_FP / (C_FP + C_FN) with C_FN = rV	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
p_star	0.125	probability of dissatisfaction	cutoff c / (rV)	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
p_star_cheap	0.062	probability of dissatisfaction	cutoff c / (rV)	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
p_star_expensive	0.521	probability of dissatisfaction	cutoff c / (rV)	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
recovery_rate	0.4	share of dissatisfied customers a contact retains	assumed rate	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
review_accuracy	0.9	share of reviews a person judges correctly	assumed reviewer accuracy	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
review_cost	$1.50	US dollars per human review	assumed cost	the coffee roaster's contact decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	Amazon Reviews 2023
share_dissatisfied	14.2%	share of reviews with one or two stars	weighted share	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023

Definitions

Calibration: Whether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive.

Methods

Two decision engines, a general language model and a trained classifier gave each sampled review a probability of dissatisfaction. Policies contact a reviewer when that probability reaches a cutoff, and each policy is scored on the star labels with assumed costs.

What counts as dissatisfied

A review counts as dissatisfied when its star rating is one or two. The rating is the reviewer's own, so it is a proxy that exists without any rater. A few low ratings complain about delivery rather than the coffee, and a few high ratings contain complaints.

Engines and the classifier

Jev 1.14.0 and OpenAI Luna return a probability for a yes-or-no question. The Claude Haiku adapter returns a prompted estimate of that probability. The TF-IDF classifier was trained on reviews of other coffee products and calibrated on a held-out part of them.

Each engine was called once per review in October 2026 and the answers were stored.

Weights, folds and intervals

The sample holds the same number of reviews for each star rating, so every share, cost and calibration curve is weighted back to the star mix of the coffee reviews. Recalibrated probabilities and data-chosen cutoffs are cross-fit over five folds that keep each product's reviews together. Intervals come from a bootstrap that resamples products within star strata, with the same resamples for every policy.

What these data cannot show

The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.

References

The chapter cites no reference.

Provenance

Build script: decision_ai/chapters/from-probabilities-to-actions/build.py (sha256 58dca1335409b82c81c737aef7f9af14d02d25c1e5e6b9be77888ea624e2ecb3)
Built on: 2026-10-29
Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs

File	SHA-256
facts.json	069f32998b43fab552b79437f4492e543998297912ab27ae03e3fa276ff03457
figures/calibration-error.json	8006eba4847d93ba8a4f148eda7dae82b7b4a3a61016a0fe2a07d35796655bbc
figures/data-chosen-cutoffs.json	a6cb47cab02f1bf4081668fe1e971c9faa0b6e4f01c3da0d37e0ee0b36285e6a
figures/net-at-data-cutoffs.json	a9a0d2a997bf6d8a12836ab74821ebf5b9e7b0cf2dc70504d2cfa0c044e68f6f
figures/net-value-by-policy.json	4f68a040c412cecaa041de32fd28cf85298f3413336cf4f64a6d8eb9acd6cc33
figures/value-by-labels.json	072e9978e513582a94013393a3d5a7717008bdd756bff21168e87e86c23f2e59
meta.json	08b7e478643cd97028ee57ddf47f2196dfb9ee8af303f5c8d85caca83910ad8b
public/data/from-probabilities-to-actions/items.json	508445f13140a53a1db0af4f7562cf285dcda90e0a2f0ec9c5a38fef7edebbf9
public/data/from-probabilities-to-actions/uplift.json	bd0966a9319584eb784730b70bc7056f7d91e54b3cc0cf4f18c0da4077d29454
refs.json	ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356

Machine formats

Report (Markdown): https://vishalsingh.org/decisionai/from-probabilities-to-actions/report.md
Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/facts.json
Table 1, figure "net-value-by-policy" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-value-by-policy.json
Table 2, figure "calibration-error" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/calibration-error.json
Table 3, figure "data-chosen-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/data-chosen-cutoffs.json
Table 4, figure "net-at-data-cutoffs" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/net-at-data-cutoffs.json
Table 5, figure "value-by-labels" (JSON): https://vishalsingh.org/decisionai/report/from-probabilities-to-actions/figures/value-by-labels.json
Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt