Chapter 1.3

From probabilities to actions

At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 29 October 2026

Full report: every estimate with its interval, and every figure as a table.

DraftFigures and numbers may change as the analysis is rebuilt.what is still being checked

First draft. The label is the star rating, used as a proxy for dissatisfaction, and every cost is an assumption.

Bean & Basket, a coffee roaster that sells bagged coffee online, wants its care team to contact unhappy reviewers with a replacement bag or a voucher. The firm is illustrative, and the reviews come from many coffee products sold on Amazon. Suppose a contact costs $6 in staff time and voucher value, a retained customer is worth $120, and a contact wins back a dissatisfied customer with probability 0.4. A contact with a dissatisfied reviewer is then worth $48 on average, and a contact with a satisfied reviewer is wasted. Each of 5,000 sampled coffee reviews has a model's probability that the reviewer is dissatisfied, and weighted to all coffee reviews, 14.2% of reviewers gave one or two stars. Which reviewers should the team contact?

At these assumed costs, Jev's raw probabilities and a trained classifier earn about the same, $5,100 and $5,000 per thousand reviews, because Jev's probabilities run too high near the cutoff. Recalibrated on labeled reviews, Jev earns about $5,300, ahead of the classifier by +$320 (95% CI $228 to $411). So for this policy and these costs, the ranking edge paid off once labels went into the probabilities. Stars stand in for dissatisfaction, the costs are assumptions, and dissatisfaction alone does not tell you who will respond to a voucher.

The decision

Owner and actions: The customer-care lead decides whom to contact; finance owns the costs.
Evidence at decision time: The review's title and text, available the day it is posted.
Simplest alternative: A keyword rule, or a classifier trained on reviews of other coffee products.
What establishes correctness: The reviewer's own star rating, used as a proxy: one or two stars counts as dissatisfied.
When the model is wrong or cannot decide: A wasted contact costs the voucher; a missed unhappy customer may not buy again.
Does the policy pay: Net value per thousand reviews under the stated costs, compared across policies.

The cutoff comes from the costs

Contacting a reviewer pays when the expected recovery exceeds the cost: when the probability of dissatisfaction p satisfies p times rV greater than c. The cutoff is

p^{*} = \frac{c}{rV}

which is 0.125 at these costs. A cutoff this low means the team should contact many reviewers who are probably satisfied, because a contact is cheap relative to what a recovered customer is worth.

The shortcut gives 0.111, a little lower than 0.125. It assumes that a correct decision costs nothing, and here a correct contact still costs c. The difference is small at these costs and grows as c approaches rV. When c exceeds rV, p* is above one and the right policy is to contact nobody, whatever the model says.

Threshold explorer

Expected value of contacting at each cutoff

Loading data

Now score each policy. The chart below sets every way of choosing whom to contact against the simplest one, contacting nobody, at the default costs and the cutoff 0.125.

Figure 1

Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier

Net value of contacts per thousand reviews against contacting nobody, at the cost cutoff p*, by policy, weighted

Ceiling
Decision engine
General language model
Trained classifier
Simple rule

Show data

Data for Model-based contact policies earn far more than a keyword rule or contacting everyone, and Jev's raw probabilities earn no clear premium over the classifier
conf_high	conf_low	estimate	group	n	policy
n/a	n/a	5978	Ceiling	5,000	Perfect foresight
5253	5135	5195	Decision engine	5,000	OpenAI Luna
5202	5083	5144	General language model	5,000	Claude Haiku adapter
5144	5004	5076	Decision engine	5,000	Jev 1.14.0
5104	4928	5015	Trained classifier	5,000	TF-IDF + logistic
2309	1978	2148	Simple rule	5,000	Keyword rule
n/a	n/a	832	Simple rule	5,000	Contact every reviewer

The four model-based dots sit a little below the ceiling set by perfect foresight, with Luna and the adapter ahead of Jev and the classifier. The keyword rule and contacting every reviewer earn much less. Each dot combines the measured mix of right and wrong contacts with the assumed costs ($6 per contact, $48 expected recovery). Whiskers are 95% intervals from a bootstrap that resamples products within star ratings; the ceiling and contacting everyone are fixed by the star mix.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Jev's probabilities earn about $5,100 per thousand reviews and the classifier's about $5,000, both well above the keyword rule and contacting everyone. Perfect foresight, a policy that knew every rating in advance, sets the ceiling at about $6,000.

No difference found

The difference between Jev's raw probabilities and the classifier is +$61 (95% CI −$47 to $159), an interval that includes zero, so this comparison shows no clear advantage for either. Luna and the Haiku adapter do better against the classifier, by +$180 (95% CI $87 to $277) and +$128 (95% CI $31 to $226). The classifier was trained on reviews of other coffee products; a firm would train its own on its own rated reviews, and collecting those labels and maintaining the model are costs these figures leave out.

The rule assumes calibrated probabilities

The formula for p* takes a probability at face value: a review given 0.125 should come from a group in which about that share of reviewers are unhappy. The engines' probabilities do not mean that here. Among reviews to which Jev gives a probability between 0.125 and 0.3, just above the cutoff, only 4.2% have one or two stars, and 85% have four or five. Jev reads doubt into reviews whose writers still rated the coffee well.

Calibration measures this gap. The engines start well behind the classifier, and recalibrating them on part of the sample, then scoring on the rest, closes the gap. Here the ranking edge turns into money: at the same cutoff, recalibrated Jev earns +$259 (95% CI $189 to $333) more than raw Jev.

Figure 2

The engines' probabilities are off by more than the classifier's until they are recalibrated

Expected calibration error of the probability of dissatisfaction, as returned and after recalibration, by engine, weighted

As returned
Recalibrated

Show data

Data for The engines' probabilities are off by more than the classifier's until they are recalibrated
conf_high	conf_low	engine	estimate	n	version
0.079	0.071	Jev 1.14.0	0.075	5,000	As returned
0.006	0.001	Jev 1.14.0	0.002	5,000	Recalibrated
0.064	0.056	OpenAI Luna	0.060	5,000	As returned
0.006	0.001	OpenAI Luna	0.002	5,000	Recalibrated
0.073	0.066	Claude Haiku adapter	0.069	5,000	As returned
0.016	0.007	TF-IDF + logistic	0.011	5,000	As returned

Compare each engine's two dots: as returned, Jev, Luna and the adapter sit well right of the classifier; after recalibration on other folds of the sample, Jev and Luna fall close to zero. Calibration error is the weighted gap between the stated probability and the observed share with one or two stars. Whiskers are 95% intervals from a bootstrap that resamples products.Note: The figure recalibrates only the two engines that return a probability distribution.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

In practice the gap means extra contacts. At p*, Jev contacts 287 reviewers per thousand reviews and the classifier 222, and most of the difference is satisfied reviewers.

Reliability diagram

Predicted probability against observed share

Loading data

A second fix leaves the probabilities alone and moves the cutoff instead. Choose the cutoff that earns the most on four folds of the sample, apply it to the fifth, and repeat. For Jev the data push the cutoff from 0.125 up to 0.38.

Figure 3

Letting the data choose the cutoff moves it above p* for the engines

Cutoff that maximizes net value, chosen on four folds of the sample and applied to the fifth, median of five folds, by model

Show data

Data for Letting the data choose the cutoff moves it above p* for the engines
Model	Median cutoff	Lowest fold	Highest fold
Jev 1.14.0	0.38	0.35	0.38
OpenAI Luna	0.22	0.20	0.29
TF-IDF + logistic	0.11	0.09	0.16

The dashed rule is the cost cutoff p* = 0.125. Dots to its right belong to models whose probabilities overstate dissatisfaction near the cutoff, so the data push the cutoff up to contact fewer satisfied reviewers.Note: Each dot is the median of the five fold cutoffs; the table gives the lowest and highest. No bootstrap interval was computed.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

Figure 4

A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier

Net value per thousand reviews at the cost cutoff p* and at the data-chosen cutoff, by model, weighted

At p*
At the data-chosen cutoff

Show data

Data for A data-chosen cutoff adds the most net value for Jev, the least calibrated engine, and nothing for the classifier
at_data	at_formula	engine	formula_hi	formula_lo	n
5335	5076	Jev 1.14.0	5144	5004	5,000
5273	5195	OpenAI Luna	5253	5135	5,000
4985	5015	TF-IDF + logistic	5104	4928	5,000

The open dot is net value at p*; the filled dot is net value at the cutoff the data chose, scored on held-out folds. The last column is the gain. Whiskers on the open dots are 95% intervals from a bootstrap that resamples products.Note: The data-chosen value is scored out of fold and carries no bootstrap interval.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026; assumed costs

How many labels does that fix take, compared with training a classifier on the same labels? Each draw below gives both arms identical labeled reviews.

Figure 5

For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried

Net value of the contact policy per thousand reviews against local labels used, weighted; whiskers run from the lowest to the highest draw

Classifier, fit and calibrated on these labels
Jev, recalibrated on these labels

Show data

Data for For the same labels, recalibrating Jev earns more than fitting a classifier at every size tried
conf_high	conf_low	labels	model	n_labels	net
2475	0	50 labels	Classifier, fit and calibrated on these labels	50	1502
5337	3387	50 labels	Jev, recalibrated on these labels	50	5058
3709	1570	100 labels	Classifier, fit and calibrated on these labels	100	2981
5329	5029	100 labels	Jev, recalibrated on these labels	100	5188
3885	1858	200 labels	Classifier, fit and calibrated on these labels	200	3257
5341	5072	200 labels	Jev, recalibrated on these labels	200	5257
4387	3338	500 labels	Classifier, fit and calibrated on these labels	500	4073
5336	5185	500 labels	Jev, recalibrated on these labels	500	5285
4628	4386	1,000 labels	Classifier, fit and calibrated on these labels	1,000	4503
5335	5248	1,000 labels	Jev, recalibrated on these labels	1,000	5300
4683	4490	2,000 labels	Classifier, fit and calibrated on these labels	2,000	4581
5337	5225	2,000 labels	Jev, recalibrated on these labels	2,000	5307
4872	4749	5,000 labels	Classifier, fit and calibrated on these labels	5,000	4816
5344	5270	5,000 labels	Jev, recalibrated on these labels	5,000	5320
4974	4897	10,000 labels	Classifier, fit and calibrated on these labels	10,000	4939
5342	5304	10,000 labels	Jev, recalibrated on these labels	10,000	5333
5045	5008	20,000 labels	Classifier, fit and calibrated on these labels	20,000	5032
n/a	n/a	20,000 labels	Jev, recalibrated on these labels	20,000	5342

Read down the rows: both arms use the same labeled reviews, and the line marks raw Jev. Each count is every local label used, including the classifier's calibration labels; the reviews being scored are not among them. Whiskers are ranges across draws and not confidence intervals. Draws differ from the previous chapter's curve, so its numbers differ a little.Note: Platt scaling in place of isotonic calibration gives the same reading.Source: Amazon Reviews 2023, coffee products; labels from reviews of other products; policy scored on the sampled evaluation reviews, weighted; engine calls October 2026; assumed costs

With two hundred labels, recalibrated Jev earns $5,300 and the classifier $3,300. With a thousand, $5,300 and $4,500. At the full sample the gap is still $5,300 against $5,000. On this reading task, a few hundred labels spent checking the endpoint went further than the same labels spent fitting this classifier. These figures leave out what each call and each model costs to run, and they describe one simple classifier and one task. The money-back question in a later chapter goes the other way.

The lesson for a manager: a model that ranks reviews well can still be a poor input to a cost formula. Check its probabilities against the population it will be used on, or let the data set the cutoff. Either fix needs labeled reviews from that population, and these figures do not charge for them.

The core lesson ends here. The next two sections are optional extensions: adding a person as a third action, and asking who actually responds to a contact.

Going further: send the review to a person

Between contacting and ignoring there is a third option. A person on the care team can read the review for $1.50 and decide, and the panel assumes that person gets it right with probability 0.9, a number no one measured here. With two cutoffs, the policy contacts reviewers above the upper one, ignores reviewers below the lower one and sends the rest to a person. Reviewer capacity limits how many reviews can go to people, and once it binds, the reviews worth reading are the ones whose expected loss falls most from a person's judgment.

Triage band

Automate, review or triage

Loading data

For one review at a time, the comparison is a table of expected losses, one row per action. The action with the lowest expected loss is the one to take.

Action loss table

Expected loss of each action

Loading data

Going further: dissatisfied is not the same as responsive

The recovery rate r describes how much a contact changes a customer's chance of staying, and star ratings cannot measure it. An unhappy customer may leave whatever the firm does, and another may return without any voucher. Contacting the most dissatisfied reviewers is the right ranking when a contact helps them at least as much as it helps anyone else at the same cost, an assumption a review label cannot check. A randomized contact experiment would provide a direct way to estimate that incremental effect.

The simulator below is a seeded simulation, not data from any firm. In it, a contact changes retention most for moderately dissatisfied customers who buy often and close to zero for the most dissatisfied. At the same contact budget, it compares targeting by the probability of dissatisfaction with targeting by the response estimated from a simulated experiment.

Uplift simulatorsynthetic

Targeting by dissatisfaction or by estimated uplift

Loading data

A cutoff applied once to each review also ignores time. A contact changes what the firm learns next, because a reply or a repeat purchase is new evidence about that customer. Chapter 3.1, Deciding when an agent is finished returns to decisions where each action changes the evidence for the next one.

What these data cannot show

The label is the reviewer's own star rating. It misses unhappy reviewers who still gave three stars and counts as dissatisfied some reviewers whose complaint was about delivery. The net values combine a measured error mix with assumed costs, so they show how policies compare under those costs and are not observed returns from contacting anyone. They also leave out what it costs to call or build each model. The data contain no contacts and no later purchases, so nothing here measures whether a contact changes what a customer does; the simulator shows why that matters and estimates nothing about real customers.

Data sources

Amazon Reviews 20235,000 coffee reviews, a sample stratified by star ratingWeights: sample_weight restores the star mix of the coffee reviews · License: No license stated; the booklet shows aggregates and short excerpts only

All data sources

Methods

Two decision engines, a general language model and a trained classifier gave each sampled review a probability of dissatisfaction. Policies contact a reviewer when that probability reaches a cutoff, and each policy is scored on the star labels with assumed costs.

What counts as dissatisfied

A review counts as dissatisfied when its star rating is one or two. The rating is the reviewer's own, so it is a proxy that exists without any rater. A few low ratings complain about delivery rather than the coffee, and a few high ratings contain complaints.

Engines and the classifier

Jev 1.14.0 and OpenAI Luna return a probability for a yes-or-no question. The Claude Haiku adapter returns a prompted estimate of that probability. The TF-IDF classifier was trained on reviews of other coffee products and calibrated on a held-out part of them.

Each engine was called once per review in October 2026 and the answers were stored.

Weights, folds and intervals

The sample holds the same number of reviews for each star rating, so every share, cost and calibration curve is weighted back to the star mix of the coffee reviews. Recalibrated probabilities and data-chosen cutoffs are cross-fit over five folds that keep each product's reviews together. Intervals come from a bootstrap that resamples products within star strata, with the same resamples for every policy.

How to cite

@misc{singh2026fromprobabilitiestoactions,
  author = {Singh, Vishal},
  title = {From probabilities to actions},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/from-probabilities-to-actions},
  note = {In Decision AI. Published 29 October 2026.}
}