# Resolving and routing customer complaints: report

Chapter 2.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints. Table n of this report is figure n of the chapter.

## Identification

- Title: Resolving and routing customer complaints
- Chapter: 2.1 (Part II, Scarce attention; Case studies)
- Question: Which product team should receive each complaint, when should a person check it first, and can a model tell which complaints will end in money back?
- Dek: A classifier trained on earlier complaints matches consumers' product choices about as often as Jev once it is fit to the actual mix, and it finds more of the complaints that end in money back, where it passes Jev with far fewer labels than the review task needs.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
  - CFPB Consumer Complaint Database. 10,000 complaints received in 2025 with a narrative of at most 3,000 characters, stratified by product; complaints from August 2023 to 2024 for the classifier; weights: sample_weight restores the product mix of the 2025 complaints within the length limit
- Canonical URL: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints
- Report: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report
- Markdown: https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md
- BibTeX key: singh2026resolvingandroutingcustomercomplaints

```bibtex
@misc{singh2026resolvingandroutingcustomercomplaints,
  author = {Singh, Vishal},
  title = {Resolving and routing customer complaints},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints},
  note = {In Decision AI. Published 29 October 2026.}
}
```

## Principal estimates

The facts the dek and the opening prose cite, in order of appearance.

| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| sample size | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 10,000 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | n_complaints |
| 2025 complaints within the length limit that the sample stands for | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 519,841 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | n_population |
| top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 81.9% | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | agree_jev |
| top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 82.4% | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | agree_tfidf |

## Findings

A bank's complaint desk gets a steady stream of written complaints, and each one needs to reach the right team. Send a mortgage complaint to the credit card team and it sits for days before someone sends it back. This chapter asks a decision engine to read each complaint and pick the product it is about, then asks the question every desk manager asks next: when should a person check the engine's pick before it goes out?

The complaints are 10,000 sampled from distinct narratives filed with the Consumer Financial Protection Bureau in 2025, standing for 519,841 narratives; repeated narratives were removed, which matters because many credit-reporting complaints use the same text. No one rated them by hand. Instead, each complaint carries two answers that already exist: the product the consumer selected when filing, and what the company eventually did about it.

Jev and a classifier fit to the actual complaint mix match the consumer's product choice about equally often, 82% and 82%, and once both are recalibrated the cost comparison shows no clear winner at the assumed costs. So the choice between them rests on what a desk can run and check, and either one still needs people for uncertain complaints. Consumer choices are sometimes wrong, and the costs and review times are assumptions. The chapter follows the desk from automatic routing to human review, a limit on review time and how to spend it, then takes up one companion question: which complaints end in money back.

<SixQuestions
  owner="The complaint desk lead decides where each complaint goes; operations owns the cost of delay."
  evidence="The complaint narrative as the consumer wrote it, and nothing added later."
  alternative="A word-counting classifier trained on complaints from earlier years."
  correctness="The consumer's own product choice, which is sometimes wrong, and the company's recorded response."
  whenWrong="A misrouted complaint waits days for the right team; a person reviewing every complaint costs staff time."
  doesItPay="Expected cost per thousand complaints under stated costs for misroutes and triage."
/>

### Picking the product

Each engine saw the narrative and a list of eleven products with a short definition of each, and picked one. Jev and Luna also returned a probability for every product; the Haiku adapter returned only its pick. The chart compares how often the top pick matches what the consumer chose.

Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev

Jev agrees on 82% of complaints and the classifier on 82%, a gap of −0.6 (95% CI −1.9 to 0.7) points. How the classifier is trained decides this comparison. The training sample holds equal numbers of each product, and fitting it without weights drops the classifier to 78%, while the weighted fit does poorly on rare products: counted equally across products, Jev scores 75% and the classifier 64%. Jev costs about $0.05 per thousand complaints.

Agreement is a soft score here. Consumers choose their own product from a form, and some choose badly: a complaint about a debt collector chasing a credit card balance can honestly belong to either team. The flows below show where Jev and the consumers part ways. The largest disagreements run between credit reporting and debt collection, two products that often appear in the same complaint.

<RoutingFlow id="ch4-flow" src="resolving-and-routing-customer-complaints/flow.json" />

### When to ask a person first

Suppose a misrouted complaint costs $30 in delay and sending a complaint to a person for triage costs $6 of staff time. Routing pays when the chance the pick is right is high enough that the expected misroute cost falls below the triage cost. At these costs the cutoff is 0.8: route when the model is more sure than that, and send everything else to a person.

<Callout kind="assumed">

Both costs are assumptions for this chapter. A desk would measure the first from how long misrouted complaints wait and the second from triage handling time. The calculation also assumes that a person always finds the right team, at a fixed cost, with no limit on how many complaints a person can read. It leaves out engine charges and the cost of building the classifier.

</Callout>

<TriageBand
  id="ch4-triage"
  mode="single"
  items="resolving-and-routing-customer-complaints/items"
  engine="jev"
  unit="complaints"
  cutoff={0.8}
  minutes={4}
  dollarsPerMinute={1.5}
  capacityPer1000={300}
  presets={{ "classifier": { engine: "tfidf" }, "high-bar": { cutoff: 0.95 } }}
/>

<TryThis preset="classifier" target="ch4-triage">

Switch to the classifier. Its stated probabilities are lower, so far more of its complaints fall below the cutoff and go to a person, more than this panel's triage capacity can take. A desk then has three moves: add reviewers, let a queue build, or lower the confidence it requires for automatic routing, which cuts the workload and accepts more routing risk. Raising that bar would send even more complaints to people.

</TryThis>

The next chart shows what each policy does with the whole stream of complaints.

Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high

With the cutoff taken at face value, Jev sends 16% of complaints to a person and misroutes 10%, and the classifier sends 43% and misroutes 4%. Priced at the assumed costs, that is $4,054 per thousand complaints for Jev and $3,685 for the classifier. Routing everything by Jev with no check costs $5,442, and sending everything to a person costs $6,000.

The gap of +$368 (95% CI $97 to $655) is about calibration more than about reading. Jev's stated probabilities run high, so it routes complaints it should send to a person. Recalibrate both models on other folds of the sample and Jev costs $3,471, sending 35% to a person, against $3,666 for the classifier.

<Callout kind="null">

With both recalibrated, the gap between Jev and the classifier is −$195 (95% CI −$419 to $27) per thousand complaints, and that range includes zero, so at these costs the data show no clear advantage for either.

</Callout>

<PolicyCompare id="ch4-policies" src="resolving-and-routing-customer-complaints/policy.json" />

That comparison gives both models plenty of labels. A desk starting out would have few, so the next chart gives both the same labeled complaints from earlier years.

Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried

With five hundred labels, recalibrated Jev costs $3,600 per thousand complaints, against $4,100 for raw Jev and $4,800 for a classifier fit on those labels. With the full sample the classifier comes closer, at $3,700 against $3,500. Labels from earlier years are what a desk would really have, and these figures leave out what each model costs to run.

### When reviewer time is fixed

So far a person has been free to read as many complaints as the cutoff sends. Suppose the desk instead has 400 reviewer minutes for every thousand complaints, and everything outside the queue goes where Jev's pick sends it. Which complaints should a person read? The complaints and the costs stay as they were; what changes is that the desk can no longer send a person everything below the cutoff.

Uncertainty is one reason to look at a complaint. Consequence is another, so the queue below also uses a second model. Its estimate of money back, from the companion section that follows, stands in for a misroute that matters more, under an assumed extra cost. It is a proxy and not a measured loss from a misroute. Read the columns separately, because review workload, misroutes, misrouted money-back complaints and assumed dollars can move in different directions.

<Callout kind="assumed">

The review budget, the review time, which grows with the length of the complaint, the extra $120 for misrouting a complaint that ends in money back, and a reviewer who always finds the right team are assumptions. Misroutes are counted against the consumer's product choice. The ranking uses recalibrated Jev for the risk of a misroute and the classifier's probability of money back for the consequence.

</Callout>

<PolicyCompare id="ch4-queue" src="resolving-and-routing-customer-complaints/queue.json" />

Reviewing the least certain picks is the natural rule, and it leaves 110.4 misroutes per thousand complaints, down from 181.5 with no review. Adding the consequence moves attention toward complaints that may involve money, leaving 1.14 misrouted money-back complaints against 1.57, a difference of −0.43 (95% CI −0.91 to 0.09), whose range includes zero. Adding review time changes more: because short complaints take less time, the same minutes cover 20% of complaints instead of 14%, and the expected cost changes by −425 (95% CI −639 to −198) dollars per thousand. That gain rests on the assumed link between length and review time, and on a short complaint being reviewed as accurately as a long one. A desk would need to measure both before running this queue.

### A companion question: which complaints end in money back

This is a different job from routing: it predicts what a company will do later, not what a complaint is about. A desk might fast-track complaints that are likely to end with the company paying the consumer back. Only 3% of these complaints closed with monetary relief, so finding them early matters. Each engine was asked whether the consumer describes money the company could return, and the classifier was trained directly on which earlier complaints ended in relief.

Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint

Fast-tracking the tenth of complaints the classifier ranks highest would catch 73% of the money-back cases, against 58% for Jev, a gap of +14.9 (95% CI 8.9 to 20.4) points. Much of any model's score here comes from knowing the product, because money back is far more common for some products than others: ranking complaints by their product's past rate alone gives an AUC of 0.906, above Jev's 0.895. Within a product, the classifier's lead narrows to 0.769 against 0.742, a difference of +0.027 (95% CI −0.009 to 0.065). The engine answers the question it was asked, whether the consumer describes a refundable loss. The classifier learned from history which complaints companies actually paid. One reading is that history carries information a single complaint does not, though this comparison cannot separate that from the difference in the question asked, the number of labels or calibration.

Could a better prompt close the gap? On a random subset of complaints, the engines were asked a sharper question, whether the company will likely give money back, in three ways: on its own, with ten earlier complaints labeled paid or not paid, and with a short description of what tended to get paid in earlier years.

Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier

The sharper question and the description together help a little, and rewording alone accounts for much of that. The labeled examples add little. None of the versions brings the engines level with the classifier. How much history does the classifier need? Refit on smaller samples of past complaints, it matches Jev at about a thousand.

Table 6. For money back, the classifier matches Jev with about a thousand past complaints

The classifier catches 49% with three hundred past complaints, 59% with a thousand and 66% with three thousand. That is far fewer labels than the review task needed, and a desk already has these records, because its own case system stores how each complaint closed.

These counts include calibration labels held out from each sample, and the complaints being scored are not used. They describe this classifier on these records, so treat them as a guide for a pilot. Whether a desk should build this model also depends on whether outcomes arrive in time, whether old records resemble today's complaints, and whether money back is what the desk wants to prioritize.

<Discuss>

On reading what a complaint is about, the classifier needs a large labeled sample to match the engines, and on predicting what a company will do it passes them with a modest one. Which of those two jobs should a desk buy a model for, and which should it learn from its own records?

</Discuss>

<InstructorNote>

Ask the class to set the misroute cost from a regulator's point of view and then from a call-center manager's, and to watch how the triage share changes in the triage panel. Then ask why a classifier trained on past outcomes can beat a model that reads each complaint more closely on the money-back question.

</InstructorNote>

### What these data cannot show

The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.

## Tables

### Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev

Complaints whose predicted product matches the one the consumer selected, percent, by model, with 95% intervals

| agree | conf_high | conf_low | kind | model | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 81.90 | 83.20 | 80.50 | Decision engine | Jev 1.14.0 | 10,000 |
| 84.20 | 85.30 | 82.90 | Decision engine | OpenAI Luna | 10,000 |
| 84.40 | 85.50 | 83.30 | General language model | Claude Haiku adapter | 10,000 |
| 82.40 | 83.70 | 81.20 | Trained classifier | TF-IDF + logistic | 10,000 |

Note: Jev minus the classifier: −0.6 percentage points, from a bootstrap that resamples complaints within products.

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json

### Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high

Where each routing policy sends complaints, percent of complaints, 2025, weighted

| outcome | policy | share |
| --- | --- | ---: |
| Routed to the consumer's product | Jev, triage below the cutoff | 73.40 |
| Routed to another product | Jev, triage below the cutoff | 10.20 |
| Sent to a person for triage | Jev, triage below the cutoff | 16.40 |
| Routed to the consumer's product | Luna, triage below the cutoff | 81.60 |
| Routed to another product | Luna, triage below the cutoff | 12.60 |
| Sent to a person for triage | Luna, triage below the cutoff | 5.90 |
| Routed to the consumer's product | TF-IDF classifier, triage below the cutoff | 53.70 |
| Routed to another product | TF-IDF classifier, triage below the cutoff | 3.80 |
| Sent to a person for triage | TF-IDF classifier, triage below the cutoff | 42.50 |
| Routed to the consumer's product | Jev recalibrated, triage below the cutoff | 60.40 |
| Routed to another product | Jev recalibrated, triage below the cutoff | 4.60 |
| Sent to a person for triage | Jev recalibrated, triage below the cutoff | 35.10 |
| Routed to the consumer's product | TF-IDF recalibrated, triage below the cutoff | 55 |
| Routed to another product | TF-IDF recalibrated, triage below the cutoff | 4 |
| Sent to a person for triage | TF-IDF recalibrated, triage below the cutoff | 41 |
| Routed to the consumer's product | Jev, route everything | 81.80 |
| Routed to another product | Jev, route everything | 18.10 |
| Sent to a person for triage | Jev, route everything | 0 |
| Routed to the consumer's product | Haiku adapter, route everything (no probabilities) | 84.40 |
| Routed to another product | Haiku adapter, route everything (no probabilities) | 15.60 |
| Sent to a person for triage | Haiku adapter, route everything (no probabilities) | 0 |
| Routed to the consumer's product | Send everything to triage | 0 |
| Routed to another product | Send everything to triage | 0 |
| Sent to a person for triage | Send everything to triage | 100 |

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json

### Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried

Expected routing cost per thousand complaints against local labels used, weighted; whiskers run from the lowest to the highest draw

| conf_high | conf_low | cost | labels | model | n_labels |
| ---: | ---: | ---: | --- | --- | ---: |
| 11720 | 4815 | 6289 | 50 labels | Classifier, fit and calibrated on these labels | 50 |
| 4395 | 3471 | 3899 | 50 labels | Jev, recalibrated on these labels | 50 |
| 6643 | 4417 | 5303 | 100 labels | Classifier, fit and calibrated on these labels | 100 |
| 5308 | 3594 | 4128 | 100 labels | Jev, recalibrated on these labels | 100 |
| 6000 | 4733 | 5125 | 200 labels | Classifier, fit and calibrated on these labels | 200 |
| 4556 | 3471 | 3868 | 200 labels | Jev, recalibrated on these labels | 200 |
| 5851 | 4468 | 4792 | 500 labels | Classifier, fit and calibrated on these labels | 500 |
| 3769 | 3532 | 3613 | 500 labels | Jev, recalibrated on these labels | 500 |
| 5187 | 4272 | 4500 | 1,000 labels | Classifier, fit and calibrated on these labels | 1,000 |
| 4054 | 3471 | 3596 | 1,000 labels | Jev, recalibrated on these labels | 1,000 |
| 4497 | 4215 | 4326 | 2,000 labels | Classifier, fit and calibrated on these labels | 2,000 |
| 3839 | 3471 | 3587 | 2,000 labels | Jev, recalibrated on these labels | 2,000 |
| 4208 | 4026 | 4119 | 5,000 labels | Classifier, fit and calibrated on these labels | 5,000 |
| 3594 | 3471 | 3528 | 5,000 labels | Jev, recalibrated on these labels | 5,000 |
| 4145 | 3889 | 3993 | 10,000 labels | Classifier, fit and calibrated on these labels | 10,000 |
| 3594 | 3471 | 3546 | 10,000 labels | Jev, recalibrated on these labels | 10,000 |
| 3837 | 3683 | 3742 | 30,000 labels | Classifier, fit and calibrated on these labels | 30,000 |
| n/a | n/a | 3542 | 30,000 labels | Jev, recalibrated on these labels | 30,000 |

Note: Platt scaling in place of isotonic calibration gives the same reading.

Source: CFPB Consumer Complaint Database; labels from August 2023 to 2024; policy scored on 2025 complaints, weighted; engine calls October 2026; assumed costs

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json

### Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint

Share of complaints closed with monetary relief that fall in each model's top tenth by P(money back), percent

| caught | kind | model |
| ---: | --- | --- |
| 58.20 | Decision engine | Jev 1.14.0 |
| 63.10 | Decision engine | OpenAI Luna |
| 51.80 | General language model | Claude Haiku adapter |
| 73.10 | Trained classifier | TF-IDF + logistic |

Note: Ranked over all complaints, the classifier's AUC is 0.936 and Jev's is 0.895; Jev minus the classifier is −0.041.

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json

### Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier

AUC for complaints closed with monetary relief, by model and how the question was posed, with 95% intervals; 2,000 sampled 2025 complaints

| auc | conf_high | conf_low | group | label | n |
| ---: | ---: | ---: | --- | --- | ---: |
| 0.947 | 0.959 | 0.932 | Classifier trained on past outcomes | Classifier | 2,000 |
| 0.881 | 0.912 | 0.847 | Jev | Jev: original question | 2,000 |
| 0.899 | 0.922 | 0.873 | Jev | Jev: reworded to predict payment | 2,000 |
| 0.903 | 0.926 | 0.878 | Jev | Jev: reworded + ten labeled examples | 2,000 |
| 0.913 | 0.931 | 0.891 | Jev | Jev: reworded + outcome criteria | 2,000 |
| 0.887 | 0.920 | 0.845 | Luna | Luna: original question | 2,000 |
| 0.903 | 0.929 | 0.875 | Luna | Luna: reworded to predict payment | 2,000 |
| 0.889 | 0.911 | 0.866 | Luna | Luna: reworded + ten labeled examples | 2,000 |
| 0.915 | 0.934 | 0.891 | Luna | Luna: reworded + outcome criteria | 2,000 |

Note: Only about one complaint in forty ended in money back, so the intervals are wide.

Source: CFPB Consumer Complaint Database, 2025 complaints; 2,000 drawn at random from the sample, weighted; engine calls October 2026

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json

### Table 6. For money back, the classifier matches Jev with about a thousand past complaints

Money-back complaints caught in the top tenth, percent, against past complaints used to train the classifier, weighted

| caught | conf_high | conf_low | labels |
| ---: | ---: | ---: | --- |
| 34 | 51.90 | 11.20 | 100 complaints |
| 49.30 | 59.50 | 30.50 | 300 complaints |
| 58.80 | 65.30 | 53.10 | 1,000 complaints |
| 66.40 | 69.10 | 63.60 | 3,000 complaints |
| 69.90 | 72.90 | 66.60 | 10,000 complaints |
| 74.90 | 75.90 | 74.20 | 30,000 complaints |

Source: CFPB Consumer Complaint Database, 2025 distinct narratives with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026; ten refits per size, three at the largest

Data: https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json

## All estimates

Every fact in facts.json (73), sorted by id.

| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| agree_diff_jev_tfidf | −0.6 | percentage points | jev minus tfidf | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −1.9 to 0.7 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| agree_jev | 81.9% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| agree_jev_equal_products | 75.2% | percent of complaints, each product counted equally | mean agreement across the eleven products | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| agree_tfidf | 82.4% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| agree_tfidf_equal_products | 63.6% | percent of complaints, each product counted equally | mean agreement across the eleven products | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| agree_tfidf_unweighted_fit | 78.2% | percent of complaints | agreement for a classifier fit without weights | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_money_diff_jev_tfidf | −0.041 | difference in AUC | Jev minus TF-IDF | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.054 to −0.029 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_money_jev | 0.895 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.881 to 0.908 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_money_product_only | 0.906 | area under the ROC curve | ranking complaints by their product's relief rate in the training years | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_money_tfidf | 0.936 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.928 to 0.945 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_within_adapter | 0.730 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.704 to 0.756 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_within_diff_tfidf_jev | +0.027 | difference in AUC | classifier minus Jev, within product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.009 to 0.065 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_within_jev | 0.742 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.709 to 0.770 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_within_luna | 0.743 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.706 to 0.772 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| auc_within_tfidf | 0.769 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.736 to 0.802 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| capture10_diff_tfidf_jev | +14.9 | percentage points | classifier minus Jev | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 8.9 to 20.4 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| capture10_jev | 58.2% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 53.0% to 63.4% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| capture10_tfidf | 73.1% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 68.4% to 77.9% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| cost1k_jev | $0.05 | US dollars per 1,000 complaints | mean billed cost per call, list price | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| lc_capture_1000 | 58.8% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| lc_capture_300 | 49.3% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 300 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| lc_capture_3000 | 66.4% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 3,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| loss_diff_jev_cal | −$195 | US dollars per 1,000 complaints | jev_cal minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −$419 to $27 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| loss_diff_jev_cut | +$368 | US dollars per 1,000 complaints | jev_cut minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $97 to $655 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| loss_jev_all | $5,442 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $5,041 to $5,861 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_jev_cal | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_jev_cut | $4,054 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,740 to $4,384 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cal | $3,666 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,466 to $3,885 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cut | $3,685 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,482 to $3,893 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| loss_triage_all | $6,000 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $6,000 to $6,000 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs; measured error mix against the consumer's product. |
| lv_clf_500 | $4,792 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_clf_full | $3,742 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_500 | $3,613 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_jev_full | $3,542 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed costs. Both arms use the same labeled draws. |
| lv_raw_jev | $4,054 | US dollars per thousand complaints | raw Jev, no local fitting | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| mis_jev_cal | 4.6% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| mis_jev_cut | 10.2% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| mis_tfidf_cal | 4.0% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| mis_tfidf_cut | 3.8% | percent of complaints | routed to a product other than the consumer's | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| misroute_cost | $30 | US dollars per misrouted complaint | assumed cost: three days of delay at $10 | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | CFPB Consumer Complaint Database |   |
| money_extra | $120 | US dollars per misrouted complaint that ends in money back | assumed extra cost | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | CFPB Consumer Complaint Database |   |
| n_complaints | 10,000 | complaints | sample size | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| n_population | 519,841 | complaints | 2025 complaints within the length limit that the sample stands for | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| narrative_cap | 3,000 | characters | length cap of the stored narratives | CFPB narratives in the July 2026 copy | 2025 |   |   | not applicable | CFPB Consumer Complaint Database |   |
| queue_diff_value_cost | +79 | per 1,000 complaints | value minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −56 to 233 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| queue_diff_value_left_money | −0.43 | per 1,000 complaints | value minus reviewing the least certain, left_money | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −0.91 to 0.09 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| queue_diff_value_per_minute_cost | −425 | per 1,000 complaints | value_per_minute minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −639 to −198 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| queue_diff_value_per_minute_left | −12.15 | per 1,000 complaints | value_per_minute minus reviewing the least certain, left | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −18.81 to −4.95 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| queue_minutes | 400 | reviewer minutes per 1,000 complaints | assumed review budget | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | CFPB Consumer Complaint Database |   |
| queue_none_cost | $5,827 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $5,402 to $6,277 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_left | 181.5 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 168.2 to 195.6 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_left_money | 3.18 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 2.26 to 4.20 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_none_reviewed | 0.0% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_cost | $3,499 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,144 to $3,852 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_left | 110.4 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 99.4 to 121.6 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_left_money | 1.57 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.89 to 2.31 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_uncertain_reviewed | 14.3% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 13.1% to 15.6% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_cost | $3,579 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,229 to $3,944 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_left | 114.7 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 103.6 to 126.1 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_left_money | 1.14 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.54 to 1.88 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_cost | $3,074 | US dollars per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $2,732 to $3,429 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_left | 98.2 | misroutes left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 87.0 to 109.5 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_left_money | 1.06 | misrouted money-back complaints left per 1,000 complaints | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.50 to 1.73 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_per_minute_reviewed | 20.2% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 18.8% to 21.6% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| queue_value_reviewed | 14.1% | percent of complaints reviewed | review queue of 400 reviewer minutes per 1,000 complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 13.0% to 15.3% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product. |
| route_cutoff | 0.8 | probability that the top product is right | 1 minus triage cost over misroute cost | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | CFPB Consumer Complaint Database |   |
| share_relief | 2.6% | percent of complaints | company response is closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| share_too_long | 7.7% | percent of deduplicated 2025 complaints with a narrative | narratives over 3,000 characters, excluded | deduplicated 2025 CFPB complaints with a narrative | 2025 |   |   | full count | CFPB Consumer Complaint Database |   |
| tri_jev_cal | 35.1% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| tri_jev_cut | 16.4% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| tri_tfidf_cal | 41.0% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| tri_tfidf_cut | 42.5% | percent of complaints | sent to triage | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database |   |
| triage_cost | $6 | US dollars per complaint sent to triage | assumed cost: four minutes of a triage handler | the complaint desk's routing decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | CFPB Consumer Complaint Database |   |

## Definitions

The chapter marks no glossary term.

## Methods

Two decision engines and a general language model read each sampled complaint narrative and answered two typed questions: which product it is about, and whether the consumer describes money the company could return. A classifier trained on earlier complaints answered the same two. Nothing was rated by hand.

### Which complaints

CFPB withdrew complaint narratives from public access in August 2026, so the only copy available keeps the first 3,000 characters of each narrative. The sample keeps only complaints whose whole narrative fits, which excludes 7.7% of 2025 complaints with a narrative. Every result describes complaints within that length.

The sample holds the same number of complaints for each of the eleven products, and weights restore each product's share of 2025 complaints.

### Labels

Routing is scored against the product the consumer selected when filing. Consumers sometimes pick the wrong product, so a disagreement can be the consumer's mistake rather than the engine's.

The money-back question is scored against the company's recorded response, closed with monetary relief. That outcome depends on the company as well as the complaint.

### Intervals

Intervals are 95% bootstrap intervals that resample complaints within product strata, keep each product's share fixed and use the same resamples for every policy. They describe sampling error for 2025 complaints and say nothing about other years or other companies.

## What these data cannot show

The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.

## References

The chapter cites no reference.

## Provenance

- Build script: decision_ai/chapters/resolving-and-routing-customer-complaints/build.py (sha256 ad8930e6ff310f8e48d53f32dc53dc37c1156cc14259c5adbe4b587e8d5a291f)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs:

| File | SHA-256 |
| --- | --- |
| facts.json | 20c650ec38ef9810222843d6e40613ef2d6545f645f749ed57035e7fe6d7ea92 |
| figures/agree-by-model.json | 1b7f77c56eee87f3469be5ac6cf9affb228c657e53801d78d2e3366e9a7fc1f8 |
| figures/cost-by-labels.json | 34360a16ffe4d9b7133cacf6e24209e9e0410bcb594a1d42581a2fab26e5a5ce |
| figures/money-back-caught.json | 03c660401c1437e49780d3a291312f0aa46f0a7debdc69855fd6a151018ea1e8 |
| figures/money-back-prompts.json | 0d8296a19c6854ea1f9c9ef1851b6e05b607916c3ef3ad6e632a3a2c0e89d754 |
| figures/outcomes-to-catch-up.json | 3a66877ee2c772f4622dda2511a750923589ebfad0c9b74ee85deca531a3aa95 |
| figures/where-complaints-go.json | 43adef141cfb6ed2a770238c094ba0436f6ea8ca0c0f591d0e1d38a6b6d305b8 |
| meta.json | e7892b08742ced88d370d9d8660f15d3481e33f08529e79bc4f5923f2673ef90 |
| public/data/resolving-and-routing-customer-complaints/flow.json | 85255672e8768ac873e89cd12604fb38c55cd4c0f5042805034be293c2a785ef |
| public/data/resolving-and-routing-customer-complaints/items.json | 27123950e0178d495717b23b6f360caf78b7babd3f5f88f989218637fe0dd855 |
| public/data/resolving-and-routing-customer-complaints/policy.json | c2c0475becd99606c6f49869aeeaece194b993c52165a170c0114b6bec2ca6b5 |
| public/data/resolving-and-routing-customer-complaints/queue.json | 98f9a4750c6905d6d2c87450977d629f97d0a14659f845a043eca707a0b9bc4c |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |

## Machine formats

- Report (HTML): https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report
- Report (Markdown): https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/facts.json
- Table 1, figure "agree-by-model" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json
- Table 2, figure "where-complaints-go" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json
- Table 3, figure "cost-by-labels" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json
- Table 4, figure "money-back-caught" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json
- Table 5, figure "money-back-prompts" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json
- Table 6, figure "outcomes-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt