Reference
Methods
How the numbers in the figures are made.
This page states the rules that every chapter shares. Each chapter's methods note gives the sample, weights, engines, run dates and interval method for its own figures, and a chapter that departs from a rule here says so.
Engine runs
Every call to a decision model or a language model was made once and cached. A chapter's methods name its engines and when they were called. No number in the text depends on a live call.
Samples and weights
Most chapters draw a stratified sample, which means a fixed number of records from each group, such as each star rating, so that rare groups have enough records to study. Each sampled record carries a sample weightSample weightA number attached to each sampled item so that weighted results describe the population the sample was drawn from rather than the sample's own mix. that restores the population's mix, and every count, share, calibration curve and cost in the text is weighted unless its source line says "unweighted".
Fitting and scoring on separate records
Thresholds chosen from data, recalibrated probabilities and fitted combinations are scored with cross-fittingCross-fittingSplitting the data into folds, fitting on all but one fold and scoring the held-out fold, so that no item is scored by a rule fitted on it., so no record is scored by a rule fitted on it. Folds are split by the unit that carries dependence, such as the product, the company, the repository or the task.
Intervals
Intervals are bootstrap intervals from resampling within the sample's strata, usually by cluster, and each replicate keeps the known mix of the strata. The same resamples are used for every engine and policy compared, so the interval on a difference reflects the pairing. The complaint chapter is an exception: a few companies hold most of its largest product, so it resamples individual complaints within product, and its intervals describe sampling error for 2025 only. Whiskers on the learning-curve charts are the lowest and highest refit, not intervals.
Refusals and errors
A call that a provider refused or that failed is stored and counted. Chapters report refusal and error rates, and no record is dropped without a recorded reason.