Calibrated Confidence

Calibration is about honesty, not accuracy. A model can be right 70% of the time and perfectly calibrated if it says 70% every time, and a highly accurate model can be badly calibrated if it says 99% when it is right 85% of the time.

This matters as soon as software acts on confidence. Deciding to auto-approve above a threshold, or to send anything below it to a person, assumes the numbers mean something (see confidence-threshold-routing). Language models are often poorly calibrated when they state confidence in words: they tend to sound sure regardless of whether they are right, which is one route to a confident hallucination.

Some decision models, such as jev, are trained specifically to output probabilities that are calibrated (rlcd).

How it works

To check calibration you need labelled examples. Collect the model's predicted probability and whether it was actually correct for each one, group the predictions into confidence bands (for example 0.5–0.6, 0.6–0.7, and so on), and compare each band's average stated confidence with its real hit rate. A plot of this is called a reliability diagram. A single summary number such as expected calibration error or the Brier score is often reported. Calibration should be checked on data that looks like your production traffic, because it can drift when the input distribution changes.

Example

A team routes support tickets by an automatic classifier and lets it act alone above 0.9 confidence. Before trusting that, they label 500 past tickets and find that among predictions at 0.9 or higher the classifier was right 97% of the time, but among those at 0.6–0.7 only 41%. The high band is safe to automate; the low band should go to a person.

Common misconceptions

FAQ