Confidence-Threshold Routing

Most traffic in an automated system is easy, and a few cases are hard. Running every case through the strongest model wastes time and money, while running everything through a cheap one produces mistakes on the hard tail. Confidence-threshold routing splits the work: a fast model answers first and reports how sure it is, and a threshold decides whether that answer is used or the case is escalated.

The idea is old (cascades of classifiers have existed for a long time), but it has become a standard agent pattern with the arrival of fast decision models such as jev, which return probabilities by design. It combines naturally with human-in-the-loop: the human is the final tier for whatever the models are unsure about.

The pattern is only as good as the confidence numbers. Thresholds only make sense if the model is calibrated (calibrated-confidence) and if they are set on data that resembles real traffic.

How it works

Typically there are two or three bands. Above a high threshold, the fast model's answer is used automatically. In a middle band, the case goes to a stronger model, or the action is held for confirmation. Below a low threshold, it goes to a person. The thresholds are set by measuring, on labelled examples, how accurate the fast model is at each confidence level and how costly a wrong automatic action is. Safe rollout starts in shadow mode: the fast model labels real traffic while the existing process still decides, and you compare before letting it act.

Example

A support inbox uses a fast classifier to route messages. At 0.9 confidence or higher it assigns the team automatically. Between 0.5 and 0.9 it asks a larger model to decide. Below 0.5 it goes to a person to sort. The specific numbers (0.9 and 0.5) are illustrative; the team picks them from labelled data and how costly a misrouted message is.

Common misconceptions

FAQ