Why thresholds change when the workflow changes

Tonight · ~15 min · read · energy: low · setup: none

A deployed operating point is not “the AUROC-optimal threshold”. This short lesson is why the right threshold depends on prevalence, workflow role, and the cost of errors — and why a threshold tuned in one setting can fail in another.

The anchor: a threshold is a decision, not an optimisation

The AUROC-optimal threshold (e.g. Youden’s point) maximises a generic trade-off on the development cohort. Deployment is not generic: it has a specific disease prevalence, a specific workflow role, and specific costs for false positives vs false negatives. A threshold that was sensible in development can be clinically wrong at deployment for any of those reasons.

The four things that set a deployed threshold

• Prevalence — the same model/score yields very different positive/negative predictive values at different disease prevalence; a threshold tuned at a high-prevalence site fails at a low-prevalence one (and vice versa).
• Workflow role — a triage threshold favours sensitivity (surface likely positives, tolerate false positives) while a diagnostic-confirmation threshold favours specificity (lesson 2).
• Cost of FP vs FN — set the threshold against the clinical cost of each error: a missed cancer (FN) is far costlier than an unnecessary biopsy (FP) in screening, which pushes the threshold toward sensitivity; the calculus differs for, say, ruled-out conditions.
• Calibration — predicted probabilities should match observed risk in the deployed population; recalibrate if the deployment distribution differs from training (Ch. 4).

Why this is a deployment problem, not a modelling one

The model produces a continuous score; the threshold turns it into a decision. Two sites with the same model and the same score distribution can need different thresholds because their prevalence and error costs differ. So a threshold is a clinical choice that must be (re)set per deployment context, with calibration checked against the deployed population — not a constant inherited from the development cohort’s Youden point. This also connects to Chapter 7: thresholds re-estimated on the external set are a form of adaptation, and must be reported as such.

Stop and think — then reveal

A model flags suspected large-vessel occlusion on CT for stroke triage. It was threshold-tuned in a hyper-acute stroke centre (high prevalence). It is deployed unchanged at a low-prevalence community hospital. What goes wrong, and what should have been re-checked?

At the low-prevalence site, the same threshold yields a much lower positive predictive value — most flags become false positives, risking alert fatigue (lesson 6) and potentially desensitising staff to true positives. The threshold should have been re-set (or the model recalibrated) for the community site’s prevalence, with PPV/NPV re-checked there and the FP load assessed against the triage role’s tolerance. A threshold is a per-deployment clinical decision, not a portable constant.

What to retain

1. A deployed threshold is a clinical decision, not the AUROC-optimal point.
2. It depends on prevalence, workflow role (triage vs confirmation), FP/FN cost, and calibration in the deployed population.
3. Two sites with the same model can need different thresholds; recalibrate against the deployed population.
4. Thresholds re-estimated on an external set are adaptation — report them as such.

Next: the part engineers underestimate — the human can become part of the failure mode.