Operating points in a changed workflow

A model usually emits a score, while a service has to decide what that score means for the work. The rule that converts a score into an alert, queue position, measurement, or escalation is an operating point. It is not simply “the AUROC-optimal threshold”. The ranking summarized by AUROC and the action rule applied at one site answer different questions. A threshold selected by a generic criterion on a development cohort may be a reasonable starting description of that cohort and still be unsuitable for a different workflow.

A threshold is a decision, not an optimisation

Choosing an operating point commits the system to a pattern of false positives and false negatives, a workload, and a response. The decision is therefore tied to the intended role and to the population in which it will be used. A threshold is not a portable clinical constant that can be copied from a paper or inherited from model development without checking what has changed.

What sets an operating point

Prevalence changes the predictive meaning of a score. Even if a model and its threshold remain unchanged, positive and negative predictive values can differ when the case mix differs. This does not mean that prevalence alone determines a threshold; it means that a threshold’s consequences must be assessed in the population and workflow where the alerts will be acted upon.

Workflow role changes the cost of the same error. A triage service may be intended to bring possible positives forward for human review, while a confirmation or measurement step may require a different balance before a result is accepted. The role-specific comparison described earlier is therefore also a role-specific threshold problem.

Relative error consequences connect the operating point to the action. The balance between a missed finding and an unnecessary follow-up is not the same in every task, and it cannot be inferred from AUROC. The relevant consequences include clinical and workflow consequences, but the evidence must state which consequences were considered rather than presenting one cost ordering as universal.

Calibration concerns whether predicted probabilities correspond to observed risk in the population in which they are used. It is distinct from discrimination and from the threshold itself. A service may need a calibration assessment or a justified recalibration strategy when the deployment population differs from development, with the transformation fitted and evaluated inside the appropriate validation boundary (Chapter 7).

Why this is a deployment problem, not a modelling one

The model produces a continuous quantity; the workflow turns it into a decision. Two sites can use the same model and face different prevalence, queue capacity, reader availability, or consequences of error. They may consequently require different operating points, or the same threshold with a different interpretation of its predictive value. Any re-estimation on an external population is an adaptation and must be reported as such. It should not be described as though the original frozen model had produced a universally portable rule.

Consider a constructed stroke-triage example. A threshold selected in a high-prevalence specialist service is copied unchanged into a lower-prevalence hospital. The number of alerts and the fraction that are true positives may change, even if the score formula has not. The local team would need to examine the case mix, alert load, predictive values, calibration, response capacity, and the consequences of false positives and false negatives before deciding whether the operating point is suitable. If it is changed, that change becomes part of the deployment specification and its evaluation.

This is why a threshold belongs in the lifecycle record alongside the model version, data population, workflow role, and decision owner. Once the score is presented to a person, the threshold becomes part of the human-plus-model intervention, which leads to the next question: how can the person be helped or misled by it?