The deployment role defines the evaluation

The same model can enter a service in several ways, and the role changes the claim that the service is making. A worklist may be reordered before anyone opens an image; a reader may see a suggestion while interpreting; a second reader may inspect an AI output after the initial read; or a system may draft a measurement that a clinician accepts, edits, or rejects. A deployment evaluation that ignores this interaction contract reports a number without saying what the number is supposed to establish.

Role decides what counts as success

Before asking whether a deployment performs well, state who sees the output, when they see it, what action it is meant to influence, and what happens when the output is missing or wrong. An AI used as a triage or prioritisation layer changes the order in which a human encounters cases. A concurrent-assistance tool changes the information available during interpretation. A second-reader tool creates a later checking step. An autonomous or standalone use assigns the decision to the software within a defined scope. A measurement or report-support tool produces an artefact that a reader may adopt or correct. These roles can be combined in one technical installation, but they are not one experiment.

The role also determines the relevant comparison. Standalone model performance asks how outputs relate to a reference under the evaluation conditions. It does not establish what happens when a radiologist has access to the same outputs. For that question, the informative comparison is the change between reader-alone and reader-plus-AI performance in a reader or workflow study, with the endpoint defined in advance. A model may be strong on its own and add little to an experienced reader, or the interaction may introduce errors that are not visible in the standalone evaluation. The distinction is developed in the reader-study section that follows.

Common roles are different contracts

In a standalone role, the system’s intended decision, operating population, failure handling, and escalation path have to be explicit because there is no immediate reader whose independent interpretation supplies the missing context. In concurrent assist, the output is visible while the reader works. The evaluation must therefore examine not only whether the suggestion is correct, but also whether it changes the reader’s decisions, time, and ability to detect errors. In a second-reader role, the question may be whether the later AI review adds findings or changes decisions after an independent first read; the timing is part of the intervention.

In triage, the output may be a priority or a flag rather than a diagnosis. The useful endpoint can concern which cases appear near the top of a queue, how quickly a flag is available, and how the resulting workload is distributed. In measurement or report support, the endpoint may concern the accuracy, consistency, edit burden, or traceability of the derived content. None of these examples makes one metric the correct metric for every implementation. They show why a single AUROC for “the deployment” can hide several distinct claims.

Role drives metrics, thresholds, and study design

The operating point follows the action. A triage system may be configured to surface more possible positives for human review, whereas a confirmation or measurement step may place greater weight on avoiding unnecessary actions. The choice is not encoded in the model’s ranking metric. It depends on prevalence, workload, the consequences of false-positive and false-negative decisions, and the way the output enters the work. The threshold section treats this as a deployment decision rather than a property to copy from model development.

Consider a constructed example in which one service reorders suspected acute findings and another proposes a measurement on the same PACS. The first service would need evidence about ranking, time-to-flag, queue load, and what readers do with the alerts. The second would need evidence about the reader’s measurement with and without the proposal, edit or rejection behaviour, and the reliability of the resulting record. Calling both uses “AI performance” and reporting one aggregate score would erase the different estimands. It would also make it impossible to decide which failure deserves investigation when the system is used in practice.

Thus role identification is the first act of deployment evaluation. It fixes the intended comparison and makes the later questions about readers, thresholds, integration, monitoring, governance, and adoption answerable without pretending that standalone model correctness is system benefit.