What role is the AI actually playing?

Tonight · ~15 min · read · energy: low · setup: none

How an AI is used changes what performance matters. This lesson is the deployment roles — because the right metric, the right threshold, and the right reader study all depend on which role the AI plays.

The anchor: role decides what counts as success

An AI that re-orders a worklist is judged very differently from an AI that drafts measurements a radiologist accepts, or an AI that acts alone. Before evaluating (or believing) a deployment claim, identify the role: it determines the relevant metric, the relevant comparison, and the relevant failure mode.

The common deployment roles

• Autonomous / standalone — the AI acts without a reader in the loop (rare, tightly scoped).
• Concurrent assist — the reader interprets with the AI output visible during the read.
• Second-reader — the AI output is reviewed after the reader’s own read.
• Triage / prioritisation — the AI re-orders a worklist (e.g. suspected bleed to the front); the image is still read by a human.
• Pre-populated structured measurements / report suggestions — the AI drafts measurements or findings the reader accepts or edits.

AI-alone ≠ radiologist+AI. A model’s standalone performance does not establish the benefit of adding it to a radiologist. The relevant metric is the delta between reader-alone and reader+AI, measured in a proper reader study (lesson 3). Many models with strong standalone numbers move the combined human+AI performance little or not at all.

Why role drives everything else

• Metric — triage cares about sensitivity at the top of the list and time-to-flag; a second reader cares about incremental findings caught; an autonomous tool cares about standalone accuracy and the cost of its errors.
• Threshold — a triage threshold (favour sensitivity, tolerate false positives) differs from a diagnostic-confirmation threshold (favour specificity) (lesson 5).
• Reader study design — concurrent vs second-reader roles need different study designs (lessons 3–4).

Stop and think — then reveal

A tool is deployed as a triage layer for intracranial haemorrhage on CT, and as a concurrent second-reader for lung-nodule measurement on the same PACS. Should the two uses be evaluated the same way?

No — the roles demand different metrics, thresholds, and study designs. Triage is judged on whether it surfaces likely-positive scans to the top of the worklist fast (sensitivity at the top of the list, time-to-flag), and tolerates false positives more readily. The concurrent measurement role is judged on whether reader+AI measurements are more accurate/consistent than reader-alone (lesson 3), with a threshold tuned for measurement confirmation. Reporting one AUROC for “the deployment” papers over the fact that these are two different experiments with two different success criteria.

What to retain

1. Role decides metric, threshold, and reader-study design. Identify the role before evaluating a deployment claim.
2. Roles: autonomous, concurrent assist, second-reader, triage, measurement/report support.
3. AI-alone ≠ reader+AI: the relevant number is the reader-alone vs reader+AI delta, not standalone AUROC.

Next: that delta, measured properly — AI-alone and radiologist+AI are different experiments.