A good test-set model is not yet a clinical tool

Tonight · ~15 min · read · energy: low · setup: none

This short opening lesson draws the line the whole chapter works across: a model that scores well on a test set has answered one question (is it right, in distribution?); clinical use asks many more. Naming that gap is the prerequisite for every later lesson.

The anchor: “is it right?” ≠ “does it help?”

A test-set AUROC answers a narrow question: given this held-out set from the development world, how well does the model discriminate? Clinical deployment asks a cascade of further questions: does adding the model to a radiologist improve outcomes? Can its results actually be retrieved in the reading workflow (Ch. 5)? Is the operating threshold right for this site’s prevalence? Does the model degrade over time? Does it change the human’s behaviour? None of those are answered by a test set.

The questions a test set does not answer

Effectiveness in context — does reader+AI beat reader-alone? (Standalone performance does not establish this — lesson 3.)
Workflow integration — can the result reach and influence the reader (Ch. 5)?
Operating point — is the threshold right for this prevalence and workflow (lesson 5)?
Human factors — does the model create automation bias or alert fatigue (lesson 6)?
Durability — does performance hold as scanners, protocols and populations drift (lesson 8)?

Why this line matters

Because the gap between “validated” and “deployed” is where most real failures — and most real harm — happen. A model can be technically excellent and still fail clinically for informatics, workflow, threshold, human-factor, or drift reasons. The rest of this chapter is the structure that catches those: roles, reader studies, thresholds, human factors, shadow deployment, and monitoring.

Stop and think — then reveal

A vendor says “our model is validated, AUROC 0.95, ready to deploy”. List three questions whose answers a test-set AUROC does not provide, each of which could make deployment fail or harm.

(1) Does reader+AI actually outperform reader-alone in a real workflow — or does the model add nothing once a radiologist is in the loop (lesson 3)? (2) Can the result be retrieved and rendered at the reading workstation for this site’s PACS (Ch. 5)? (3) Is the operating threshold right for this site’s prevalence, and will performance and calibration hold as the site’s scanners/protocols/populations drift (lessons 5, 8)? Any of these can make a 0.95-AUROC model clinically useless or actively harmful.

What to retain

A test set answers “is it right, in distribution?” Clinical use asks “does it help, safely, in this workflow, over time?” — different questions.
A test set does not answer: in-context effectiveness, workflow integration, operating point, human factors, or durability.
The validated→deployed gap is where most real failures and harm occur; the rest of the chapter structures the catch.

Next: the first of those questions — what role is the AI actually playing?