Skip to content
Imaging Evidence Path

Shadow deployment before clinical deployment

Tonight · ~20 min · read · energy: low · setup: none

A sensible deployment does not jump from “good retrospective numbers” to “live use”. This lesson is the intermediate stages — external testing, silent/shadow deployment, prospective evaluation — that catch the failures lessons 1–6 warned about, before they reach patients.

The anchor: close the gap in stages, not a leap

Recall lesson 1: a test-set model is not a clinical tool. The gap between them is closed in stages, each of which catches a different class of failure: external testing catches domain shift; shadow deployment catches workflow, retrieval and threshold problems without risking patients; a reader study catches human-factors effects (lessons 3–4); prospective deployment then tests it live. Skipping stages is how correct models cause harm.

A sensible (not universal) path

flowchart LR
    A["retrospective<br/>development"] --> B["independent<br/>external testing<br/>(frozen model)"]
    B --> C["silent / shadow<br/>deployment<br/>(AI runs, outputs hidden,<br/>compared to real reads)"]
    C --> D["reader / clinical-<br/>workflow study"]
    D --> E["prospective<br/>deployment<br/>(live, with oversight)"]
    E --> F["post-deployment<br/>monitoring"]
  • Independent external testing — frozen-model evaluation on a genuinely independent cohort (Ch. 4); catches domain shift.
  • Silent / shadow deployment — the AI runs on real incoming studies, but its outputs are hidden from readers and compared to the real reads after the fact. This catches workflow/integration failures (Ch. 5), threshold problems (lesson 5), and prevalence/PPV issues without exposing patients to the model’s decisions.
  • Reader / clinical-workflow study — MRMC crossover (lesson 3) to measure the reader+AI delta and human-factors effects.
  • Prospective deployment — live use with oversight, having passed the above.
  • Post-deployment monitoring — the subject of lesson 8.

Not every system follows exactly this path; regulators and local governance shape it. But jumping from “good retrospective numbers” straight to “live autonomous use” skips the steps that catch real failure modes.

Why shadow deployment is the under-used stage

Shadow deployment is where you learn whether the model integrates and behaves in the real environment without risk: does the result get retrieved (Ch. 5)? What is the real-world FP load and PPV? Where does it fail? Because the outputs are hidden, none of these discoveries harms a patient — they just inform whether to proceed. Many deployments skip it and discover integration and threshold failures only after go-live, when they are already affecting care.

Stop and think — then reveal

A team wants to skip shadow deployment because “the external test was already good”. Name two failure classes the external test does not catch that shadow deployment would.

(1) Workflow/integration failures — can the result actually be retrieved and rendered at this site’s PACS, attached to the right study, in time to matter (Ch. 5)? An external test on exported data does not exercise the live PACS integration. (2) Real-world prevalence/ threshold/PPV behaviour and alert load — the external test cohort may not match the live case mix, so the FP load and PPV at the chosen threshold (lesson 5), and the alert fatigue it could cause (lesson 6), only become visible under real incoming studies. Shadow deployment catches both without exposing patients.

What to retain

  1. Close the validated→deployed gap in stages: external testing → shadow deployment → reader study → prospective deployment → monitoring.
  2. Shadow deployment runs the AI on real studies with outputs hidden, catching workflow, threshold, and prevalence failures without patient risk — the under-used stage.
  3. Each stage catches a different failure class (domain shift, integration, human factors); skipping stages is how correct models cause harm.
  4. Regulators and local governance shape the exact path, but “good retrospective → live autonomous” is never acceptable.

Next: once live, the model starts ageing the day it is deployed.