Skip to content
Imaging Evidence Path

The model starts ageing the day it is deployed

Tonight · ~20 min · read · energy: low · setup: none

Deployment is not the end of validation. A model degrades in the field as scanners, protocols, populations and even reference standards drift. This lesson is the drift taxonomy, the monitoring it forces, and the standards-level support for it (IHE AIRA).

The anchor: the world the model was trained on does not hold still

A model is validated against a snapshot of the world. The world moves: scanners are upgraded, protocols revised, the patient mix shifts, and even the reference standard changes (new reporting criteria). Each is a drift that can silently degrade performance. Without monitoring, that degradation persists until harm occurs — which is why deployment is the start of validation, not the end.

The drift taxonomy

  • Data drift — input distribution changes (different scanners, protocols, demographics).
  • Protocol / scanner drift — the site upgrades hardware/software; the acquisition pipeline the model was trained on changes (Ch. 6).
  • Population drift — the patient mix shifts (prevalence, comorbidities).
  • Label / clinical-practice drift — the reference standard itself changes (new reporting criteria, revised guidelines).
  • Performance drift — the downstream effect: discrimination/calibration decay over time.

What monitoring must do

Post-deployment monitoring quantifies these. Concretely, a deployed system needs:

  • continuous metrics on (sampled, audited) real outputs, tracked over time against a reference where obtainable;
  • drift detectors on the input distribution (has the case mix/scanner mix shifted?);
  • alerting on performance or calibration decay, with defined thresholds;
  • versioning and provenance — which model version produced which result, when, at which site, so a later failure can be traced (Ch. 5);
  • rollback — the ability to revert to a prior version when a new one degrades.

This is where IHE AIRA (AI Result Assessment for Imaging, Ch. 5) lives: it is the standards-level support for assessment of and feedback about AI results — quality assurance, human acceptance of AI outputs, monitoring, post-deployment performance surveillance, and drift detection. Without monitoring (and ideally AIRA-style feedback), silent degradation can persist until harm occurs.

Model updates restart validation

Changing the model after deployment creates a new validation/governance problem: a new version is a new model, needing its own (at least targeted) validation, and the deployment path (lesson 7) applies again, scaled to the change. Continuous/online learning (updating from live data) is especially hard: the training distribution becomes the live clinical stream, blurring the train/test boundary and raising leakage and stability concerns (Ch. 4). Versioning must distinguish which model produced which historical result, so prior results stay attributable to the version that made them.

Stop and think — then reveal

A deployed lung-CT triage model has stable monthly AUROC for a year, then sensitivity drops 8% in month 13 with no code change. Name two drift sources that could explain it and what monitoring should have flagged earlier.

(1) Protocol/scanner drift — the site upgraded its CT reconstruction kernel or scanner software in month 13, changing the texture distribution the model was trained on (Ch. 6). (2) Population/prevalence drift — the case mix or disease prevalence shifted, so the same operating threshold now has different sensitivity (lesson 5). Monitoring should have flagged the input drift earlier — a change in the scanner-mix or case-mix distribution, or a calibration shift — before it showed up as an AUROC/sensitivity drop, so the team could investigate before performance degraded. AUROC alone is a lagging indicator; input-drift detection is the leading one.

What to retain

  1. Deployment is the start of validation. The world drifts (data, protocol/scanner, population, label/practice), producing performance drift.
  2. Monitoring needs continuous metrics, drift detectors, alerting, versioning/provenance, and rollback.
  3. IHE AIRA is the standards-level support for AI-result assessment/feedback/monitoring.
  4. Model updates restart validation (scaled to the change); continuous learning blurs the train/test boundary — versioning must keep historical results attributable.

Next: put the chapter together — what evidence would make you trust this system?