The human can become part of the failure mode

Tonight · ~20 min · read · energy: low · setup: none

Engineers tend to think of the model as the thing that can fail. In deployment, the human+model system is what fails, and the human is often where the failure mode lives. This lesson is the human factors that decide whether a “working” model improves or harms care.

The anchor: the unit of deployment is the human+model system

A model does not act alone in most roles (lesson 2); it acts through a radiologist who reads with it, trusts it partially, and works under time and cognitive pressure. That interaction can create failures that neither the model nor the reader would make alone. These are not soft concerns — they determine whether a model helps or harms.

The human-side failure modes

• Automation bias — trusting the AI more than warranted, especially when it is usually right. Readers accept wrong suggestions they would otherwise have caught.
• Alert fatigue — too many notifications → real alerts ignored. A high-FP triage layer (lesson 5) can desensitise staff to true positives.
• Context blindness — the AI sees only the image; the clinician’s wider context (priors, symptoms, labs) is absent from the model, so the model’s confidence can be misplaced.
• Cognitive offloading — readers stop doing the mental work the AI claims to do; skills erode and rare-but-important findings get missed.
• Failure to notice incorrect AI suggestions — a confident-but-wrong AI output is harder to override than no output, because the reader has to actively disagree with an authoritative-looking suggestion.
• Trust calibration — users must learn when to trust the model and when not; miscalibrated trust (over- or under-) is the human-side failure mode. A reader study should probe whether AI errors become reader errors (lesson 3).

Why this changes the evaluation

A deployment evaluation that measures only the model’s accuracy misses these entirely. The right questions are about the combined system: do reader+AI errors cluster on the model’s errors (automation bias)? Does alert volume degrade attention (alert fatigue)? Do readers notice when the model is wrong (overridability)? These are measurable — but only if the evaluation includes the human, which is why a reader study (lessons 3–4) and shadow deployment (lesson 7) are not optional polish.

Stop and think — then reveal

A model is 95% correct. After six months of use, the radiology team’s unaided sensitivity on the model’s 5% error cases appears to drop. Name the likely human-side mechanism and what would mitigate it.

Automation bias / cognitive offloading: because the model is usually right, readers have stopped independently checking the cases it flags as negative, so on the rare cases the model is wrong they now miss what they would previously have caught — the model’s errors have become reader errors. Mitigations: trust-calibration training (teaching when to distrust), intermittent unaided reading or audits to keep skills sharp, surfacing model uncertainty (Ch. 4) so readers know when to override, and monitoring reader+AI errors for clustering on model errors. The point: a 95%-accurate model can reduce the system’s safety on its own error cases if the human side is ignored.

What to retain

1. The unit of deployment is the human+model system; the human is often where the failure mode lives.
2. Human-side modes: automation bias, alert fatigue, context blindness, cognitive offloading, failure to override wrong suggestions, miscalibrated trust.
3. These are measurable — but only if the evaluation includes the human (reader study, shadow deployment).
4. A usually-right model can reduce system safety on its own error cases via automation bias — monitor for it.

Next: how to catch these before they reach patients — shadow deployment before clinical deployment.