Human factors and automation effects

Engineers often model failure as a property of the algorithm: the output is correct or incorrect, and the system is safe if the first category is large enough. In most clinical uses, that is not the unit that acts. A person sees an output, interprets it under time and cognitive pressure, and decides whether to accept, investigate, or override it. The relevant object is therefore a human-plus-model system. A model that performs well in isolation can change the reader’s attention and trust in ways that alter the combined result.

The human-plus-model system

A radiologist brings clinical context that may not be present in the model input: symptoms, prior examinations, laboratory results, referral questions, and knowledge of the patient’s circumstances. The model may bring a fast, consistent view of a defined image or measurement. Their combination is not automatically better than either component. A suggestion can focus attention on a relevant finding, but it can also make a wrong conclusion look more authoritative. An alert can surface a case, but too many alerts can make the queue harder to manage. These are causal changes to the workflow, not soft properties that can be inferred from a model’s accuracy.

Human-side failure modes

The following are a non-exhaustive analytic set, not a prevalence-ranked taxonomy. Their direction and magnitude depend on the task, interface, readers, and workflow.

Automation bias occurs when a reader gives an AI suggestion more weight than its evidence warrants. A usually-correct system can make its rare wrong suggestions difficult to challenge, especially when the reader is busy or the output is displayed as a definitive label. The important question is not only how often the model is wrong, but how often a reader notices and corrects those wrong outputs.

Alert fatigue is a workload problem. A triage threshold that produces many false positive alerts may be acceptable for one queue and disruptive for another. Repeated low-value notifications can compete with the alerts that matter, changing attention before anyone has measured the effect in the local workflow.

Context blindness arises when the model’s view is narrower than the clinician’s decision. A high score may describe image evidence while omitting a relevant symptom, prior study, or competing diagnosis. This does not make the score useless; it makes the boundary of its evidence part of the interface and the reader’s task.

Cognitive offloading can occur when a reader stops performing a check that the system appears to perform. Over time, a team may rely on a generated measurement or negative flag without maintaining the independent scrutiny required for rare or atypical cases. The direction and magnitude of this effect are empirical, so a deployment claim should not treat it as either inevitable or absent without observing the readers.

Finally, trust calibration is a two-sided problem. Under-trust can waste a useful signal; over-trust can turn model errors into reader errors. Training, interface design, uncertainty presentation, escalation rules, and opportunities to inspect the source image all influence that balance. They are parts of the intervention and must be described as such.

An explanation is an intervention, not a safety mechanism

An explanation is often described as if its presence made an output safer. That is not a property that follows from the word explainable. An explanation changes what a reader attends to and how persuasive the suggestion feels. Its effect depends on the quality and format of the rationale, the correctness of the suggestion, and the reader’s ability to appraise it. A plausible rationale can increase appropriate trust when the suggestion is right and increase adherence to a wrong suggestion when it is not. A weak rationale may reduce uptake, but that is not a general safety strategy either.

The source material for this chapter records a large randomized experiment on explanation format involving 2,020 vignette-level assessments. It reports a 12.2 percentage-point improvement in diagnostic accuracy (57.4% to 69.6%) for chain-of-thought explanations and up to 80% adherence to an incorrect language-model diagnosis when both the diagnosis and explanation were incorrect in the differential-diagnosis condition. That result is from the cited study, not a universal effect size or a deployment threshold (PMID 42026140; full text inspected in the source record). A related chest-radiograph randomized study is recorded as abstract-inspected and reports increased simple trust with local explanations without making correctness the only determinant (PMID 39560483). These evidence levels matter: they identify what was available for the claim and do not license a broader conclusion about every explanation interface.

The same caution applies to the reported thoracic reader-model study, for which the source record identifies the journal abstract but not an inspected full text (Song et al., Radiology 2026;320(2):e253397, doi:10.1148/radiol.253397). It is a pointer to a rationale-quality and expertise question, not a basis for a general rule. The durable lesson is narrower: explanation is an intervention on reader behaviour. It needs evaluation on correct and incorrect suggestions, with the reader and task defined, rather than a demonstration of attractive heatmaps or fluent text.

What the evaluation must observe

An evaluation that measures only the model’s output misses the failure modes created by the interaction. A reader study can examine whether reader-plus-AI decisions differ from reader-alone decisions, whether readers detect model-error cases, and whether the interface changes time, agreement, workload, or override behaviour when those are part of the claim. The design has to state which readers, cases, model role, and reference are involved. The relevant outcomes are not automatically the same as the standalone model metrics.

Shadow operation can add a different view by recording the model’s outputs while keeping them hidden from readers. It can expose alert volume, latency, missing results, case-association problems, and the distribution of model errors before the output is used to guide a decision. It cannot measure the reader effect while the output is hidden, and it does not make the overall operation risk-free; it is one stage in a larger evaluation.

Consider a constructed scenario in which a model is reported as correct on 95% of cases. If, after adoption, readers miss a larger share of the model’s wrong cases than they did before using it, the combined system has changed even if the standalone metric has not. Possible mechanisms include automation bias and cognitive offloading, but the mechanism should be tested rather than assumed. A local response might include trust-calibration training, periodic unaided audits, a way to inspect the source evidence, and monitoring of reader-plus-AI decisions on model-error cases. Those measures are hypotheses about mitigation until the relevant workflow shows that they work.

The unit of deployment is consequently the human-plus-model system, and the unit of evidence must match it. The next step is to observe that system in increasing contact with its real environment without treating a single transition as proof of safety.