Skip to content
Imaging Evidence Path

Why a correct AI result can still be clinically invisible

Tonight · ~20 min · read · energy: low · setup: none

This is the lesson engineers underestimate: the model can be perfectly correct and the result can still change nothing about care, because the informatics around it failed. This lesson is the failure-mode catalogue of deployment — the substance of why this chapter exists.

The anchor: correctness is necessary, not sufficient

A model that segments the tumour correctly, measures it correctly, and stores it correctly at the protocol level can still be invisible to the radiologist, attached to the wrong patient, or unauditable. None of those are model errors — they are informatics errors, and they are the failure mode you meet in real deployments. The model “works”; the care does not change. Diagnosing these requires the workflow vocabulary of lesson 1 and the transport vocabulary of lessons 2–4.

The failure-mode catalogue

Diagnose these for any real pipeline:

  • Silent resampling — a service resamples and reports on a grid the clinician never sees, so a measurement refers to a volume the reader cannot inspect.
  • Identity mismatch / wrong patient — a derived object attached to the wrong study/patient. A top failure mode and a safety issue.
  • Lost provenance — no record of model/version/inputs; a later failure cannot be traced to its cause.
  • Worklist collision — two orders/studies confused at the worklist boundary.
  • AI result not retrievable — the SEG/SR exists but the reading workstation cannot query or display it (the transport-independent capability failure of lesson 4).
  • No audit trail — impossible to reconstruct events after the fact.
  • De-id leaks — PHI survives into a “de-identified” export.

A worked diagnosis

A site reports “the AI is deployed but the radiologists ignore it”. Walk the catalogue:

  1. Can the viewer retrieve the result? If the workstation cannot render DICOM SEG, the result is invisible regardless of correctness — capability failure.
  2. Is the result linked to the right study? If a routing/identity bug attached it elsewhere, the radiologist never sees it in context — identity failure.
  3. Is the result timely? If it lands after the read is final, it cannot change the decision — workflow failure.
  4. Does anyone trust it? If provenance is absent, readers cannot judge what produced it and may rationally discount it — provenance failure.

Often the model is exonerated and an informatics step is the culprit. This is why “deployed AI” papers must be read with the failure-mode catalogue in hand: a claim of clinical impact needs evidence that the result actually reached and influenced the reader, not just that the model scored well.

The connection to deployment lifecycle

These failures do not stop at go-live; they evolve. Provenance gaps become drift mysteries (Ch. 9); viewer-capability gaps resurface when object types change; identity rules break when sites merge systems. The IHE profiles of lesson 8 exist precisely to address identity, retrieval and provenance at the standards level — which is why “which IHE profiles does it conform to?” is a fair question for any “deployed AI” claim (often the answer is none).

Stop and think — then reveal

A vendor demo shows an AI producing beautiful segmentations live. Six months after deployment, radiologists say they “never see it”. The model’s offline metrics are unchanged and excellent. What three informatics investigations should you run before touching the model?

  1. Retrieval/display — can the reading workstations actually query and render the object type the AI stores? (Capability failure is the most common cause of “never see it”.)
  2. Routing and timing — is the result linked to the correct study, and does it arrive before the read is final? A correct, late, mis-linked result changes no decision.
  3. Provenance and trust — is there a record of what produced each result, and do readers know how to judge it? A result readers rationally distrust is a result that has no effect. Only after these are exonerated does re-examining the model make sense.

What to retain

  1. A correct model can be clinically invisible: identity, retrieval, timing, provenance and workflow failures each negate a correct output.
  2. Failure-mode catalogue: silent resampling, identity mismatch, lost provenance, worklist collision, not-retrievable, no audit trail, de-id leaks.
  3. Diagnose “deployed but ignored” by checking viewer capability, study linkage, timing, and provenance before blaming the model.
  4. These failures evolve after go-live; IHE profiles (AIW-I/AIR/AIRA) address them at the standards level — ask which profiles a “deployed AI” conforms to.

Next: the single most important return-object distinction — structured result vs screenshot.