Skip to content
Imaging Evidence Path

AI-alone and radiologist+AI are different experiments

Tonight · ~20 min · read · energy: low · setup: none

This is the methodological heart of deployment evaluation: a model’s standalone performance and its effect on a reader are different experiments, and only the second is a fair test of “does adding this AI help?”. This lesson is the reader-study design that answers it.

The anchor: the question is a delta, not a score

“Does the AI help?” means: does radiologist+AI do better than radiologist-alone, on the same cases, measured honestly? A standalone AUROC cannot answer that, because a model that is excellent alone may add nothing once a radiologist is already reading (the radiologist already catches what the model catches), or may even hurt (by misleading the reader on cases they would have got right). The relevant measurement is the delta in a reader study.

The reader-study design

A reader study measures radiologist performance with vs without AI on a defined case set. The concepts to demand:

  • Reader-alone vs reader+AI — the core comparison; the delta is the result.
  • Multi-reader, multi-case (MRMC) — many readers × many cases, to separate reader-level from case-level effects (the standard design; ROC/AUROC analysed per reader and averaged). Without multiple readers, you cannot tell a reader-specific effect from a general one.
  • Crossover design — each reader reads each case both unaided and with AI, separated by a washout period to reduce memory/order effects.
  • Order effects — reading with AI first can prime the unaided read (or vice versa); washout and counterbalancing mitigate them.
  • Outcomes reported — sensitivity, specificity, AUROC, reading time, and inter-reader variation. A claim of “efficiency” needs a reading-time metric; a claim of “consistency” needs inter-reader agreement.
  • Overreliance / automation bias — readers accept wrong AI suggestions or stop looking carefully; a reader study should probe whether AI errors become reader errors.
flowchart LR
    R["readers"] -->|"session 1"| A["unaided read"]
    R -->|"washout"| B["AI-assisted read"]
    A --> D["delta: sensitivity,<br/>specificity, AUROC, time,<br/>inter-reader agreement"]
    B --> D

Why crossover and washout matter

If a reader sees a case with AI and then unaided (no washout), they remember the case and the AI’s answer — so the “unaided” read is contaminated by memory, biasing the delta toward “no difference”. Crossover (each reader does both conditions) with a washout interval (long enough to reduce memory) and counterbalancing (order of conditions varied across readers) is what makes the delta interpretable. A reader study without these is a weaker design.

The trap of standalone-then-deploy

The most common deployment overclaim is inferring “it helps radiologists” from “it scores well alone”. The two are not just different — they can contradict. A model can raise standalone AUROC and leave reader+AI unchanged (the reader already did as well), or even lower it (automation bias on the model’s errors). Only an MRMC crossover reader study settles it; the next lesson walks a real one (Herpe et al.).

Stop and think — then reveal

A vendor shows a reader study: one expert reader, 40 cases, read once with AI and the results compared to that same reader’s earlier unaided reads on the same cases (no washout, no counterbalancing). They report a sensitivity gain. Why is this not a credible reader+AI claim, and what design would be?

Three flaws: (1) a single reader cannot separate a reader-specific effect from a general one (no MRMC); (2) no washout means the unaided reads are contaminated by memory of the cases/AI answers, biasing the delta; (3) no counterbalancing leaves order effects uncontrolled. A credible design is MRMC crossover: multiple readers of varying experience, each reading the same case set both unaided and with AI in counterbalanced order separated by a washout, reporting sensitivity/specificity/AUROC, reading time, and inter-reader agreement, against an independent reference standard. Only that design supports a “reader+AI helps” claim.

What to retain

  1. “Does the AI help?” is a delta (reader-alone vs reader+AI), not a standalone score — standalone AUROC cannot answer it.
  2. Demand MRMC crossover with washout: many readers × many cases, each reader in both conditions, counterbalanced, with an independent reference standard.
  3. Report sensitivity, specificity, AUROC, reading time, and inter-reader agreement — and probe automation bias.
  4. A model can raise standalone AUROC and leave reader+AI unchanged or worse; only a reader study settles it.

Next: read a real reader-study result — Herpe et al.