Reader-alone and reader-plus-AI studies

The standalone model and the human who uses its output are different experimental objects. A model can rank cases well against a reference and still add little when a radiologist is already reading. It can also change the reader’s attention or trust in a way that introduces errors. The question “does adding this AI help?” therefore needs a comparison of the reader alone with the same reader using the AI, on a defined set of cases and for a defined endpoint. A standalone AUROC cannot supply that comparison.

The question is a delta, not a score

The useful quantity is not a model score in isolation but a difference between two conditions in which the reader, cases, reference definition, and endpoint are made comparable. The exact estimand may be a change in sensitivity or specificity, a change in an agreement measure, a difference in reading time, or another prespecified workflow outcome. A positive delta in one outcome does not automatically imply a positive delta in another, and none of these deltas by itself establishes patient benefit or clinical utility.

Designing the comparison

A reader study begins by specifying the intended role and the unit of the claim. The CLAIM 2024 reporting guidance is a useful prompt, not a guarantee that one design is sufficient. A multi-reader, multi-case design is often useful when the question concerns performance across readers and cases: it provides more than one reader and more than one case from which to estimate variability. The reader and case structure must then be reflected in the analysis; simply averaging a collection of scores can hide which source of variation the study observed. A single-reader study can still be descriptive or exploratory, but it supports a narrower generalisation than a design that samples the reader population more broadly.

One possible design is a crossover, in which each reader encounters both the unaided and AI-assisted conditions. This can make within-reader comparisons useful, but it also creates a memory problem when the same case is seen more than once. A washout period may reduce recall, although its adequacy depends on the cases, readers, task, and interval. The order of conditions can also matter: seeing an AI suggestion first may prime a later unaided read, while starting unaided may alter how a reader attends to the assisted condition. Counterbalancing or another prespecified strategy can address order effects, but neither crossover nor washout guarantees an unbiased or otherwise adequate study.

The outcomes must follow the claim. Sensitivity, specificity, discrimination, and calibration answer different questions. A claim about efficiency requires an efficiency endpoint such as reading time measured under the relevant conditions. A claim about consistency requires an agreement or variability analysis. A study that claims an interaction is safe should examine what happens on model-error cases, how often readers override suggestions, and whether alerts change attention or workload. These are not a fixed list that every reader study must report; they are examples of the questions that become necessary when the intended claim includes them.

Crossover, washout, and order

The design details matter because the AI condition is an intervention on the read, not just another measurement of the same model. If a reader sees a case and its suggestion and then performs the supposedly unaided read soon afterward, memory can contaminate the comparison. If all readers receive the conditions in one order, the observed difference may combine an AI effect with learning, fatigue, or case-order effects. A study should make these threats visible and explain how its chosen design and analysis address them. The appropriate interval, order scheme, reference construction, and outcome model are context-dependent; they cannot be replaced by a universal recipe.

The trap of standalone-then-deploy

The deployment overclaim is to treat a strong standalone result as proof that readers will improve. The model may identify findings readers already identify, so the reader-plus-AI result changes little. Conversely, a confident wrong suggestion may lead a reader away from an answer they would otherwise have reached. The direction and size of the interaction are empirical questions. A well-designed reader study can address them, but no single design label settles every deployment claim.

Consider a constructed vendor study with one expert reader and 40 cases. The reader reads each case with AI and the results are compared with that reader’s earlier unaided reads, without a memory-reducing interval or a plan that varies condition order. A reported sensitivity increase would be difficult to interpret: reader-level generalisability is narrow, repeated exposure can contaminate the comparison, and order effects are not addressed. A stronger design might include several readers and cases, both conditions, a prespecified order and washout strategy, an appropriate reference standard, and outcomes matched to the intended role. Those features improve the evidence; they do not turn the resulting study into universal proof of benefit.

The next section applies this reading discipline to an explicitly bounded paper example, keeping its reported evidence separate from the design principles themselves.