Case study: reading a reader study

This case applies the reader-study distinction to Herpe et al. The evidence available for this authoring pass is metadata and the paper’s abstract. The full PDF was not retrievable at write time, so every paper-specific statement below is explicitly abstract-level. In particular, this section does not infer session order, counterbalancing, detailed statistical analysis, endpoint construction, or limitations that are not established by the accessible source.

The paper

Herpe et al. - AI assistance impact on radiologist interpretation of knee MRI, Eur Radiol 2026;36(2):1294-1305, PMID 40745051 (peer-reviewed; abstract inspected - full PDF not retrievable at write time, so the facts below are abstract-level).

The design (abstract-level)

The abstract describes a retrospective two-session reader study conducted in January-April 2024. It reports six radiologists with 2-10 years of musculoskeletal experience and 165 knee MRIs. It states that the MRI studies were evaluated with and without AI assistance by six radiologists in two sessions one month apart. It does not state whether each reader saw the same cases in both conditions. The abstract also reports that the AI model was trained on 23,074 studies separate from the study dataset, and that the reference was based on consensus from three expert musculoskeletal radiologists. The lesions named are the ACL, meniscus, cartilage, and MCL.

Those statements describe what the abstract reports; they do not reconstruct the study’s full protocol. The accessible evidence does not establish which condition came first, whether condition order was counterbalanced, how the interval functioned for each endpoint, how the statistical analysis handled readers and cases, or how the reference consensus was constructed. Those questions remain open until the full text is inspected. They must not be filled in from a familiar reader-study template.

The reported result

The abstract reports sensitivity of 81% to 86%, specificity of 88% to 93%, and accuracy of 86% to 91% for the two reported reading conditions, with p<0.001 for the reported comparisons. It also reports inter-reader agreement measured by Fleiss’ kappa changing from 0.54 to 0.78, with p<0.001. These are the abstract-reported values. This authoring pass does not recreate their denominators, intervals, analysis, or estimand from those values alone.

The bounded interpretation is that, on the retrospective case set and under the conditions described by the abstract, the reported reader metrics differed between the two conditions, including the reported agreement measure. That is evidence for a retrospective reader comparison with a stated result. It is not evidence by itself of a live workflow effect, a reading-time or efficiency benefit, patient outcomes, or clinical utility. The source material available here does not provide enough evidence to make any of those claims. A clean reader-plus-AI difference also does not tell us how readers behave under local workload, how results are retrieved and displayed, or how the system changes as the data distribution drifts.

Reading the result with the lens

The abstract supplies a reader count, a case count, two-session wording without an asserted complete per-reader case pairing, an interval, a separately described training set, a consensus reference, and several reported outcomes. It does not supply the protocol detail needed to call the design a particular order-balanced crossover or to claim that a particular washout strategy controlled memory. Nor does the phrase “efficiency” establish a reading-time result when no reading-time metric is available in the accessible evidence. The safe reading is therefore deliberately narrower than a purchase or deployment recommendation.

Imagine, as a constructed contrast, that a hospital cites this paper to justify a promise to speed up reporting. The abstract-level values may motivate a question about reader performance and agreement, but they do not support a speed claim. That claim would require a study with an appropriate time or workflow endpoint in a relevant setting. This is the evidence-to-claim discipline from Chapter 9 applied to deployment: preserve the result, preserve its evidence level, and do not let a related word in an abstract carry a stronger conclusion than the reported endpoint supports.

The case also shows why reader evidence is one stage of the lifecycle rather than its endpoint. Even if the abstract-level comparison is later confirmed in full text, the next questions concern the operating point, human factors, local integration, staged evaluation, monitoring, and governance. The following section starts with the operating point.