Chapter 9 - Reading Evidence and Limiting Claims

A paper can report an impressive performance number while leaving the reader unable to say what was actually tested. The problem is not that the number is necessarily wrong. It is that a number has a meaning only in relation to a population, an observation unit, a label, a fitted pipeline, a test procedure, and a question. Remove those conditions and the number becomes a free-floating property of the model, ready to support a claim wider than the study can bear.

Reading evidence therefore starts in reverse. Instead of following the paper’s persuasive order from result to conclusion, work backwards from the conclusion to the design that would have to support it. Who was the intended target population? Which people or examinations actually entered the sample? What did one observation represent, and which observations were allowed to share a split? What label or reference standard stood in for the condition of interest? Which transformations were fitted, and where was their information allowed to flow? Was the test internal, temporal, external, frozen, or adapted? Finally, what quantity did the metric estimate, and what decision or claim is being attached to it?

This is the same validation discipline developed earlier for building a study, now applied while reading one. Chapter 7 developed independence, fitted-state boundaries, and testing designs. Chapter 8 showed why an imaging model inherits the geometry, acquisition, representation, and task definition of its inputs. Here those ideas become an appraisal method. The goal is not to turn every paper into an examination or to reject every imperfect retrospective study. It is to state precisely what the study does show, what it leaves uncertain, and what a stronger design would have to add.

Reconstruct the study before accepting its story

The central habit of this chapter is to write a short reconstruction before accepting a paper’s conclusion. A reconstruction is not a summary of the abstract. It is a compact account of the design in terms that make the claim auditable: question and estimand, target and sampled populations, observation and split units, inputs, outcome and reference standard, fitted-state boundary, test procedure, metric, and conclusion. When a paper does not provide one of these pieces, write not reported rather than silently supplying a plausible answer.

Consider the sentence, “the model achieved AUROC 0.94.” The number is useful only after several questions have answers. AUROC is threshold-independent: a threshold choice does not change the AUROC value. Its interpretation still depends on the observation unit, sampling frame, target population, and case mix. AUROC could describe ranking of patients, lesions, examinations, or slices. The outcome could be an independent clinical reference, a reader’s interpretation, or a label extracted from a report. The test could be a patient-disjoint hold-out or a random collection of neighbouring slices. A threshold may have been fixed in advance, selected in a development partition, or chosen on the reported test data. That choice changes sensitivity, specificity, predictive values, and downstream action claims at the operating point, not AUROC. The same numerical token can therefore accompany different estimands and different claims. The value 0.94 in this paragraph is a constructed illustration, not a reported study result.

The chapter follows the reconstruction in the order in which its decisions become visible. First, rebuild the cohort and its units. Then inspect how the reference standard was made. Next, determine whether the evaluation boundary was protected and whether an external test was frozen or adapted. Only then interpret discrimination, calibration, utility, and uncertainty. Reporting and risk-of-bias frameworks help name omissions, but they do not turn a completed checklist into proof. The final section puts the whole method through a dated, source-identified paper case and separates the paper’s reported facts from the conclusions this chapter is entitled to draw from them.

The result is a narrower but more durable reading skill. A retrospective study may provide a useful association or a useful prediction without establishing a cause, a biological mechanism, or a clinical benefit. A model may travel to one tested cohort without being transportable to every hospital. A missing method may prevent a strong claim without proving that the unreported procedure was bad. Evidence is not weakened by stating these boundaries; it becomes legible.

The first step is to make the study skeleton explicit.