Reconstruct the study before accepting its story

The first pass through a paper is often shaped by its story: an important problem, a novel model, a striking result, and a conclusion that seems to join them neatly. The second pass should reverse that direction. Treat the paper as an account of a study whose parts must be recovered and connected. The purpose of reconstruction is not to produce a longer abstract. It is to expose the conditions under which each result was obtained.

A paper is a story built on a study

The design is usually distributed across the abstract, methods, results, tables, supplements, and sometimes a data-availability statement. Those parts are written in a narrative order, so a reader can reach the conclusion before noticing that the sample was selected from a narrow referral pathway, that the label came from a process dependent on the image, or that the reported external test followed adaptation. Reconstructing the study makes those dependencies explicit.

Start by distinguishing three populations that a paper may compress into one. The target population is the people, examinations, lesions, or future encounters to which the authors want the result to apply. The source population is the accessible frame from which observations could be obtained: particular centres, services, dates, referral routes, and eligibility rules. The analytic sample is what remained after exclusions, missing-data rules, quality filters, and any availability of the reference standard. A result can be internally consistent in the analytic sample and still have limited transportability to the target population because the source or sample differs from it.

Cohort construction is consequently part of the result, not administrative prelude. Record how participants or examinations entered the study, how cases and non-cases were identified, which exclusions were applied, and whether the sample was enriched for a condition. Excluding technically poor or ambiguous cases may be defensible for a narrowly defined task, but it can also remove the cases that make the task difficult. A case-control or referral sample can alter the spectrum and prevalence encountered by the model. Neither situation is automatically invalid; both narrow the population and operating conditions that the result describes.

The word estimand keeps this distinction from becoming vague. It names the precise quantity the study is trying to learn for a specified population and unit. In a prediction study, the prediction target is the outcome or quantity to be predicted; the label or reference standard is the measurement process used to represent that outcome. The estimator and evaluation procedure describe how the target is estimated or assessed. A prediction estimand might concern ranking or risk for eligible examinations at a specified time horizon. An association estimand might describe a relationship between a feature and an observed outcome after stated adjustments. A causal estimand would describe a contrast between defined interventions or exposures under explicit assumptions. These are different questions even when they use the same image and the same regression or neural network.

The reconstruction paragraph

The useful shorthand is a six-part study-question and claim-reconstruction template, not a formal definition of an estimand. It places the target estimand or prediction target alongside the population, inputs, label/reference measurement, estimator/evaluation procedure, and resulting claim so that their relationships can be audited. Each part has an expansion that prevents a consequential decision from hiding inside a familiar word.

1. Question and estimand. Is the study detecting, diagnosing, predicting, prognosing, segmenting, measuring, or describing an association? State the outcome, time horizon, intended use, and quantity being estimated rather than accepting a broad phrase such as “clinical prediction.”
2. Population and cohort construction. Separate target population, source population, and analytic sample. Include inclusion and exclusion rules, recruitment or referral path, dates, sites, missingness, and any enrichment that changes the case spectrum.
3. Inputs and observation unit. Say what the model actually sees: an examination, series, reconstruction, phase, mask, clinical variable, or combination. State whether one row represents a patient, lesion, scan, slice, or another object, and identify repeated or clustered observations.
4. Outcome and reference standard. Name the label being predicted and the process that produced it. A report-derived label, reader judgement, pathology result, consensus, or algorithmic annotation does not carry the same interpretation.
5. Split and evaluation procedure. Describe training, validation, and test construction, the split unit, fitted transformations, threshold selection, external-domain development, and whether the evaluated model and operating point were frozen.
6. Claim. Reproduce the strongest conclusion in a form that names the tested population, procedure, metric, and missing evidence. Replace an unqualified word such as “generalizable” with the specific transfer the design actually examined.

This paragraph is also where an observational design must be kept separate from a causal story. A retrospective association can reflect confounding, selection, measurement, or a variable that is downstream of the condition. A predictive model can rank outcomes well while using a proxy that would not respond to an intervention. Unless the study identifies a causal question, an appropriate design, and the assumptions needed for that question, predictive performance is not evidence that an exposure or feature caused the outcome.

Gaps are findings, not invitations to guess

If the paragraph cannot be written from the paper, mark the missing element. An unclear unit prevents the reader from knowing what the denominator means. An unclear reference standard prevents a disease claim from being separated from a reader-reproduction claim. An unclear split prevents an independence claim. An unclear fitted-state boundary prevents a clean evaluation claim. These gaps do not prove that the hidden procedure was incorrect, but they prevent the reader from granting the stronger interpretation.

This is why the reconstruction precedes detailed metric appraisal. Asking whether a test was independent presupposes that the reader knows what was used for development. Asking whether a metric supports a clinical decision presupposes a defined outcome, operating point, and target population. The paragraph is a gate on the reader’s own confidence, not a pass/fail score assigned to the authors.

A compact paper preview

The paper identified by Tran et al. (PMID 35726103) is a useful preview of the method. Its stated problem is ACL-tear detection from knee MRI. The source describes an examination-level model, report-derived labels processed with an NLP model, a patient-disjoint development split, and two external datasets. It also uses language about external validation whose meaning depends on whether the model and threshold were frozen or adapted. Those nouns are enough to generate the next questions before any headline number is accepted: what is an examination in the analysis, what does the report-derived label represent, what entered each external test, and which result belongs to which procedure?

The full case later in the chapter supplies the paper’s reported numerical details. Keeping the preview compact avoids making the case a second abstract; the point here is that a reconstruction exposes the audit hooks before the reader is persuaded by the result.

The next hook is the one most likely to make a reported N look larger than the independent evidence: the relation between observation unit and split unit.