Independent observations and clustered data

Rows are not automatically observations

A medical-imaging record is often hierarchical. One patient may contribute a volume, several scans, many slices, multiple lesions, or repeated measurements. Those rows can be useful computational units without being independent observations for the question being asked. Treating every derived row as a new patient changes the effective sample size, the uncertainty calculation, and possibly the target of prediction.

Consider a tumour represented by 60 axial slices. Neighbouring slices share anatomy, acquisition conditions, and often the same label. If slice 47 enters training and slice 48 enters the test fold, the model has encountered almost the same anatomical instance on both sides of the boundary. A high score from that design may describe recognition of correlated patient-specific structure rather than performance for a new patient. The example is a construction for exposing the split problem, not a universal prediction about how much a score must change.

Split unit versus analysis unit

Three units should be written separately in a protocol. This three-unit taxonomy is the organizing framework used in this chapter; it is not a universal taxonomy that dictates one split for every question.

• the analysis unit, whose outcome or measurement enters the estimand;
• the split unit, whose related observations must travel together when the evaluation is intended to represent a new unit; and
• the resampling unit, which must preserve the dependence structure when an uncertainty procedure resamples the data.

Reporting guidance for prediction models makes the target, data source, participants, performance, and intended use explicit. Cluster-specific guidance discusses clusters such as sites or studies. Repeated observations within one person need the same dependence-aware reasoning, but the grouping decision remains conditional on the estimand and the claim rather than being a universal patient-level rule.

For a patient-level diagnosis, all three may be the patient. For a lesion-level question, the lesion may be the analysis unit while lesions from the same patient remain a cluster for splitting and inference. For a longitudinal question, a scan may be an analysis record but repeated scans from one patient are not automatically independent. A slice-level model can be appropriate for a task whose output is explicitly slice-level, but a patient-disjoint test is still needed when the intended claim concerns unseen patients.

The patient is therefore a common split unit, not a law that replaces the estimand. If the estimand concerns within-patient change, separating baseline and follow-up observations may destroy the very contrast of interest. If the estimand concerns a model’s performance on new patients, separating those observations can make the evaluation optimistic. The design must state which case it represents rather than using the word “patient-level” as a substitute for reasoning.

Dependence has several visible shapes

The simplest safe construction for a new-patient prediction keeps all slices, lesions, scans, and test-retest repeats from one patient in the same partition. That rule prevents the evaluation from borrowing related anatomy from the training side. A grouped split does not by itself solve every problem: the model may still learn a scanner, site, protocol, or acquisition-time cue that is associated with the outcome.

The RIDER-style test-retest setting makes the grouping issue concrete. The two same-day scans are useful for studying repeatability, but putting one repeat in training and the other in testing would evaluate recognition of the same patient’s anatomy, not travel to a new patient. Keeping the pair together is appropriate for a new-patient prediction claim. A repeatability estimand may instead compare the two scans within a patient, provided that the analysis says so and does not present that comparison as independent-patient validation.

Site and scanner are additional axes rather than alternative definitions of the patient. A random patient split within one site can estimate performance for held-out patients from that source. Holding out a site or acquisition period asks a different transport question. Neither is automatically the stronger design for every estimand; each changes the population and variation represented by the test.

Leakage and shortcut learning are different diagnoses

Leakage is an information-boundary failure. Information from the evaluation world, including overlapping patients or test-fold relationships, influences preprocessing, selection, tuning, or fitting. It can make an internal estimate optimistic even when the split labels look correct.

A shortcut, confound, or site-dependent cue is different. The evaluation may be honestly held out, while the model uses a signal that is correlated with the outcome but is not the intended biological or clinical relationship. A scanner fingerprint, referral pathway, or acquisition-era marker can be predictive in one study and fragile after the setting changes. External or temporal testing can expose that fragility, but it does not repair leakage that occurred during development.

The two failures can coexist, and their repairs are not interchangeable. First establish which observations may cross the boundary. Then inspect which sources of variation the model can use. This separation carries into every later section: the fitted-state boundary addresses leakage, while transportability and measurement characterization address signals that may not travel.

As a deliberately simple construction, suppose 30 patients each contribute 20 slices, producing 600 computational rows. A random five-fold split of those rows can place slices from one patient in both training and testing, even if an AUROC of 0.92 is reported. The number cannot be interpreted as new-patient performance without knowing the patient grouping. The relevant sample size for that claim is the number of patients, while the slice rows remain useful inputs to a grouped analysis.

With the split unit made explicit, the next question is whether the model’s fitted state has also remained inside the same boundary. Leakage can enter even when every patient appears in only one fold.

Sources

• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Debray TPA, et al. TRIPOD-Cluster explanation and elaboration. BMJ. 2023;380:e071058. doi:10.1136/bmj-2022-071058.
• Bernett J, et al. Guiding questions to avoid data leakage in biological machine learning applications. Nature Methods. 2024;21:1444-1453. doi:10.1038/s41592-024-02362-y.