The observation unit and the split unit

A paper’s rows do not tell you by themselves how many independent opportunities to generalise were tested. One patient may contribute multiple lesions, scans, series, or slices. Those rows can be useful observations for a particular analysis, but they are not automatically independent patients. The reader has to identify both the unit at which the outcome is analysed and the unit that must remain together across a development and evaluation boundary.

One row is not automatically one independent observation

The analysis unit is the object for which the study defines an outcome or measurement. It may be a patient-level diagnosis, a lesion-level prediction, a scan in a longitudinal series, or a slice used by a within-examination model. The choice changes the estimand. A lesion model can ask about lesions while its uncertainty and deployment claim concern new patients. A scan-level model can ask about change over time while its observations remain repeated measurements within a person.

The split unit is the unit whose shared information must not cross the training, validation, and test boundaries for the intended claim. For a new-patient claim, this is often the patient. All of that patient’s lesions, slices, repeated scans, and derived objects should then travel together. For a different estimand, a more specialised grouping may be appropriate, but the paper must state why the grouping matches the claim.

This distinction is different from the effective sample size. Counting rows does not automatically count independent information. Patient count is often closer to the independent information scale than row count, but it is not a universal effective-sample-size formula for every clustered estimator. The honest description states the row-level analysis, the grouping structure, the split unit, and the method used for uncertainty or aggregation.

Analysis unit, split unit, and estimand

For patient-level diagnosis, patient is commonly both the analysis and split unit. For lesion-level prediction, lesion can remain the analysis unit while patient is the split unit. A lesion-weighted target is legitimate when declared, but it answers a lesion-weighted question; patient-level claims and uncertainty require patient-level aggregation or cluster-aware analysis. For longitudinal work, scan can be the analysis unit, but the repeated-measures structure and the time ordering must be represented. Slice-level training can be a valid engineering choice, yet a slice-randomised test cannot support a new-patient claim when neighbouring or repeated slices cross the boundary.

The most useful reading question is therefore not “was the split patient-level?” in isolation. It is: “What unit does this metric describe, what unit does the claim concern, and what shared information was kept together?” A patient-level AUROC obtained after aggregating lesion scores answers a different question from a lesion-level AUROC that weights a patient with many lesions more heavily. Neither is automatically right; the estimand and denominator must be visible.

A constructed example: 600 images and 800 lesions

Consider a deliberately constructed dataset described as 600 images. Suppose those rows are 30 patients with 20 slices through each lesion. If the intended claim is performance on unseen patients, the independent grouping is closer to 30 patient clusters than to 600 unrelated images. The number 600 remains a valid count of image rows for a slice-level operation, but it is not by itself the number of independent patient tests. The resulting metric could change after patient-level splitting, and its uncertainty could widen; the direction and size of the change are properties of the data and analysis, not a theorem that AUROC must fall.

A second constructed example contains 800 lesions from 40 patients and applies an independent 80/20 split to lesions. In the balanced simplification, each patient contributes about 20 lesions. The probability that a patient has at least one lesion on the test side is 1 - 0.8^20, so the expected test-side representation is 40 * (1 - 0.8^20) = 39.54, or about 39.5 patients. The probability that a patient has lesions on both sides is 1 - 0.8^20 - 0.2^20, so the expected number represented on both sides is 40 * (1 - 0.8^20 - 0.2^20) = 39.54, again nearly all 40 patients. About eight test patients would instead describe a patient-level 80/20 split, which keeps every lesion from a patient together. After that change, the metric, its denominator, and its interval should be recomputed. It is entirely possible for the result to change in either direction, although a split that removes shared patient cues may reveal optimism in a particular dataset.

Leakage and shortcut learning are different diagnoses

When correlated observations cross an evaluation boundary, the problem is procedural leakage: development has access to information that the intended test should represent as unseen. A different problem occurs when the split is honest but the model uses a site marker, scanner fingerprint, acquisition protocol, or other proxy associated with the outcome. In ordinary prediction terminology, that is a dataset shortcut, a spurious association, or a site/workflow correlate. Such a variable may be genuinely available at prediction time yet fail to transport under a domain change without having leaked test rows into training. The two problems can coexist, but fixing one does not diagnose the other.

For a test-retest design, both scans of one patient usually belong to the same fold when the claim concerns new patients. Otherwise the model may learn from one scan and be evaluated on another view of the same anatomy. For a longitudinal estimand about within-patient change, the analysis can deliberately use both timepoints, but the repeated structure, temporal direction, and generalisation target must be stated instead of hidden by treating the rows as unrelated.

The unit and split decision now expose another dependency: what exactly is the outcome that each row is being compared with? That outcome is a reference process, not a label floating free of its origin.