What is the independent observation?

Tonight · ~20 min · read · energy: low · setup: none

ISLP hands you a dataframe where each row is an independent observation. Medical imaging hands you a folder where one patient contributes a hundred nearly identical slices, two scans, three lesions. Treat each as a row and your N is a lie. This lesson is the single distinction that prevents the most common radiomics error: the split unit is usually the patient, even when the analysis unit is not.

The anchor: inflating N with correlated rows

A patient with one tumour appears in your data as, say, 60 axial slices. Those slices are not independent — neighbouring slices are nearly identical images of the same anatomy. If you split at the slice level, slice 47 of patient A lands in the training set and slice 48 (almost the same image) lands in the test set. The model has effectively seen the test image. Performance looks great; generalisation is an illusion. This is slice leakage, and it is the classic radiomics error.

Split unit vs analysis unit

The useful, precise distinction:

The independence / split unit is usually the patient, while the statistical unit of analysis depends on the estimand (the question being asked).

Patient-level diagnosis → analysis unit = patient; split unit = patient.
Lesion-level prediction → analysis unit = lesion, but multiple lesions per patient are clustered — the model must account for within-patient dependence, and the split unit stays the patient.
Scan-level / longitudinal analysis → analysis unit = scan (a repeated measure), with within-patient dependence modelled explicitly.
Slice-level training is technically valid as a modelling choice, but the train/test split must not leak a patient’s correlated slices across folds unless the design explicitly justifies it (e.g. a within-slice task evaluated patient-disjointly).

The error to avoid is conflating the two: treating each slice/scan as independent inflates N and leaks across the boundary.

The three leakage shapes this produces

Patient-level splitting — all data from one patient (both test–retest scans, all slices) goes in a single fold. The default rule.
Slice/scan leakage — the classic radiomics error: split at the slice level so neighbouring slices land on both sides.
Site/scanner effects — if a site’s protocol correlates with outcome, a within-site test is an easier, in-distribution bar; holding a whole site out is a stronger — and humbler — test.

The RIDER case makes the first two concrete: a RIDER subject has paired test–retest scans. Both repeats of one patient must stay in the same fold, or the model trains on one scan of a patient and tests on the other scan of the same patient — a glorified split-half of the same anatomy.

flowchart LR
    P["Patient A"] --> S1["scan 1<br/>(test-retest)"]
    P --> S2["scan 2<br/>(test-retest)"]
    S1 --> SL1["slices …"]
    S2 --> SL2["slices …"]
    SL1 --> F["same fold"]
    SL2 --> F
    P2["Patient B"] --> F2["another fold"]

Leakage vs shortcut — do not conflate

This will matter again in Chapter 7, so define it now. Two different failure modes that often coexist:

Leakage — information from the evaluation/test world influences model development (selection, preprocessing statistics, tuning, overlapping patients). It is a procedural error that makes reported performance optimistic even on the internal test set. Fix: methodological discipline.
Shortcut / confounding / site dependence — the model learns a signal correlated with outcome but not causally the biology you intend (a scanner watermark, a site marker, a corner pixel, a later-acquired cohort with better outcomes). The test set was honestly held out; the model simply exploits a non-generalising cue. Fix: external/temporal testing, confounder adjustment, bias audits.

A model can be leakage-free yet shortcut-driven (honest numbers, fragile model), or leakage-ridden yet shortcut-free (inflated numbers). Diagnose both, separately.

Stop and think — then reveal

You have 30 patients, each with one CT and one tumour mask. You extract one feature vector per slice (say 20 slices through each tumour), giving 600 rows, then run a 5-fold CV that shuffles the 600 rows and splits them. You report AUROC 0.92. Why is this number not credible?

Because the rows are not independent: slices from the same patient (and even neighbouring slices within a patient) are highly correlated, and a random shuffle puts slices of the same patient in both training and test folds. The model has effectively memorised patient-specific cues, so 0.92 mostly measures that leakage, not generalisation to unseen patients. The honest N here is 30 patients, not 600 slices; split at the patient level (all of patient A’s slices in one fold) and the AUROC will usually drop — often substantially.

What to retain

Split unit = patient is the default; the analysis unit depends on the estimand. Never split correlated slices/scans across the train/test boundary.
Conflating them inflates N and leaks — the classic radiomics error.
Leakage (procedural, optimistic numbers) ≠ shortcut (honest hold-out, fragile model). Diagnose both separately.
Test–retest scans of one patient (RIDER) belong in the same fold.

Next: with independence fixed, hunt the paths by which leakage sneaks into a pipeline beyond the obvious split.