How leakage enters an imaging pipeline

Tonight · ~25 min · read · energy: medium · setup: none

“Don’t fit on the test set” is the version of leakage you learn in a tabular course. In imaging, leakage has far more doors: preprocessing statistics, feature selection, hyperparameter tuning, even the segmentation model. This lesson is a catalogue of the doors, so you can walk a pipeline and name each one.

The anchor: the test set is a stranger, not a collaborator

The principle behind all leakage is simple: nothing computed using the test data may influence the model. The test set must be a stranger the model meets only at evaluation. In a tabular course that means “don’t fit on the test rows”. In imaging it means also: don’t compute any cohort-level statistic the model depends on using test rows, don’t select features using test rows, don’t tune hyperparameters on test rows, and don’t let a pretrained segmentation that saw your test patients produce the masks. Each of those is a door leakage walks through.

The leakage catalogue

• Preprocessing leakage — fit any cohort-level statistic (feature selection, imputation, scaling, discretisation thresholds, PCA) inside the CV loop. The subtle distinction: deterministic per-sample image preprocessing (a fixed HU clip, a fixed isotropic resample) may stay outside, because it uses no cohort statistics. A scaler’s mean/std, a PCA basis, or a ComBat batch model does use cohort statistics — so it must be fit on training data only and applied to the held-out fold.
• Feature-selection leakage — selecting features using the full cohort before CV is one of the most common radiomics optimism sources. The selection “saw” the test fold’s feature–outcome relationships.
• Hyperparameter leakage — model/CV hyperparameters tuned on the outer test set. Countermeasure: nested CV (an inner loop for selection/tuning, an outer loop for honest performance) — lesson 3.
• Target leakage via segmentation — if the segmentation model was trained on data that overlaps your test patients, the mask itself leaks. The feature table is “clean” but its inputs are not. Check the segmentation model’s training data against your test set.

A worked walkthrough

Take a radiomics pipeline and audit it for the four doors:

CT + mask → HU clip [-1000,400]   → 1mm resample   → PyRadiomics features
   → z-score each feature         → select top-20 by univariate test
   → fit lasso (alpha tuned by inner CV) → evaluate AUROC on held-out fold

Step	Uses cohort stats?	Where it must live
HU clip [-1000,400]	No (fixed constants)	outside CV is fine
1 mm resample	No (fixed target)	outside CV is fine
z-score (mean/std per feature)	Yes	inside CV, fit on train
top-20 univariate selection	Yes	inside CV, on train
lasso alpha (inner CV)	Yes (inner CV only)	inside CV
AUROC on held-out fold	—	the evaluation

The first two are safe outside because they are fixed, deterministic, per-sample operations. The next three use cohort statistics and must be fit on training data only within each fold. Move any of them outside and the test fold has shaped the model.

The sharp version of the rule: if a step uses more than one sample’s data to decide anything about the model, it belongs inside the validation loop. Fixed per-sample transforms are the only thing that may live outside.

Leakage vs shortcut, revisited

Recall from lesson 1: leakage makes numbers optimistic; shortcuts make models fragile. The catalogue above is all leakage — fixable by moving steps inside CV. A model with no leakage can still be a shortcut (e.g. it learned a site marker from an honestly held-out test set) — that is a generalisation problem, fixed by external testing, not a CV discipline problem. Both must be checked; neither excuses the other.

Stop and think — then reveal

You fit a z-score scaler on all patients before cross-validation, then run 5-fold CV and report AUROC. Where exactly did test information enter the pipeline?

At the scaler. The mean and standard deviation of each feature were computed using every patient, including those in each test fold. The model therefore trained on features whose scale was set partly by test data — a small but real optimism injection, and a leak. The fix: inside each fold, compute the scaler’s mean/std on the training partition only, apply that transform to the held-out fold, then evaluate. The same logic applies to feature selection, ComBat, PCA, and imputation: fit on train, apply to test, inside the loop.

What to retain

1. Principle: nothing computed using test data may influence the model. The test set is a stranger.
2. Four doors: preprocessing statistics, feature selection, hyperparameter tuning, and segmentation training overlap. Audit a pipeline for all four.
3. Fixed per-sample transforms (HU clip, fixed resample) may stay outside CV; anything using cohort statistics goes inside.
4. Leakage inflates numbers (fix by CV discipline); shortcuts make models fragile (fix by external testing). Both must be checked.

Next: the structural answer to hyperparameter and selection leakage — nested cross-validation.