Chapter 4 — Statistical learning and validation

This chapter · ~2.5 h total across 8 lessons · The medical-imaging translation of statistical learning · Energy: low to medium · Setup: optional Python for the survival smoke test

This chapter does not teach statistical learning. You are working through StanfordOnline Statistical Learning with Python (ISLP; James, Witten, Hastie, Tibshirani, Taylor, Springer 2023) — that course owns the general ML statistics. This chapter owns the medical-imaging translation: which ISLP ideas bite hardest when the rows are patient-derived correlated images, and what ISLP does not prepare you for.

The thread this chapter pulls

ISLP examples are i.i.d. rows in a table. Medical images are not. The same patient appears as many correlated slices; a feature table has hundreds of columns on tens of patients; a held-out test set can still leak through preprocessing; a great AUROC can be a clinically useless model. Every one of those is an imaging-specific failure that a general ML course mentions in passing and that radiomics papers get wrong routinely. This chapter is the methodology that stands between a feature table (from Chapter 3) and an honest clinical claim.

Lessons

1. What is the independent observation? — patient vs lesion vs scan vs slice.
2. How leakage enters an imaging pipeline — preprocessing, feature selection, segmentation, repeated scans.
3. Why model selection belongs inside validation — nested CV, with a small numerical sketch.
4. p >> n: why radiomics breaks ordinary intuition — regularisation, instability, selection.
5. A high AUROC can still be a poor clinical model — discrimination, calibration, uncertainty, decision utility.
6. Internal, temporal and external testing are different claims — domain shift, site/scanner effects.
7. Survival models in an imaging project — censoring, Cox, C-index, value beyond the clinical baseline.
8. Build the validation design before fitting the model — a worked TRACE-CT-style design.

The ISLP translation table (a map, not a substitute for the course)

Read each ISLP chapter against its imaging consequence. Skim what your CS background already covers; attend to what bites when the data are images.

ISLP topic	Attend to	Imaging consequence
Resampling / CV	k-fold, LOOCV, bias–variance of the CV choice	patient-level grouping; slice/scan leakage; nested CV; external/temporal testing
Linear / logistic regression	coefficient interpretation, interactions	the clinical tabular baseline to beat before any image model
Model selection & regularisation	the lasso path, one-SE rule	p >> n radiomics; selection must be inside CV
Trees / ensembles	RF correlation, GBM tuning, variable-importance caveats	strong tabular baselines beat naive deep models on heterogeneous clinical variables
Survival analysis	censoring, KM, Cox PH, proportional hazards, C-index	censored outcomes; Cox; “does imaging add value beyond baseline?”
Unsupervised	PCA interpretation, choosing k	batch/site/scanner effects masquerading as phenotypes
Multiple testing	p-value adjustment, FDR	hundreds of features → feature fishing → the radiomics reproducibility problem

If you only have an hour, lessons 1–2 (independence and leakage) and lesson 5 (AUROC is not enough) are the core that most radiomics papers get wrong.