Statistical learning and validation — reference

Reference · Dense lookup for the chapter’s imaging-specific material — leakage patterns, robustness placement, and the tabular missing-data / synthetic-data unit (TRACE-CT M1-07/M1-08, both MEASURED). General statistical learning lives in ISLP; teaching versions live in the chapter.

Independence & leakage

• Units: choose analysis, split, and resampling units for the intended claim; for patient-level prediction, related rows such as repeated scans travel together (Independent observations and clustered data).
• Leakage paths: preprocessing fit on all data · feature selection outside CV · segmentation pipelines seeing test labels · repeated-scan duplication (Fitted-state and leakage boundaries). Nested CV resolves selection-vs- assessment (Nested model selection).
• The measurement-layer version: features can carry site/scanner/kernel/ segmentation fingerprints the model legitimately learns — confounding / shortcut learning, a distinct failure mode from leakage (fix: design, stratification, robustness evidence, external testing — not just CV hygiene) (Chapter 5).

Missing data (M1-07, bounded case)

Mechanism	Meaning	Consequence
MCAR	missingness independent of everything	complete cases may form a random subsample under stated analysis conditions
MAR	missingness explained by observed data	principled imputation can work
MNAR	missingness depends on the missing value	imputation alone insufficient; sensitivity analysis

• Methods ladder: median/mode (train-only) → predictive (target-wise CatBoost: ordered boosting + categorical handling) → MICE (conditional models iterated; canonical = M datasets pooled via Rubin’s rules).
• TRACE-CT honesty label: sklearn IterativeImputer+BayesianRidge is a chained-equation proxy producing one completed table — not full multiple imputation; imputation uncertainty not estimated.
• Evaluation: artificial masking on observed cells (frozen seeds, identical masks across methods); never use outcomes as imputer targets/inputs; identifiers never predictors.
• Split discipline (frozen in M1-07; pinned public aggregate record): subject-level 70/15/15 (101/22/21), stratified event × gender; imputers fit on train only; membership local + SHA-256 committed.
• Measured: 60/60 runs, zero failures, held-out reconstruction + train-only dependency metrics per run.

Synthetic tabular data (M1-08, bounded case)

	CTGAN (NeurIPS 2019)	TabDDPM (ICML 2023)
Training	adversarial generator/discriminator	diffusion: forward noising + learned denoising
Mixed types	mode-specific normalization; training-by-sampling	Gaussian (continuous) + multinomial (categorical) noise
Subgroup rows	true conditional generation	unconditional + generation-order selection (never “conditional TabDDPM”)
Sampling cost	fast	reverse chain per row
TRACE-CT use	51 conditioned Female rows (deficit to 1:1)	pinned upstream commit; first-51 Female rows, cap 2000, yield recorded

• Chapter synthesis of separate appraisal questions (not a universal taxonomy): per-field fidelity (KS ≤ 0.10 / prevalence ≤ 5 pp challenge-style) + dependency preservation (Pearson/Spearman · Cramér’s V · correlation ratio η, kept separate) + downstream utility on untouched real test + disclosure risk indicators. No quality-filtering of generated rows; no real rows downsampled.
• Risk stack (indicators, never a privacy guarantee): Gower NN synthetic→train vs held-out-real→train; exact matches (all fields equal); near-duplicates (Gower ≤ 0.05); impossible combinations; subgroup distortion; reportability gates for subgroup metrics. Synthetic ≠ anonymised.
• Bounded project measurements (pinned public aggregate record, negatives retained): reference Cox C-index 0.789; TabDDPM-augmented 0.743–0.817 (CI-overlapping — no demonstrated gain); CTGAN conditional mapping unproven; no KPI1-C passes; elevated TabDDPM memorisation-risk signals 2/3 seeds. Seeds 1729/2718/3141, all reported.
• Generators ≠ per-cell imputers; no clinical-validity/privacy claim.

Survival evaluation

Right censoring → Kaplan–Meier (describe) · Cox PH (model; check proportional hazards) · C-index (discrimination only). Honest question: incremental value beyond the clinical baseline (clinical-only vs clinical+radiomics; LR test / ΔC-index with CI). Calibration: integrated Brier score + calibration slope/ intercept only with enough events (M1-08 gate: ≥ 10 events, else NOT_REPORTABLE with reason); subject-level bootstrap CIs; report every seed. Frozen M1-08 estimator: lifelines CoxPHFitter, penalizer = 0.1 (fixed before any held-out evaluation, never tuned).

p >> n and model honesty (chapter core)

Hundreds of correlated features on tens of patients: regularisation mandatory; selection inside validation; stability-as-description ≠ stability-as-selection (Chapter 6); AUROC is not enough — discrimination, calibration, uncertainty, decision utility (Discrimination, calibration, utility, and uncertainty); internal/temporal/external are different claims (Transportability and testing claims); design before fitting (Design before fitting).

References (the tabular/synthetic/survival unit)

• van Buuren S, Groothuis-Oudshoorn K. mice. J Stat Softw 2011;45(3):1-67 — doi:10.18637/jss.v045.i03.
• Austin PC, White IR, Lee DS, van Buuren S. Missing Data in Clinical Research: A Tutorial on Multiple Imputation. Can J Cardiol. 2021;37:1322-1331 — doi:10.1016/j.cjca.2020.11.010; PMC8499698.
• Prokhorenkova L, et al. CatBoost. NeurIPS 2018 — arXiv:1706.09516.
• Xu L, Skoularidou M, Cuesta-Infante A, Veeramachaneni K. Modeling Tabular Data using Conditional GAN. NeurIPS 2019 — arXiv:1907.00503.
• Kotelnikov A, Baranchuk D, Rubachev I, Babenko A. TabDDPM. ICML 2023, PMLR 202:17564-17579 — arXiv:2209.15421, PMLR page.
• Kaabachi B, Despraz J, Meurers T, et al. A scoping review of privacy and utility metrics in medical synthetic data. NPJ Digit Med. 2025;8:60 — doi:10.1038/s41746-024-01359-3, PMCID PMC11772694.
• TRACE-CT (read-only): the frozen M1-07/M1-08 configurations, metric results, and model card are inventoried in the TRACE-CT learning map. Status: M1-07 and M1-08 MEASURED; no clinical-validity, privacy, or challenge-data claim.