Chapter 7 — Statistical Learning and Validation

Validation begins with the claim

Chapter 6 ended at the boundary between a characterised measurement and a model. A feature table can be reproducible under a declared image, mask, geometry, and extraction contract, yet a model fitted to that table can still answer a different question from the one a paper claims. The validation problem is therefore not simply to choose a score or split a table. It is to make the question, the fitted procedure, the evaluation design, and the licensed claim line up.

The first object to write down is the estimand: the quantity or comparison the study intends to learn. In a prediction study this may be expected performance for a specified population, outcome definition, prediction time, and deployment setting. In an association or explanatory study it may be a coefficient or contrast under a stated statistical model. In a measurement study it may be agreement under a declared perturbation. The same rows can support different estimands, but changing the estimand changes what counts as an appropriate observation, split, uncertainty calculation, and interpretation.

The second object is the fitted procedure, not just a named algorithm. A procedure includes the input definition, preprocessing, feature selection, hyperparameter search, model fitting, threshold choice, and any recalibration or adaptation. Some of these steps produce a state from data: a mean and standard deviation, an imputation model, a selected feature set, a fitted coefficient vector, or a threshold chosen by optimisation. That state has an information boundary. The evaluation design must prevent the held-out world from shaping it.

The third object is the evaluation design. It specifies the unit that is allowed to cross a split, how repeated observations are grouped, where each data-dependent step is fitted, what test setting is being represented, and how uncertainty is attached to the result. A result from patient-disjoint nested cross-validation answers a different question from a result obtained after slice-level randomisation, whole-cohort feature selection, or retraining on a target site. The metric name cannot erase those differences.

Only then can the study state its claim. A discrimination estimate is not automatically a calibration claim. A calibrated prediction is not automatically useful in a decision. An association is not a causal effect, and a model that travels to one new setting has not demonstrated universal transportability. Prediction and inference may use related data and models, but they have different targets and different failure modes. Performance and clinical utility are related, but utility requires a decision context, threshold or policy, and consequences for errors.

The chapter develops these boundaries in order. It starts with the observation and split unit, because a patient-derived table is usually a clustered data structure rather than a collection of independent rows. It then follows information through fitted preprocessing and nested selection, before turning to the p >> n regime common in radiomics. The performance section separates discrimination, calibration, utility, and uncertainty. It distinguishes an interval for an aggregate performance object, uncertainty in a calibration relation, uncertainty in an estimated individual risk, and a prediction set or interval for a future outcome; none can stand in for the others. The later sections examine transport across settings, time-to-event outcomes, and protocol-first design.

The final two sections make the same reasoning concrete in dated, bounded TRACE-CT-derived cases. The missing-data case concerns a method comparison in a clinical table, not clinical validity. The synthetic-data case separates fidelity, downstream utility, and disclosure-risk indicators rather than compressing them into a claim that synthetic data are safe. Project chronology, exact paths, and fast-changing tool details remain supporting or companion material; the reader-facing argument stands without access to the laboratory.

The bridge from Chapter 6 is deliberate: once a measurement enters a model, the measurement contract becomes one input to the estimand rather than a substitute for validation. The next section asks the most basic design question: what is one observation, and which unit is allowed to appear on both sides of an evaluation boundary?

Sources

• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Debray TPA, et al. TRIPOD-Cluster explanation and elaboration. BMJ. 2023;380:e071058. doi:10.1136/bmj-2022-071058.
• James G, Witten D, Hastie T, Tibshirani R, Taylor J. An Introduction to Statistical Learning with Applications in Python. Springer; 2023. The book is a general statistical-learning reference; this chapter develops the imaging-specific consequences of its ideas rather than replacing it.