Skip to content
Imaging Evidence Path

Build the validation design before fitting the model

Tonight · ~30 min · read + a design exercise · energy: medium · setup: none

Everything in this chapter is a property of the design, not of the model. By the time you fit anything, the answer’s credibility is already mostly decided. This lesson is the worked design you build first — a TRACE-CT-style protocol — so that fitting the model is the last and easiest step.

The anchor: the protocol is the paper

A radiomics study’s honesty is determined by its design: how patients were split, what was fit inside the loop, what the test set’s relationship to training is, and whether the question asks for incremental value. Write that protocol down before fitting, and most methodology errors become impossible to commit by accident. This is why TRACE-CT’s docs/experiment-protocol.md exists as a fixed artefact, not as an afterthought.

The design checklist (build this first)

Walk these in order. Each maps to a lesson in this chapter.

  1. Estimand and unit. What is the question? What is the analysis unit, and what is the split unit? (Default split unit = patient — lesson 1.)
  2. Splits. Patient-disjoint train/val/test (or nested CV — lesson 3). No repeated scans or slices across the boundary. Pre-commit the split.
  3. Leakage audit. List every cohort-level step (scaling, feature selection, ComBat, imputation, PCA, segmentation source) and confirm each is fit inside the loop on training only — lesson 2.
  4. p >> n handling. Regularisation (lasso/ridge/elastic-net, one-SE rule), multiple-testing correction or pre-registered feature set, selection inside CV — lesson 4.
  5. Metrics beyond discrimination. Calibration, decision-curve/net-benefit, patient-level CIs — lesson 5.
  6. Testing type. Internal, temporal, and/or external; document the domain shift and whether the model is frozen or retrained — lesson 6.
  7. Baseline. A strong tabular/clinical baseline; the claim is incremental value — lesson 4, lesson 7.
  8. Reporting. Which guideline applies (CLAIM 2024 / TRIPOD+AI / CLEAR / PROBAST+AI) — Ch. 7.

A worked TRACE-CT-style design

Sketch the protocol for a plausible radiomics+survival question on NSCLC-Radiomics:

  • Question — does a lung-tumour radiomic feature add prognostic value for overall survival beyond stage and histology?
  • Unit — patient (split unit); the tumour is the analysis unit, one tumour per patient here.
  • Splits — patient-disjoint nested CV (outer 5, inner for lasso alpha and feature subset). RIDER test–retest patients stay in one fold.
  • Pipeline inside CV — CT → HU clip [-1000,400] (fixed, outside) → 1 mm B-spline (fixed, outside) → NN mask (fixed, outside) → PyRadiomics at binWidth 25 (fixed, outside) → robustness filter on train → scaler on train → lasso selection on train → fit. ComBat (if multi-site) fit on train only.
  • Model — Cox PH; check proportional hazards. Compare C-index and likelihood-ratio of clinical-only vs clinical+radiomics.
  • Metrics — C-index with CI, calibration of survival predictions, decision-curve if a threshold is relevant.
  • Testing — internal nested CV estimate; flag external validation as required before any clinical claim.
  • Reporting — CLEAR (radiomics) + TRIPOD+AI/PROBAST+AI (prediction model).

The point of writing this before fitting is that steps 3–4 (the leakage and p>>n discipline) are now structural — they cannot be forgotten because the protocol names them and fixes where each lives.

The transferable skill

This is the same lens Chapter 7 applies to reading a paper: you reconstruct the protocol from the methods and audit it against this checklist. Designing your own study and critiquing someone else’s are the same skill in opposite directions. Once the checklist is reflexive, a paper that reports a single AUROC from a whole-cohort feature selection on 60 patients reads as what it is — not as evidence.

Stop and think — then reveal

A collaborator proposes: “extract features, run a univariate screen on all patients, keep the significant ones, fit a Cox, report the C-index.” In one sentence, what is fatally wrong, and what single design change fixes the most of it?

The univariate screen on all patients leaks every test fold into feature selection, so the C-index is optimistically biased and the “significant” features are partly chance (lesson 2, lesson 4). The single most important fix: move the screen inside patient-disjoint nested CV (fit on outer-train only), so selection never sees the fold being scored — then add the clinical baseline and test for incremental value.

What to retain

  1. The study’s honesty is decided by the design, not the model. Write the protocol first.
  2. The design checklist: estimand/unit → patient-disjoint splits → leakage audit → p>>n handling → metrics beyond AUROC → testing type → clinical baseline → reporting guideline.
  3. Fix where each cohort-level step lives (inside vs outside CV) in the protocol — that single act prevents most methodology errors.
  4. Designing your own study and critiquing a paper are the same skill; this is the lens Chapter 7 applies in reverse.

You have finished Chapter 4. Continue to Chapter 5 — Imaging informatics, or jump to Chapter 7 — Research methodology to see this checklist applied to reading real papers.