Skip to content
Imaging Evidence Path

Can this feature be trusted?

Tonight · ~25 min · read · energy: medium · setup: none

So everything changes a feature. Which features survive the changes that should leave them alone? That is the robustness question, and it splits into two halves that get conflated: repeatability (same scanner, same session) and reproducibility (across scanners and sites). This lesson gives you the distinction, the agreement statistics used to measure it, and the test–retest idea that makes RIDER so useful.

The anchor: a predictive feature that swings under no change is not a biomarker

Suppose a feature predicts survival in your cohort. Now you translate the mask by one voxel, or resample to a slightly different spacing, or the patient is scanned again the next day — and the feature value jumps. A feature that is unstable under perturbations the biology did not undergo cannot be a biomarker: its “signal” is partly the pipeline, not the tumour. Robustness analysis exists to find and filter those features out before modelling.

Repeatability vs reproducibility — not synonyms

Question Setting What it tests
Repeatability same patient, same scanner, same session (or same-day repeat) does the feature barely change when nothing changes?
Reproducibility across scanners / sites / protocols does the feature agree across different acquisition conditions?

Repeatability is the easier bar (everything held constant); reproducibility is the harder one and is usually where radiomics breaks. RIDER’s same-day test–retest lung CT scans measure repeatability — the patient is scanned, leaves, comes back, and is scanned again on the same scanner, so any feature difference is acquisition noise, not biology.

Agreement statistics, conceptually

  • ICC (intraclass correlation coefficient) — agreement of repeated measurements vs between-subject variability; used for inter-rater and test–retest agreement. High ICC ⇒ the feature ranks subjects consistently across repeats.
  • CCC (concordance correlation coefficient) — agreement on the line of identity (precision × accuracy). TRACE-CT uses CCC for repeat-extraction agreement, and it is used similarly for test–retest feature stability.

A common rule of thumb: keep features with ICC/CCC above a threshold (e.g. ≥ 0.85) as “stable”, and drop the rest. Two cautions:

  • The threshold is a convention, not a law; report the distribution, not just the kept/dropped count.
  • Do this selection inside cross-validation, or it leaks (Ch. 4).

Perturbation analysis

Test–retest data is not always available. The cheaper, always-available cousin is perturbation analysis: nudge the ROI, image, or parameters by a small amount and measure feature drift. Translate or rotate the mask by a voxel, vary the interpolation, vary binWidth slightly, perturb the ROI boundary. A feature that swings under a perturbation you cannot see is not a biomarker, however predictive in a model. Perturbation and test–retest agree on the principle: stress the feature with changes the biology did not undergo, and keep only what is stable.

flowchart LR
    F["feature battery"] --> R["repeatability<br/>(test-retest / ICC)"]
    F --> P["perturbation<br/>(nudge mask, params)"]
    R --> S{"stable?"}
    P --> S
    S -->|yes| KEEP["keep for modelling<br/>(inside CV)"]
    S -->|no| DROP["drop — pipeline noise,<br/>not signal"]

What robustness is not

Robustness is necessary but not sufficient. A perfectly repeatable feature can still be useless — it might be stable because it measures a scanner artefact rather than the tumour (a stable shortcut). And a feature can be reproducible across scanners yet not predictive. So the chain is: robust (stable) → not-leaking → predictive → externally validated → then a candidate biomarker. Skipping the first step puts pipeline noise into your model dressed as signal.

Stop and think — then reveal

You filter your 200 features down to 60 “stable” ones using test–retest ICC on the whole cohort, then run cross-validation on those 60. What is wrong with this ordering, and how would you fix it?

The filtering used the whole cohort before cross-validation, so the test folds influenced which features survived — that is feature-selection leakage, and it makes the cross-validated performance optimistic (see Ch. 4). The fix is to do the robustness filtering inside each CV fold (or on a separate training-only partition): in each fold, compute stability on the training data, select stable features, train, and only then evaluate on the held-out fold. The principle: any step that uses data to choose the model — including which features count as “stable” — belongs inside the validation loop.

What to retain

  1. Two questions, kept distinct: repeatability (same scanner/session — does it stay put?) and reproducibility (across scanners/sites — does it agree?).
  2. Measure agreement with ICC/CCC; RIDER same-day repeats measure repeatability.
  3. Perturbation (nudge mask/params) is the always-available cousin of test–retest; a feature that swings under a perturbation the biology did not undergo is not a biomarker.
  4. Robustness-filter inside CV; and remember robust ≠ predictive ≠ biomarker — it is the first gate, not the verdict.

Next: put it into practice — RIDER as a robustness experiment.