Internal, temporal and external testing are different claims
Tonight · ~20 min · read · energy: low · setup: none
“Validated” is the most over-used word in imaging-AI papers, because it mixes several very different claims. This lesson separates them, so that when you read “the model was externally validated” you know exactly what was — and was not — tested, and what kind of generalisation that licenses.
The anchor: where was the test set born?
The strength of a performance claim depends on the relationship between the test set and the development world. A test set drawn from the same scanner pool as training supports one (narrow) claim; a test set from a hospital the model has never seen supports a much stronger one. Conflating them is how “validated” gets to mean almost nothing.
Three testing types, three claims
Prefer precise terms over the ambiguous word “validation”:
- Internal testing — performance on a held-out set from the same source (cohort/site/scanner). The lower bar; it tests memorisation/generalisation within the training distribution. Necessary, but says little about other sites.
- Temporal testing — held-out later data from the same source. Catches protocol/scanner drift over time: the model trained on 2021–2023 CTs may degrade on 2025 CTs because the acquisition pipeline shifted.
- External testing — performance on a genuinely independent cohort (different site/scanner). The stronger, more honest generalisation claim; the one most credible radiomics results rest on, and the one many lack.
(“Validation” is ambiguous — it can mean the tuning set or the test set — so say which.)
Domain shift is the rule, not the exception
A model trained on Siemens B70f lung-kernel CT may not behave the same on GE soft-kernel CT. The distribution of inputs at deployment differs from training — domain shift — and it is the default situation in medical imaging, not a rare edge case. Sources include scanner vendor, reconstruction kernel, slice thickness, dose, kVp, contrast phase, coil (MRI — Ch. 8), and population demographics. The discipline:
- Document the acquisition of both training and test data.
- Test the shift explicitly (hold out a site, a vendor, a time window).
- Name the shift when reporting — “generalisable across GE/Siemens at 1.5/3 T” is a real claim; “generalisable” with no scope is not.
This is exactly the frozen-vs-retrained distinction that matters in external validation: a model frozen and shipped to a new site is a different experiment from a model retrained/fine-tuned on the new site’s data. The Tran et al. dissection in Chapter 7 turns this into a worked example: the abstract’s flattering external numbers were partly retrained, and the frozen-travel numbers told a humbler story.
What each testing type licenses
| Testing type | Licenses a claim about… | Does not license… |
|---|---|---|
| Internal | within-distribution generalisation | performance at other sites |
| Temporal | robustness to drift at this source | performance at other sources |
| External | generalisation to an independent site | universal generalisation |
The pattern: each type licenses a scoped claim. The wider the claim, the wider the test must be. “Our model works” is never licensed by an internal test alone.
Stop and think — then reveal
A paper says “our model was validated on an independent external dataset and achieved AUROC 0.91”. Before you accept “generalisable”, what two facts about the external test must you establish?
- Was the model frozen or retrained on the external data? A frozen model that travels to the new site is a genuine generalisation test; a model fine-tuned on the external dataset’s own train split has already seen the new domain and the 0.91 measures transfer-after-adaptation, not raw travel.
- What is the scope of the external site (vendor, scanner, field strength, protocol, prevalence) relative to training? “External” from the same vendor and protocol as training is a narrow shift; “external” across vendor and field is a wide one. The licensed claim is only as wide as the actual shift tested.
What to retain
- Use precise terms: internal / temporal / external testing. “Validation” is ambiguous.
- Each type licenses a scoped claim; wider claims need wider tests.
- Domain shift (vendor, kernel, scanner, protocol, population) is the rule in imaging — document it, test it, name it.
- Distinguish frozen-travel from retrained external numbers; the abstract often reports the more flattering, retrained one.
Next: a special outcome type that needs its own validation — survival models in an imaging project.