Transportability and testing claims

Where was the test set born?

The word “validated” hides the relationship between development and evaluation. Before interpreting a score, reconstruct where the test observations came from, which units were held out, which acquisition or clinical axes changed, and what the fitted procedure was allowed to do on arrival. A held-out group from the same source can answer a useful within-source question. It does not automatically answer a question about another hospital, a later protocol, or a different population.

The relevant comparison is not a simple ladder in which every external-looking test is stronger. A test may be independent on one axis and nearly identical on another. Its claim is scoped by the variation it actually represents and by the adaptation permitted before scoring.

Three test settings, three scoped claims

Internal testing evaluates held-out observations from the development source under a declared split. It can provide evidence about performance for new units from that source and design. It does not by itself establish behavior at another site or after a change in acquisition.

Temporal testing uses later observations from a source whose earlier data contributed to development. It can probe changes over time in protocol, scanner mix, referral patterns, labels, or workflow. The time boundary matters, but it does not guarantee that every relevant change has been represented.

External testing evaluates a cohort independent of the development data under the stated relationship between the two settings. A different institution or scanner is a common way to create such a test, but the phrase alone does not describe the full shift. A test can differ by site while sharing vendor, protocol, population, and reconstruction. The paper must say what was actually new and how independence was protected.

The term “validation” may be used for tuning, internal testing, or external testing in different papers. Replacing it with the precise test design makes the claim inspectable.

Domain shift is a set of axes

An imaging input is shaped by acquisition and reconstruction as well as by the patient. Relevant axes can include institution, scanner, vendor, reconstruction kernel or iterative method, slice thickness, dose, contrast phase, field strength for MRI, population, prevalence, reader workflow, and time period. A shift on one axis may interact with another. It is therefore safer to describe the axes explicitly than to treat “external” as a checkbox. This is a non-exhaustive list: the axes that matter depend on the measurement, outcome, and intended claim.

Axis	Question for the test data	Why the answer changes the claim
patients	Are the individuals new, and how does their case mix compare?	It defines the immediate prediction population.
time	Were the acquisitions or labels collected later?	Protocol, scanner, and workflow can drift.
institution	Did referral, care pathway, or reader practice change?	The task and spectrum may differ.
scanner and vendor	Did the device or manufacturer change?	Acquisition and reconstruction fingerprints can change.
reconstruction and protocol	Did kernel, dose, contrast, spacing, or phase change?	The measurement entering the model may change.
prevalence and population	Is the event frequency or disease spectrum different?	Calibration and predictive values can move even when ranking is stable.

The absence of a tested axis does not license a claim across that axis. A study that changes site but not protocol has tested one relationship; a study that changes time but not institution has tested another. The width of the wording must follow the width of the observed shift.

What was allowed to change on arrival?

Transportability also depends on the fitted state. State the target data each operation may use. In a frozen external test, the model and its preprocessing state are applied without learning from target observations. This tests travel of the declared procedure. A recalibration uses a declared target sample to refit a probability intercept or slope; if target outcomes are used, they are part of that fitted state and the evaluation must be separate. Threshold re-selection changes the operating policy, so the target information and decision rule used to set it must be reported. Fine-tuning updates selected model parameters using declared target-domain data, while complete retraining fits a new model using the source and/or target data declared by the protocol; it may retain only the recipe. For each operation, report whether target inputs, labels, outcomes, or only other permitted information were available.

These designs can all be legitimate. They are not interchangeable evidence. A number obtained after target-domain adaptation is evidence about the adapted procedure, not a frozen-model travel test. If a target split is used both to adapt and to score without a further held-out evaluation, the result is an in-sample check of the adapted state rather than independent testing.

The retained claim-scope schematic is intended to make this separation visible:

Testing type, shift axes, adaptation, and claim strength are distinct dimensions, not one simple hierarchy.

What each test licenses

An internal result can support a scoped within-source prediction claim. A temporal result can add evidence about the tested period change at that source. An external result can support a claim about the independent setting and axes it actually varied. None establishes universal generalisation. A model that does not transport may still be useful after a declared adaptation, but the adaptation is part of the clinical and statistical design rather than a footnote.

When reading a paper that reports an external AUROC, ask first whether the model was frozen or retrained, then which site, scanner, protocol, population, and time axes changed. Also ask whether thresholds and calibration were transported or re-estimated. The answer determines whether the number describes raw travel, adaptation, or a new model.

The same claim boundary applies to survival models, where censoring and the choice of comparison metric add another layer to the validation design.

Sources

• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ. 2024;384:e074820. PMC10788734.