Frozen and adapted testing

The word “external” describes where data came from. It does not, by itself, describe whether the model encountered those data as an untouched evaluation. A new site may supply a legitimate development partition for adaptation and a separate test partition for evaluation. It may also supply labels that were used to choose a threshold on the very test set later reported. Those designs have different meanings even though both can be called external validation.

Data origin is not test independence

Before accepting an external result, trace the information boundary. Were the weights, feature-selection decisions, preprocessing state, hyperparameters, threshold, and any learned segmentation or representation fixed before the test data were used? If not, identify the new-domain data that were used for development and separate them from the data that supplied the final estimate.

The boundary applies to more than the final model. Patient, scan, and slice overlap can carry identity or anatomy across folds. Scalers, imputers, feature selection, PCA, harmonisation, and other transformations can carry cohort information if fitted using held-out observations. Hyperparameters and operating thresholds can be selected with test outcomes. A segmentation model whose training patients overlap the downstream test patients can also transmit information before the classifier is evaluated. The exact repair depends on the pipeline and estimand, but the reader should be able to follow every data-dependent state to the observations that were allowed to fit it.

A transformation that was genuinely fixed before the study need not be fitted inside every fold, but the paper should still state its origin and purpose. “Fixed” is not a licence to hide a design choice made after looking at the test results. Conversely, a transformation fitted on a new-domain development partition is adaptation rather than automatically invalid leakage, provided the final test partition remains untouched for the reported estimate.

Internal, temporal, and external tests

For this chapter, internal, temporal, and external are operational labels, not a universal taxonomy. An internal test is held out from the same source process as development. It can estimate performance for the sampled source conditions when the split and fitted-state boundary are protected. A temporal test uses later data from a source process and can expose changes over time, but it does not test every spatial or protocol difference. An external test uses an independent cohort or source process and can probe transport along the differences that are actually present: site, vendor, field strength, protocol, referral mix, prevalence, annotation process, or calendar period. Prediction-model literature may call temporal validation a form of external validation, whereas CLAIM and CLEAR reserve external testing for data from another institution.

No label is a universal strength ranking. A later cohort can be independent but narrow; a different site can share a protocol; an external cohort can be adapted before its test data are used. Whatever nomenclature is used, the paper’s transportability claim must name the tested axes rather than treating “external” as a synonym for generalisation everywhere.

Frozen travel and adapted transfer

There are two distinct questions. Frozen testing asks whether the original model travels: the weights, preprocessing state, and operating threshold are held as declared while the model meets the new cohort. Adapted testing asks whether the method can be transferred after the new domain has been seen. It may involve retraining, fine-tuning, recalibration, or choosing a new threshold. An adapted result can be practically valuable, but it does not measure raw travel of the original model.

Thresholds deserve the same attention as weights. AUROC is threshold-free, but sensitivity, specificity, accuracy, predictive values, and decisions depend on an operating point. A threshold selected on an external development partition can be part of a declared adaptation procedure. A threshold selected on the external test partition uses the reported evaluation outcomes and cannot be described as a pre-specified untouched operating point. Reporting a frozen AUC alongside an adapted threshold-dependent result is clearer than merging both under one external-validation label.

A constructed external-validation example

Suppose a constructed report says “external-validation AUROC 0.92,” then states that the model was fine-tuned on an external training split and that the threshold was chosen on the external test split. The number may describe performance of an adapted method, but it does not establish that the original frozen model travels. It also does not establish performance at a pre-specified threshold, because the operating point was selected using the reported test outcomes. The value 0.92 is an illustrative token, not a study result.

If the threshold had instead been chosen on the external training or validation partition and the test partition remained untouched, the operating-point result could be an honest estimate for the adapted procedure. It would still not be a frozen-travel estimate. The right description is therefore procedural: “frozen model tested as-is,” “method fine-tuned on a new-domain development partition,” or “threshold selected on the test data,” followed by the cohort and metric.

The claim follows the procedure

Only an otherwise valid, sufficiently precise, and relevant evaluation can support the corresponding statement. An internal result may support a within-source statement. A temporal result may support a statement about the tested later period and its observed changes. A frozen external result may support travel to the tested external conditions. An adapted external result may support transfer after the declared exposure and adaptation. The procedure type limits the maximum defensible scope, but does not by itself license even that bounded claim: sampling, reference standard, leakage control, analysis, and representativeness must also be adequate. None of these procedures alone supports an unbounded claim across hospitals, protocols, populations, or future time.

Once the evaluation procedure is known, the reader can finally ask whether the reported metric answers the decision described in the conclusion. Independence protects a number from one class of optimism; it does not make discrimination equivalent to calibration, utility, or clinical benefit.