Fitted-state and leakage boundaries

The test set is a stranger, not a collaborator

Leakage is best understood as a fitted-state problem. A transformation or model has a state when data determine parameters, selected variables, thresholds, weights, or other choices that will be used later. During an evaluation, that state must be learned from the permitted training world. The held-out world can be transformed by the frozen state and scored, but it must not help create the state that will be evaluated on it.

This is more precise than the short rule “do not fit on the test set.” In an imaging study, the boundary can be crossed by a scaling statistic, an imputation model, a discretisation threshold, a selected feature set, a hyperparameter choice, a harmonisation model, or a segmentation model whose training patients overlap the evaluation patients. The table may contain no obvious test label, yet the inputs can already have been shaped by the evaluation world.

The leakage catalogue

The audit should name the state produced by each step and the data allowed to fit it. Cohort-fitted preprocessing uses information from more than one sample: means and standard deviations for scaling, imputation parameters, a PCA basis, feature-selection statistics, or a batch-effect model. These states belong inside the relevant training portion of the validation design and are then applied to the held-out portion.

Feature selection is a particularly easy boundary to miss. Selecting the top features from the full cohort before cross-validation uses the held-out rows’ feature-outcome relationships, even if the final regression is fitted separately in each fold. Hyperparameter search has the same structure. If many candidate configurations are compared on the folds that are later reported, the selection has read the evaluation scores. The nested design in the next section separates those operations.

Segmentation creates another boundary. A mask generated by a model trained on overlapping evaluation patients is not an independent input merely because the downstream feature extractor never saw the outcome. The provenance of the mask, including the data used to train the segmentation model and any target-informed postprocessing, is part of the predictive procedure.

Here, target-informed segmentation processing means using information from the evaluation target, such as its patient membership, labels, outcomes, or held-out images, to train, tune, correct, threshold, or filter a mask. If a segmentation model was trained on patients overlapping the evaluation patients, or if postprocessing used evaluation information, the mask pipeline has carried information across the boundary and evaluation independence is compromised. A segmenter can be a fixed component only when its training provenance and relation to the evaluation patients are declared; “fixed” describes how it is applied, not automatically how it was trained. This component-level accounting applies the information-boundary and reporting questions from leakage, risk-of-bias, and radiomics-reporting guidance; those sources do not by themselves validate a particular segmentation model.

The phrase “fixed preprocessing” also needs care. A per-sample operation with constants declared before the evaluation may not estimate a cohort statistic; that is different from choosing the constants after inspecting the complete dataset or outcome. A design choice can be fixed at deployment and still be test-informed if it was selected by looking at the held-out data. The relevant question is not only whether a library fitted a parameter, but whether evaluation information influenced a choice that affects the reported procedure.

A pipeline walkthrough

Consider this deliberately compact radiomics pipeline:

CT + mask -> fixed intensity handling -> fixed resampling -> feature extraction
   -> feature scaling -> feature selection -> tuned model -> held-out score

The first operations may be fixed per-sample transformations if their constants and implementation were specified independently of the evaluation data. The scaler, feature-selection rule, and tuned model produce or depend on fitted states. They must be fitted using the permitted training data in each validation unit. If a harmonisation or imputation model is used, it receives the same treatment. The held-out rows are transformed by the resulting state and used for the score only.

The distinction is visible in the retained leakage-boundary schematic:

D_tr and D_te are the training and held-out partitions. In the clean variant, g fits a state from D_tr before applying it to either partition and f is trained on the transformed training rows. In the contaminated variant, the held-out rows help determine the state used to score them. The numerical probe shown in the existing schematic is an illustrative construction of that boundary, not a universal effect size or direction.

The implementation detail matters because the result describes the whole evaluated procedure, including where its fitted states were learned. Moving a data-dependent operation outside the loop changes that procedure; an estimate produced by the contaminated procedure may not be a valid estimate for the intended training-only procedure.

Leakage and shortcuts remain separate

Leakage addresses the process by which information entered development. A shortcut addresses what information the model used after development. A model can be leakage-free and still depend on a site marker that will not travel. It can also use the intended signal while reporting an optimistic score because a scaler or selector read the held-out rows. External testing cannot retroactively make a leaked internal estimate valid, and fold-wise fitting cannot show that a shortcut transports to another setting.

For a simple audit, write a row for every operation and answer four questions: what data can it read, what state or choice does it produce, where is that state stored, and at which boundary may it be applied? The answer should be available before the score is interpreted. This audit turns leakage from a vague warning into a traceable property of the pipeline.

The next section applies the same information rule to the more subtle case in which the model itself is selected by a search: the loop that chooses must be inside the loop that estimates performance.

Sources

• Bernett J, et al. Guiding questions to avoid data leakage in biological machine learning applications. Nature Methods. 2024;21:1444-1453. doi:10.1038/s41592-024-02362-y.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505.
• Kocak B, et al. CLEAR. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.