Falsification, baselines, and negative controls

The previous case showed that repeatability and measurement performance do not establish construct validity. A model can perform well while using a different signal from the one its description names. It may use volume, shape, an intensity distribution, spatial organisation, an acquisition fingerprint, or some combination of these. A performance number alone cannot identify which one it read.

The useful mental model is to intervene on the supposed cause. If the claim is that a model succeeds because of spatial texture, construct a controlled representation in which spatial organisation is destroyed while other properties are preserved. If performance survives, the texture interpretation is weakened. If performance falls, the model needed spatial organisation, but that spatial organisation may still reflect acquisition or reconstruction rather than biology. The control narrows the interpretation; it does not supply causality by itself.

A scoped evidence boundary

Neither a cancer-wide proposition that volume is strongly prognostic across most solid cancers nor a family-wide proposition that radiomic features generally depend on volume is established by the evidence considered here. The bounded studies below cannot be aggregated into either universal claim. Their findings concern particular tumour types, modalities, feature definitions or families, volume ranges, endpoints, and validation designs.

Volume is therefore used here as a candidate baseline and possible alternative explanation, surrogate, or confounder in a bounded design. A study should test the relationship under its stated tumour type, modality, feature definition or family, volume range, endpoint, and validation design rather than treating it as a law of radiomics. In this chapter, terms such as volume dependence, surrogacy, and confounding describe relationships among measured variables; data leakage is reserved for information crossing the training/evaluation boundary.

This chapter’s falsification framework

This chapter uses three kinds of control as an organizing framework. They answer different questions; this is a chapter-level framework, not a claim that the field has one standard taxonomy. The distinction between sham or representation controls and label-permutation tests is discussed by Buvat and Orlhac.

A label-permutation null test permutes the outcome labels and reruns the whole pipeline. It is a pipeline and statistical sanity check. If performance does not fall as expected, leakage, a software defect, or another analysis problem may be inflating the result. A label null does not test whether the original model read texture.

A representation negative control alters the images or feature representation so that the information named in the interpretation cannot survive, while preserving the labels and the other properties the control is intended to preserve. Voxel shuffling within a region is one example, as in the bounded analysis by Welch et al.. A result that changes after this intervention tests the claimed interpretation more directly, although the surviving spatial signal still requires interpretation.

Within this framework, an ablation study removes or replaces model components, such as a feature family, input modality, or block of variables. It asks which component carries performance in that model. An ablation can establish a model-attribution result, but it cannot establish that the component measured the tumour rather than a confounder.

All three controls are useful. Only the representation control directly challenges a claim about what information the model used, and even that challenge is bounded by what the control preserved and destroyed.

The baseline ladder

Within the same chapter-level falsification framework, the baseline ladder is a prescriptive comparison design, not an empirical ranking or a universal field checklist. Before interpreting a complex signature, compare a set of models that isolate plausible sources of performance. The comparisons should use the same patient-level split assignments and the same leakage-safe treatment of every data-dependent step. CLEAR and TRIPOD+AI make such comparison and development/evaluation procedures reportable; they do not empirically validate the ladder or any model. The ladder is not a claim that one model is universally best; it is a way to make the incremental claim inspectable.

Comparison	What it helps test
Clinical-only	What the clinical record predicts without imaging features
Volume-only	Whether a size variable explains performance in this bounded design
Shape-only	What mask-derived geometry contributes without intensities
First-order-only	What the intensity distribution contributes without spatial relationships
Texture-only	What the spatial representation contributes without the other feature groups
Full signature	What the proposed combined feature set achieves
Full signature plus volume	Whether adding volume changes the result or adds information in this design

The clinical-only model is the reference when the claim concerns incremental value beyond variables already available in care. The volume-only, shape-only, and first-order-only models are alternative explanations, not universal biological laws. Texture-only isolates the representation named by a texture claim. The full signature shows the combined result, while full signature plus volume makes the role of volume visible rather than leaving it implicit.

The comparison is invalid if the full model receives a more favorable split, preprocessing path, feature-selection procedure, or tuning budget than the baselines. This is a model-development and reporting requirement, not an empirical result supplied by a guideline. A baseline ladder is part of the claim’s evidence, not a collection of informal sanity checks added after the preferred result is known.

A bounded published disagreement

The history of one published signature illustrates why these comparisons must remain scoped. In the original Aerts study, a four-feature survival signature was reported to outperform volume alone, and the signature combined with volume was reported to outperform volume alone (Aerts et al.). Those are claims about that signature, endpoint, cohort, feature definitions, and analysis.

A later bounded analysis re-fitted an Aerts-style four-feature survival model on the original lung and H&N data and then evaluated the model after voxel index values were randomly permuted within the images. The reported external C-index on real images was 0.64; after voxel randomisation it was 0.64 for H&N and 0.60 for lung, while tumour volume alone was 0.64 and correlated strongly with three of the four features (Welch et al.). Within those specific designs, the result weakens the interpretation that the signature’s performance depended on meaningful spatial texture. It does not prove that all radiomics signatures are volume-driven, nor does it resolve the broader evidence boundary.

The disagreement is also definition- and cohort-dependent. An earlier PET primary study reported that quantization, matrix construction, and tumour-volume range changed texture-volume relationships; in its tested NSCLC PET cohort, heterogeneity and volume were independent prognostic factors, whereas complementarity was weaker and non-significant in the esophageal cohort (Hatt et al. 2015). A later review discusses these issues (Hatt et al. 2019). Neither result overturns the lung and H&N analysis or establishes a universal correction. It shows why the feature definition must be frozen before two signatures are treated as the same object.

Other bounded studies add context without forming a cancer-wide conclusion. In a H&N CT/PET study, a clinical model containing volume and HPV was reported to outperform models that added radiomics features (Ger et al.). Work by Fave et al. reported that preprocessing choices and volume collinearity changed which features appeared prognostic (Fave et al.). These examples have different tumour, modality, endpoint, and feature-definition scopes. They should remain visible as separate pieces of evidence and disagreement, not be averaged into a universal statement about radiomics.

Voxel shuffling within the region

A direct control for a claim that performance comes from spatial organisation is to randomly permute voxel positions within the region of interest. This is the voxel-randomisation intervention used in the bounded analysis by Welch et al., but its informativeness depends on exactly which properties the control preserves and alters. The mask is kept exactly the same. Volume and shape are therefore preserved, as is the complete multiset of intensity values and its histogram. What is destroyed is the original spatial arrangement of those values inside the mask.

Synthetic construction, not patient data. A performance drop after shuffling shows the model was using spatial information — it does not prove biological texture or causality, since acquisition and reconstruction are spatially organised too. No drop does not show the signal is texture. Control runs need the same leakage-safe splits as the original model.

Under the fixed mask, first-order features that depend only on the voxel-value multiset remain unchanged. Texture features can change because their matrices count neighbouring pairs, runs, zones, dependencies, or neighbourhood differences. Shape features remain unchanged because the mask is unchanged. This is the intended asymmetry: the control preserves non-spatial properties while disrupting the spatial relationships that texture features use.

The control must be applied to the representation that the claim concerns. If the model uses filtered images, the effect of shuffling depends on when the shuffle occurs and whether the filter is recomputed. If the mask, preprocessing, or fitted model changes between the original and control runs, the comparison no longer isolates the intended representation. The original and control analyses need the same split structure, fitted preprocessing, feature-selection rules, and scoring procedure.

What a performance change establishes

A performance drop after in-region shuffling supports the narrower statement that the model used spatial organisation that the shuffle destroyed. It weakens the competing explanation that the preserved histogram, mask, or shape alone was sufficient for the observed performance. It does not establish that the spatial organisation was biological texture. CT acquisition and reconstruction also create spatially organised patterns, and segmentation boundaries can carry spatial information. The model may therefore be reading physics, geometry, or a confounded acquisition signature.

No performance drop weakens the claim that the destroyed spatial organisation was necessary for the model’s result. It may indicate that the model used preserved intensity or mask properties, or that the control did not disrupt the relevant representation. It is not proof that texture was absent from the original data.

A label-permutation null and a voxel shuffle can therefore produce different kinds of reassurance within this chapter’s framework. The label null asks whether the pipeline can produce performance without a label relationship; the permutation distinction is discussed by Buvat and Orlhac. The shuffle asks whether performance survives destruction of a claimed representation. Neither one establishes causality. An ablation can show that a component mattered to the fitted model, but it cannot determine whether that component represented biology or a confounder.

The same logic applies to a model that passes every control. A successful control study supports only the specific interpretation allowed by its preserved and destroyed properties, cohort, endpoint, and split design. It does not replace external validation, calibration, or a clinical-utility study. TRIPOD+AI separates these prediction-reporting questions, while CLEAR makes the study design and claims inspectable; neither guideline supplies the missing empirical evidence.

From interpretation to statistical validation

The framework’s full reasoning chain is therefore deliberately staged. First compare the proposed signature with clinical, volume, shape, first-order, texture, full, and full-plus-volume baselines. Then use a label null to check the pipeline, representation controls to challenge the claimed information source, and ablations to understand component contribution. Keep the split discipline identical throughout, and preserve bounded disagreement when published analyses use different definitions or endpoints.

A result that survives this scrutiny is more interpretable, but it is not yet a clinical claim. Chapter 7 — Statistical Learning and Validation takes up the remaining question: how patient-level splits, feature selection, uncertainty, calibration, external evaluation, and clinical utility determine what a predictive result can support.

Sources

• Aerts HJWL, et al. Nat Commun. 2014;5:4006. doi:10.1038/ncomms5006.
• Welch ML, et al. Vulnerabilities of radiomic signature development: the need for safeguards. Radiother Oncol. 2019;130:2-9. doi:10.1016/j.radonc.2018.10.027, PMID 30416044.
• Hatt M, et al. Radiomics: Data Are Also Images. J Nucl Med. 2019;60(Suppl 2):38S-44S. doi:10.2967/jnumed.118.220582.
• Hatt M, Majdoub M, Vallières M, et al. J Nucl Med. 2015;56(1):38-44. doi:10.2967/jnumed.114.144055, PMID 25500829.
• Ger RB, et al. PLOS ONE. 2019;14(9):e0222509. doi:10.1371/journal.pone.0222509.
• Fave X, et al. Transl Cancer Res. 2016;5(4):349-363. doi:10.21037/tcr.2016.07.11.
• Buvat I, Orlhac F. J Nucl Med. 2019;60(11):1543-1544. doi:10.2967/jnumed.119.235325.
• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.