Case study: measure before intervening

The previous section ended at the boundary between a characterised measurement and a later intervention. Once acquisition differences are visible, it is tempting to harmonise the feature table, add filtered images, apply learned preprocessing, select the most stable columns, and fit a model in one pass. That sequence may produce a result, but it makes the result difficult to interpret. If several interventions change the table at once, an apparent improvement or deterioration cannot be attributed to one of them.

This section uses the logic of a dated RIDER-style protocol freeze as a bounded, time-limited illustrative case. Its first question is not whether a feature predicts an outcome. It is whether a declared measurement behaves under repeatability and reconstruction comparisons. The design is self-contained: the reader needs the distinction between the baseline and later interventions, not access to project files, internal status, or implementation history.

One experiment, one question

A useful freeze states three things before results are inspected: what is held fixed, what varies, and what question the result answers. In the baseline case, the image and region definitions, geometry, preprocessing, discretisation, feature implementation, and analysis rules form a fixed measurement contract. The declared repeatability and reconstruction comparisons are the variations. The result answers how the unadjusted measurement behaves under those conditions.

The same structure applies to a follow-up intervention. The baseline contract and comparison design remain the reference, one intervention is added or changed, and the new result asks what that intervention did. A harmonisation step, filtered image, learned transformation, or modelling choice is not silently folded into the baseline. It is given a new definition and a new comparison.

This separation is ordinary experimental reasoning. If harmonisation, a wavelet expansion, learned denoising, and feature selection are introduced together, a subsequent score cannot show which choice caused the change. A poor result is equally ambiguous: it might reflect a harmful transformation, a mismatched feature definition, or an interaction among the interventions. The experiment has produced one number but not an attributable explanation.

The baseline can also produce an undesirable result without being a failed study. If a feature is poorly repeatable or sensitive to reconstruction, that is valid evidence about the measurement under the tested conditions. Hiding that result by showing only the surviving features answers a different question: how a selected subset behaved. The raw distribution of measurement performance is itself part of what the baseline was designed to establish.

The unadjusted baseline

The current RIDER Lung CT Version 3 dataset supplies the multiple-reconstruction setting. The original design paper by Zhao et al. documents the same-day repeat and repositioning context; its repeatability analysis used one thin-section reconstruction for tumour-size measurements, not the later six-reconstruction package. The RIDER-style design combines two useful comparisons. A same-patient, same-scanner, same-session repeat probes repeatability while allowing repositioning, physiology, acquisition noise, segmentation, and extraction variation to remain visible. Comparing reconstructions derived from the same acquisition probes reconstruction agreement while deliberately varying the reconstruction axis. These comparisons are analysed separately because they represent different perturbations.

For each comparison, the feature values, paired differences, agreement estimate, and uncertainty belong to the declared measurement chain. The baseline does not claim that a feature is a biological truth merely because its values are repeatable. It records what the measurement does when the chosen conditions are repeated or changed. A low agreement result can be useful because it tells a later study that the feature should not be treated as stable under that condition. It does not identify the responsible link without an additional diagnostic contrast.

This is why the baseline must remain unadjusted for the question it was designed to answer. If the reconstruction axis is the source of variation under study, correcting feature distributions before measuring it changes the estimand. The adjusted table may be useful for another purpose, but it cannot stand in for the original measurement-performance result.

The same logic applies to a stability threshold. A cutoff can summarise a prespecified descriptive convention, but it does not convert a feature into a valid biomarker or make a poor baseline result disappear. Whether a feature is acceptable depends on the intended use, its error behavior, the decision context, and the conditions in which it will be measured.

Why harmonisation is a separate intervention

ComBat adjusts feature distributions for a declared batch effect. That can be appropriate when the batch structure and its assumptions match the intended analysis. It is not a universal correction for every difference between sites.

The main difficulty is that batch can be entangled with biology or outcome. A site may have a different scanner and also a different patient mix or care pathway. Treating the entire site difference as technical nuisance can remove signal that the estimand intends to retain, or can leave residual bias after the adjustment. The batch label identifies a grouping; it does not prove the cause of the difference.

Feature-level ComBat operates on already extracted feature values rather than on the acquisition process that produced them. If a reconstruction has destroyed spatial content, transforming those values does not reconstruct the image or restore spatial content absent from the extracted inputs. It may make values look more comparable while concealing the raw reconstruction sensitivity that the baseline was meant to describe. The adjusted value is therefore the output of a new measurement intervention.

The original empirical-Bayes method is described by Johnson, Li, and Rabinovic, and imaging-specific assumptions are discussed by Orlhac et al.. Those sources do not turn ComBat into a universal prescription. They support treating its role as a design decision.

In this case, harmonisation follows the unadjusted characterisation because the baseline asks what the acquisition and reconstruction differences do to the measurement. That ordering is not a universal ban on ComBat. If the primary estimand is a harmonised feature for a particular multi-site use, harmonisation may be part of that measurement’s intended definition. The study must then state that role and evaluate the harmonised measurement for that purpose.

Filters and learned preprocessing require their own freeze

A filtered image is another intervention before feature extraction. Wavelet bands and Laplacian-of-Gaussian images change the image domain on which first-order or texture features are calculated. A learned denoising or super-resolution transformation changes the input in a different way. If such a transformation is added to the baseline, the resulting agreement describes the transformed pipeline, not the original measurement.

The right response is not to prohibit filters or learned preprocessing. It is to make their question explicit. A follow-up study can retain the baseline contract, add one declared transformation, remeasure the same paired comparisons, and compare the new behavior with the baseline. If agreement improves, the result supports a statement about the transformed pipeline under the tested conditions. It does not show that the original feature became intrinsically robust, that the transformation reconstructed the image or restored spatial content absent from the extracted inputs, or that the transformation will improve prediction.

The same principle protects interpretation when several image domains are available. Adding filtered features expands the candidate table and may add related views of the same image. Their filter definitions, physical scale, boundary handling, discretisation, and aggregation are part of the new contract. The IBSI standardisation work is relevant to the computation of such features, but a standardised calculation still needs a separately stated measurement question.

Selection and modelling come later

The baseline should describe the distribution of measurement behavior before columns are hidden by a selection rule. A later model may use robustness, harmonisation, or another property to choose its inputs, but that choice is then part of model development. Reporting guidance such as the CLEAR radiomics reporting checklist and TRIPOD+AI treats the development and evaluation boundary as a reportable part of the design; these guidelines do not empirically validate a model.

This distinction does not depend only on whether the selection uses the outcome. An outcome-independent stability analysis can be a legitimate descriptive characterization. Once its result determines which features enter a predictive model, however, the selection becomes part of the model-input definition. If the selection or another transformation is fitted from data, it must respect the relevant validation boundary. For cross-validation, fit the permitted data-dependent steps on the training portion and apply the resulting state to the held-out portion. ComBat, scaling, feature selection, and stability-based selection follow the same information rule when they construct model inputs.

There is no universal sequence that places every one of these operations before or after every other operation. Exact placement depends on the estimand, the role of the intervention, whether it acts on images or features, and what information is available at the intended deployment point. What is universal within a given predictive design is the need to preserve the training and evaluation boundary and to declare the sequence before interpreting the result.

At the model handoff, the feature table may still have p >> n: the candidate set can be much larger than the number of patients. Shared inputs or related transformations can induce substantial dependence or correlation among candidates; a stability filter does not make the remaining columns independent. The CLEAR radiomics reporting checklist makes feature reduction and model selection reportable parts of the pipeline, but it does not establish empirical validity. Regularisation, honest selection, and patient-level validation remain necessary. If the question is incremental value, compare the imaging model with an appropriate clinical tabular baseline rather than with chance alone.

An internal predictive score is still only evidence for the tested prediction design. It does not by itself establish construct validity, transportability, calibration, or clinical utility. The baseline’s poor repeatability cannot be rescued retrospectively by choosing a more favorable subset, and an improved post-intervention score cannot identify which part of the intervention supplied the improvement unless that intervention was isolated and validated. Prediction-reporting guidance such as TRIPOD+AI separates discrimination, calibration, external evaluation, and clinical utility as distinct reporting questions; CLEAR supports making the comparison and processing path inspectable. These are reporting roles, not evidence that a model is calibrated, transports, or improves utility.

The hierarchy as a sequence of estimands

The case can therefore be read as a sequence of linked but distinct questions. First freeze and measure the unadjusted baseline. Then describe and investigate its failure modes. If a harmonisation, filter, learned preprocessing step, or model is justified, freeze that new intervention as a separate design. Remeasure the same relevant quantities under the new contract. Only after those measurement questions are answered should an outcome-informed predictive analysis select inputs and estimate performance.

Each transition changes what the result is allowed to mean. The baseline supports a claim about measurement behavior. The follow-up supports a claim about the intervention’s effect on that behavior. The model supports a claim about prediction under its validation design. None of those claims automatically establishes the next one.

Repeatability and measurement performance therefore do not establish construct validity. The next question is whether the signal is really radiomic: Is the signal really radiomic? applies falsification, baselines, and negative controls to the interpretation of a successful radiomics model.

Sources

• Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007;8(1):118-127. PMID 16632515, doi:10.1093/biostatistics/kxj037.
• Orlhac F, et al. A Guide to ComBat Harmonization of Imaging Biomarkers in Multicenter Studies. J Nucl Med. 2022;63(2):172-179. PMID 34531263, doi:10.2967/jnumed.121.262464.
• Zwanenburg A, Vallières M, Abdalah MA, et al. The Image Biomarker Standardisation Initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology. 2020;295(2):328-338. doi:10.1148/radiol.2020191145, PMID 32154773.
• Whybra M, et al. IBSI 2. Radiology. 2024. doi:10.1148/radiol.231319.
• Quantitative Imaging Biomarkers Alliance (QIBA), RSNA - official QIBA information.
• Zhao B, Schwartz LH, Kris MG, Riely GJ. Coffee-break lung CT collection with scan images reconstructed at multiple imaging parameters (Version 3) [Dataset]. The Cancer Imaging Archive. doi:10.7937/K9/TCIA.2015.U1X8A5NR, collection page.
• Zhao B, et al. Radiology. 2009;252(1):263-272. doi:10.1148/radiol.2522081593, PMID 19561260, PMCID PMC2797680.
• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.