Repeatability and reproducibility

The previous section showed that binWidth is part of the definition of a texture measurement. Fixing that contract is necessary, but it does not answer whether the resulting feature behaves acceptably when the conditions that should leave the target unchanged are repeated or deliberately varied. A feature can be precisely defined and still be sensitive to repositioning, reconstruction, segmentation, or acquisition conditions.

That is the measurement-performance question. It must be kept separate from the later questions of whether a feature is biologically meaningful, predicts an outcome, or improves a clinical decision. Agreement analysis describes how a defined measurement behaves under a declared comparison. It does not promote the measurement to a biomarker.

Three questions, three comparison settings

The word reproducibility is often used for several different experiments. They should be separated because each varies a different part of the measurement system.

Question	Comparison setting	Main question
Repeatability	same patient, same scanner, same session	Does the feature remain similar under a near-identical repeated measurement?
Reconstruction agreement or robustness	same acquisition, different reconstruction	Does the feature agree when the reconstruction choice changes?
Reproducibility	different scanners, sites, or protocols	Does the feature agree across broader acquisition conditions?

Repeatability is intended to probe variation under conditions where the target is expected to have changed little. The observed difference can nevertheless contain repositioning, physiology, acquisition noise, segmentation variation, geometry and preprocessing effects, and extraction variation. Reconstruction agreement asks a narrower question by varying a declared reconstruction axis while retaining the acquisition as the comparison basis. Reproducibility asks what happens when several acquisition conditions differ, and therefore requires a careful description of which scanners, sites, protocols, or other factors were varied.

These comparisons may use related data, but they are not interchangeable analyses. A same-patient repeat does not become a reconstruction experiment merely because both images have different file labels. Conversely, comparing two reconstructions from one acquisition does not establish reproducibility across sites. The intended comparison and the source of variation must be declared before a coefficient is interpreted. The statistical considerations for quantitative imaging repeatability and reproducibility are discussed by Ye, Lim, and Huang.

Agreement describes the tested measurement chain

Intraclass correlation coefficients and concordance correlation coefficients can summarize agreement or reliability for paired measurements, but the coefficient belongs to the experiment that produced it. It is not an intrinsic label attached to a feature name.

A test-retest coefficient is calculated from values produced by the complete tested chain: image acquisition, reconstruction, patient positioning, segmentation, geometry, preprocessing, discretisation, and feature implementation. If the coefficient is low, the defensible conclusion is that the feature did not agree under that chain and those conditions. The coefficient alone does not identify whether the main contribution came from the reconstruction, the mask, the image noise, the preprocessing, or another link.

The distinction also applies to the scale of the coefficient. CCC evaluates agreement with the line of identity, while ICC compares repeated-measure variation with between-subject variation under a particular ICC formulation. Both are influenced by the cohort and the estimator convention. The same absolute scan-to-scan difference can have a different relative coefficient in a narrow cohort and in a cohort with a wider range of feature values. A coefficient therefore does not replace paired differences, their distribution, or an absolute error analysis.

This is why a low agreement value is not, by itself, a diagnosis of segmentation failure. It is evidence about the tested measurement process. To identify a plausible source, the experiment needs additional contrasts whose assumptions are explicit.

Diagnostic contrasts and their assumptions

One contrast uses the same image with independent segmentations. If the image and extraction procedure are held fixed while the masks differ, the resulting difference primarily probes segmentation variation. It does not isolate segmentation perfectly if the segmentation process also changes geometry, label representation, or preprocessing.

Another contrast uses registered repeat images with one transferred mask. This reduces independent-segmentation variation, but it retains the effects of the repeat image, registration, and mask transfer. Registration and transfer introduce their own assumptions and errors. A higher agreement value under that contrast does not prove that the segmentation is accurate, just as a lower value does not identify one specific failure.

These are diagnostic designs rather than replacements for the primary repeatability or reproducibility estimand. They help narrow the set of plausible explanations. The result remains conditional on the comparison that was actually performed.

Perturbation is a measurement experiment

A paired test-retest dataset is valuable, but the same reasoning can be applied with controlled perturbations. Change one declared part of the measurement process, such as a small boundary displacement, an interpolation choice, a resampling grid, or a discretisation parameter, while keeping the intended object and the remaining contract fixed. Then measure the feature drift.

A synthetic boundary perturbation makes the logic visible:

Only the ROI boundary changes in this synthetic example, so the changed value illustrates measurement sensitivity rather than object change.

The changed number in this example is not evidence that the synthetic object changed. It is evidence that the feature responds to the altered region. In a real analysis, the same interpretation applies to a controlled reconstruction or discretisation change: the result describes sensitivity to that part of the contract, not a biological response.

Perturbation analysis and test-retest analysis answer related but different questions. Test-retest asks what happens under a specified repeated acquisition. Perturbation analysis asks what happens under a deliberately chosen change in the image, region, or processing. Neither method identifies a cause without a design that varies the relevant link while controlling the others.

Stability is not biomarker validity

Stability is a property of a measurement under specified conditions. It does not establish that the measurement represents the intended biology. A feature can be stable because it encodes a persistent acquisition or segmentation property rather than the biological process of interest. Conversely, a feature can predict within one fixed acquisition environment while being sensitive to a change that would occur when the environment changes.

The questions therefore remain separate. Measurement performance asks whether the value behaves consistently under the declared comparison. Biological validity asks whether it represents the intended target. Prediction asks whether a model using it performs for a stated outcome and population. Clinical utility asks whether using that result changes a decision or outcome in a useful way. Evidence at one level does not automatically answer the next.

This separation also prevents an unstable feature from being dismissed by an absolute slogan. Whether a degree of instability is acceptable depends on the intended use, the feature’s error scale, the relevant decision threshold, and the conditions under which the measurement will be used. The appropriate response may be to exclude the feature, model its uncertainty, restrict the claim, or report the limitation. That decision follows from the intended estimand and error model, not from the family name alone.

Thresholds are conventions, not laws

Studies often use an ICC or CCC cutoff to summarize which features meet a declared stability convention. Such a cutoff can be useful as a descriptive benchmark when it is selected in advance and interpreted within the study’s purpose. It is not a law of measurement, a universal quality boundary, or evidence that every feature above it is valid.

A coefficient near a cutoff should not be treated as fundamentally different from one just below it. Report the feature-level estimates, uncertainty, and distribution of results rather than reducing the experiment to a kept-versus- dropped count. A threshold may also hide the fact that different features within one family respond differently to the same perturbation.

Describing stability and using stability to choose inputs for a predictive model are different operations. The first can be a measurement-performance analysis with its own prespecified comparison. The second is feature selection. If the selection determines which columns enter a model, it belongs to a prespecified model-development and validation design. Reporting guidance such as the CLEAR radiomics reporting checklist and TRIPOD+AI treats feature selection and the separation between model development and evaluation as reportable parts of the design; it does not empirically validate the resulting model. The relevant fitted or selected steps must be restricted to the data available for that part of the design and then applied to held-out data without refitting.

The empirical support for stability-based filtering is therefore context-dependent rather than a guarantee of predictive improvement. A reliability-based selection study may report a gain in one setting, while different cohorts, feature definitions, endpoints, or validation designs may produce another result. The gain must be demonstrated for the intended prediction task; it cannot be assumed from a stability coefficient. The reliability study by Müller-Franzes et al. is one public example of why the selection question should be studied rather than treated as settled.

The practical rule is simple: characterize the defined measurement first, keep the comparison and its assumptions visible, and treat any later stability-based selection as a modelling decision that requires its own validation evidence. The feature is not made trustworthy merely by passing a threshold, and a feature that falls below a convention is not evidence that the experiment was useless.

The next section turns these distinctions into a bounded case study with paired imaging data: RIDER as a robustness experiment separates same-patient repeatability from agreement across reconstructions and uses controlled perturbations to connect a prediction about measurement behavior with an observed result.

Sources

• Ye S, Lim JY, Huang W. Statistical considerations for repeatability and reproducibility of quantitative imaging biomarkers. BJR Open. 2022;4(1):20210083. doi:10.1259/bjro.20210083, PMID 36452056.
• Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45(1):255-268. doi:10.2307/2532051, PMID 2720055.
• Pfaehler E, et al. Phys Imaging Radiat Oncol. 2021;20:69-75. doi:10.1016/j.phro.2021.10.007, PMID 34816024.
• Haarburger C, et al. Scientific Reports. 2020;10:12688. doi:10.1038/s41598-020-69534-6.
• Müller-Franzes et al. Diagnostics. 2022;12(2):247. PMID 35204338.
• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.