High-dimensional, small-sample inference

Coincidence is cheap when columns outnumber patients

Radiomics can produce a table with many candidate columns and comparatively few patients. Write that relationship as p >> n: the number of candidate features is large relative to the number of independent units relevant to the estimand. The notation is not a diagnosis of a model, but it warns that the data may not constrain the many possible explanations for an observed association or score.

With hundreds of candidate features, some may appear related to an outcome in a particular sample by chance. Correlated features make the interpretation harder: several columns can encode overlapping views of the same measurement chain and can induce dependence or correlation, so the nominal column count is not the number of independent biological hypotheses. A flexible model can also fit idiosyncrasies of the development cohort that do not persist in another sample. These are risks to be measured and controlled, not universal verdicts about every high-dimensional model.

Three consequences to inspect

The first consequence is selection instability. If a small perturbation of the cohort or split changes which features survive, the selected feature list is not a stable biological story. Instability can be a useful result: it indicates that the data do not identify one narrow representation under the declared design. Hiding it behind one preferred feature list makes the inferential claim look more precise than the study supports.

The second consequence is multiplicity. Screening many features for an association creates many opportunities for small p-values or large apparent effects. The Benjamini-Hochberg procedure is a step-up procedure for controlling the false discovery rate under its stated conditions for a defined family of hypotheses. That is an inferential multiplicity role, not predictive feature selection. Predictive performance and inferential discovery are not the same target.

The third consequence is optimism from selection. If the same observations choose the features and evaluate the resulting model, the score is conditional on a search that has already used those observations. The leakage and nested-selection sections give the information-boundary repair: selection and any fitted preprocessing must occur within the permitted training data, with an outer evaluation for the full procedure when the search is part of the claim.

What the regime requires

Regularisation is a predictive design choice that restricts or shrinks the effective model complexity. Ridge, lasso, and elastic-net penalties make different trade-offs between retaining correlated variables and producing a sparser representation. A rule such as the one-standard-error choice can be used when it is declared as part of the selection procedure. None of these choices is a universal guarantee of reproducibility, and the chosen penalty must be selected inside the relevant validation boundary.

The elastic-net analysis by Zou and Hastie is a direct anchor for the correlated-predictor and p >> n discussion. Its grouping behaviour and penalty trade-offs do not guarantee feature stability or biological identification.

The second discipline is to make the search visible. Record the candidate set, the screening rule, the tuning budget, the split unit, and the fitted-state boundary. If the feature set is prespecified, say what was prespecified and why. If it was discovered from the data, treat discovery and performance estimation as connected parts of the procedure. If a scientific feature claim is the target, define the hypothesis family and the inferential procedure. Benjamini-Hochberg can control false discovery rate under its stated conditions for the hypotheses being tested; it does not turn a model-driven feature search into selective inference. Without a separately justified selective-inference method, describe the search as predictive selection and do not attach post-selection inferential claims. A predictive regulariser is not an inferential correction.

The third discipline is to report what did not remain stable. A feature that appears in one fold but not another may still be useful as one input to a predictive procedure, but its presence does not establish a unique mechanism. The distinction between prediction and inference should remain visible in the results and in the language of the claim.

A clinical baseline changes the question

When the proposed claim is specifically that imaging adds information beyond variables already available in the intended workflow, chance is usually a weak comparison. In that case, a prespecified clinical or tabular baseline asks what can be predicted from those variables. The incremental question is whether the image-derived inputs improve the target under the same split, preprocessing, tuning, and evaluation rules.

The baseline may be more competitive, less competitive, or similar in a given study. Its value is not an assumed ranking. It identifies the comparison needed to distinguish “the model predicts” from “the imaging measurement adds value beyond the clinical record.” The same incremental framing returns in the survival section, where a radiomic covariate is compared with a clinical-only survival model.

As a constructed warning example, imagine 80 patients, 200 candidate features, a whole-cohort screen retaining 10 columns, and a reported cross-validated AUROC of 0.85. Before interpreting the number, ask whether the screen read the folds later called test folds, whether the candidate columns were correlated, and how the selected list changes under resampling. The number is not a measurement from this chapter; it is a compact way to expose the questions that the design must answer.

The high-dimensional problem is therefore not solved by choosing a fashionable algorithm. It is solved by matching the inferential or predictive target to a transparent selection procedure, a strong baseline, and an evaluation that has not been used to choose its own winner. Even after that, a performance estimate still has several distinct dimensions, beginning with discrimination.

Sources

• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B. 2005;67:301-320. public full text.
• Benjamini Y, Hochberg Y. Controlling the False Discovery Rate. Journal of the Royal Statistical Society: Series B. 1995;57(1):289-300. public full text.