p >> n: why radiomics breaks ordinary intuition

Tonight · ~20 min · read · energy: medium · setup: none

Ordinary ML intuition is calibrated to “more rows than columns”. Radiomics hands you the opposite: 200 correlated features on 80 patients. In that regime, things that feel safe — a flexible model, a feature selected for looking good — become sources of unreproducible overfitting. This lesson is why p >> n breaks the intuition and what that forces you to do.

The anchor: when columns outnumber rows, coincidence is easy

With 200 features and 80 patients, there is a lot of room for a feature to accidentally correlate with the outcome. Run a univariate scan of all 200 and several will look “significant” by chance alone (the multiple-testing problem). Fit a flexible model that can use many features and it will find a combination that fits the training patients almost perfectly — including their noise. Neither discovery will survive a new cohort. That is the p >> n trap: the model does not lack capacity, it lacks the data to constrain it.

Three consequences that follow

• Overfitting is the default, not the exception. A model with more features than patients can almost always drive training error to zero by fitting idiosyncrasies. Cross-validation will catch some of this, but only if the leakage doors of lesson 2 are closed and selection is inside CV (lesson 3).
• Feature selection is unstable. With correlated, redundant features (the radiomics battery — Ch. 3), which features “survive” selection flips between folds and between cohorts. A model whose selected features change every run is a model whose biology story changes every run. This instability is itself a finding: report it, do not hide it.
• Multiple testing inflates discoveries. Scanning hundreds of features against an outcome produces false positives at scale; without correction (or pre-registration) the “significant” features are mostly noise.

What p >> n forces

Three disciplines, all non-optional in this regime:

1. Regularisation. Ridge, lasso, elastic-net shrink coefficients so the model cannot use all p features freely. Lasso drives some to zero (selection); ridge keeps them but small. The ISLP one-standard-error rule picks the most parsimonious model within one SE of the best — a deliberate bias toward fewer features, because parsimony is reproducibility.
2. Honest selection inside CV. Selection on the whole cohort is the most common optimism source (lesson 2); nested CV (lesson 3) is the structural fix.
3. Pre-specification or correction. Pre-register the feature set if you can; if you cannot, correct for multiple testing and report how many features were screened.

The stronger-baseline lesson

p >> n also explains a structural fact: a strong, regularised tabular baseline (age, stage, histology — a handful of well-chosen clinical variables) often beats a naive high-dimensional image model, because it has fewer ways to overfit and asks a sharper question. The honest radiomics question is therefore not “does the image model beat chance?” but “does imaging add value beyond the clinical baseline?” — the same framing as the survival question in lesson 7. TRACE-CT makes tabular baselines (CatBoost-class models, CTGAN/TabDDPM in dependency smoke tests) first-class citizens precisely so the image model is benchmarked against something honest, not against nothing.

Stop and think — then reveal

You have 80 patients and 200 radiomic features. You select the top 10 features by univariate p-value on the whole cohort, fit a logistic regression, and get cross-validated AUROC 0.85. What two things should make you distrust this before you even look at the number?

1. p >> n with whole-cohort selection — the top-10 by p-value were chosen using all 80 patients, so the test folds influenced the selection (leakage, lesson 2). With 200 screened features, several of the “top 10” are likely chance correlations (multiple testing).
2. Selection instability — re-run on a resampled cohort and a different 10 features will likely survive, meaning the model’s “signal” is partly which features happened to correlate this time. The 0.85 reflects optimism from both leakage and overfitting; nested CV with regularisation will usually lower it substantially.

What to retain

1. p >> n (more features than patients) makes coincidence easy: overfitting is the default, selection is unstable, multiple testing inflates discoveries.
2. It forces regularisation (lasso/ridge/elastic-net, one-SE rule), honest selection inside nested CV, and multiple-testing correction or pre-registration.
3. Report selection instability — features that flip between folds are a finding, not an embarrassment.
4. The honest question is whether imaging adds value beyond a strong tabular baseline, not whether the image model beats chance.

Next: even with all of that fixed, a high AUROC can still be a poor clinical model.