Handcrafted and Learned Representations

Compare information, not teams

The same imaging cohort can be represented by clinical variables, volume and shape, first-order intensity summaries, handcrafted texture features, learned network activations, or embeddings from a model pretrained on an external corpus. These are not rival teams with an intrinsic winner. They are different transformations of the available information, each with its own assumptions, measurement dependencies, and opportunities for shortcut learning. This is a working comparison rather than an exhaustive taxonomy; studies may use other representations or combinations.

The useful question is incremental:

What does the more complex representation add beyond the simpler ones, and does that increment survive an honest evaluation?

This question protects against a common comparison error. A representation can produce a higher internal AUC because it encodes volume, site, segmentation style, or another variable already related to the label. That may be a real predictive association, but it does not establish that the representation has captured the intended biology or that its increment will travel to another cohort.

What each representation makes explicit

Handcrafted radiomics starts with mathematical summaries selected by people. Its definition can be inspected: a reader can state which intensity distribution, spatial relationship, or shape operation was calculated under a declared image, mask, geometry, and discretisation contract. That makes the feature definition more visible, not necessarily the causal meaning of a feature. The vocabulary chosen by the analyst also limits what the representation can express, and the calculation remains conditional on the measurement layer. Inspectability makes a feature definition more visible; it does not guarantee interpretability, causal meaning, or biological validity.

Learned representations optimise an encoding from data for a pretraining or prediction objective. They can express patterns not anticipated by a handcrafted feature family, but they also inherit the data scale, labels, preprocessing, split design, and shortcuts available during learning. An embedding is not automatically more robust or more biologically meaningful because its features were not named in advance. Its interpretation is usually indirect and depends on the evaluation design.

Foundation-model embeddings add another dependency: information learned from an external corpus and its training procedure. That can be useful when local labels are scarce, but it creates provenance questions about corpus composition, overlap, licensing, task relevance, adaptation, and hidden shortcuts. Specific model names, corpus rosters, capability claims, and licence details change quickly, so they belong in dated companion-Web content and must be verified against the relevant official terms or source. The stable question is what was pretrained, what was frozen or adapted, and how the embedding was tested on the task at hand.

A baseline ladder isolates the increment

A practical comparison uses the same patient splits and the same fitted-state rules for every representation. A baseline ladder can contain:

1. clinical variables only;
2. volume and shape only;
3. handcrafted radiomics;
4. a learned representation, whether from a task-trained network or a pretrained embedding;
5. clinical variables plus handcrafted radiomics;
6. clinical variables plus the learned representation;
7. handcrafted radiomics plus the learned representation; and
8. the combined clinical, handcrafted, and learned representation, only when the validation design justifies the extra complexity.

The ladder is a design and reading instrument, not a promise that every rung must be used in every study. It does make missing comparisons visible. Different splits can produce different apparent results because each fitted procedure sees a different set of observations. Any scaling, selection, tuning, or representation adaptation that depends on data must respect the validation boundary described in the statistical-validation chapter. When the feature table has p >> n, concatenating representations increases the space in which variance and selection can operate. p >> n is a symbolic description of a high-dimensional regime, not a universal numeric threshold. Instability depends on effective sample size, signal, feature correlation, model complexity, and the selection procedure. Regularisation can constrain a fit, but it does not replace nested selection or independent evaluation.

The ladder also exposes label-derivation artifacts. If a label partly encodes a quantity that a representation measures directly, such as size-related staging and volume or a volume-derived outcome and a segmentation mask, the resulting performance can be predictable by construction. That can be a valid result for the operational prediction task, but it is not evidence of an independent biological signal. The baseline must be interpreted in relation to how the label was made. This pattern generates a hypothesis about label construction; a targeted test is needed before it is treated as a causal explanation.

Fusion is a comparison design, not a guarantee

Early fusion concatenates feature tables before fitting one model. It can expose cross-representation interactions, but it also expands the feature space and the number of fitted choices. Late fusion fits representation-specific models and combines their predictions. It can be useful when the representations are partly complementary, but it does not learn every cross-representation interaction. A stacked design feeds representation-specific scores into a small final model; if those scores are learned from data, the stack must use out-of-fold predictions rather than training-set outputs.

Every fusion choice adds another place for selection and leakage. Complementarity must be demonstrated on the declared task and test setting, not assumed from the fact that two feature families were built differently. A combined model that improves an internal result but does not improve an external or otherwise appropriate test has not established a durable increment.

Dated comparisons illustrate the rule

The dated comparisons associated with this material are useful as case studies, not as a permanent ranking. A controlled benchmark can find task-dependent differences between handcrafted and learned extractors. A clinical fusion study can show an internal combination advantage that narrows, disappears, or changes character externally. A trial-cohort comparison can show that an ensemble adds little when its component representations are redundant. A comparison across different regions can change the apparent ranking because the region definition changes the information presented to both families.

Those patterns are hypotheses to inspect in the actual study, not evidence that one family always wins. Current papers, preprints, identifiers, numerical results, and model versions are maintained separately and require source-grounded verification before they carry a claim in the stable manuscript. The portable lesson is to read the external row, the baseline rungs, the region definition, and the split and adaptation procedure before interpreting a representation ranking.

Representation depends on the measurement layer

A representation does not float above the image. Radiomics depends on the image, mask, geometry, preprocessing, and feature implementation. Learned embeddings depend on those inputs plus the training objective and data history. A change in region, reconstruction, resampling, or intensity handling can change the information supplied to both families. A comparison that does not state the ROI and its provenance is therefore under-specified, even if the model names and classifier are clear.

The simpler rungs are also controls for interpretation. Clinical variables and volume or shape can reveal whether a complex representation adds information at all. A representation that wins only when a label-derived quantity is present may be measuring the label construction. A learned representation that appears strong on one site may be using a site or reconstruction cue. These baseline patterns generate hypotheses, not causal findings; targeted tests are needed before attributing the result to a specific shortcut. They do not condemn the method; they determine what the result is allowed to claim.

Handoff: reconstruct the study behind the result

Before accepting a representation claim, write down the cohort, analysis unit, patient split, image and mask contract, label construction, fitted states, baseline rungs, external or temporal test, and any adaptation. Then ask what increment remains after those choices are held in common. This is the same measurement-aware and task-aware discipline developed throughout the chapter.

The next chapter turns from model outputs to the evidence claims made in papers. It begins by reconstructing a study before accepting its story: which units were observed, what crossed the split, how the reference was made, and exactly what a reported result licenses.