Chapter 8 — Medical Imaging AI: Tasks, Representations, and Evaluation

Images are acquired, located measurements

A medical-imaging model does not receive a picture in the same sense that a consumer-vision model receives a photograph. It receives an operational representation of image evidence: measurements made under an acquisition protocol, reconstructed onto a spatial grid, and accompanied by a contract about what the samples mean and where they are located. The tensor is a convenient implementation of that representation; it is not the whole input or direct ground truth about biology. If the contract is lost, a model may still accept the array and may produce plausible-looking output while answering a different question from the one the study intended. Whether that happens depends on the contract defect and the task; it is a possible failure path, not a universal consequence of every contract error.

This distinction changes the role of an AI system in the measurement path. A segmentation produces a region whose boundary may define a later volume or radiomic calculation. A detector produces locations that may be matched to lesions or readers. A classifier maps an image or a derived representation to a label or risk, while a registration method estimates a relationship between coordinate systems. Reconstruction and synthesis transform image evidence rather than simply assigning a label. These task names describe intended outputs, not guarantees that the outputs are geometrically valid, clinically useful, or faithful to information that was actually acquired. The distinctions are not mutually exclusive: a system may combine task families or emit multiple outputs, for example instance segmentation alongside a registration and a resampled image.

The evaluation must therefore follow the task and the claim. An overlap score cannot by itself establish that a mask gives an acceptable volume. A classifier score cannot by itself establish calibration, useful decisions, or transport to a new acquisition setting. A realistic synthetic image cannot by itself establish preservation of anatomy, pathology, or quantitative measurements. The question is always what the output will be used to do, which sources of variation the model was allowed to see, and what evidence supports that use.

The chapter develops that reasoning as one path. It first treats geometry, modality, and augmentation as part of the input contract. It then follows a segmentation mask into downstream measurement and asks why overlap, surface, reference-standard, and task-level checks answer different questions. The next section uses self-configuration as an engineering lesson rather than as a permanent ranking of tools. Reconstruction is then placed inside the measurement definition, followed by synthetic imaging and the distinction between estimating degraded evidence and inferring a target that was not acquired. The final sections integrate model evaluation with subgroup diagnosis, then compare handcrafted and learned representations by the incremental information they add.

Current foundation-model and tool rosters change faster than the stable measurement questions. They belong in the maintained imaging-AI reference and companion Web, or in a dated illustration whose age and scope are visible. The durable question is not which model name is currently fashionable. It is what training data, adaptation, representation, evaluation setting, and provenance a model depends on, and whether those conditions match the claim.

The preceding validation material established split units, fitted-state boundaries, uncertainty, and transportability. This chapter applies those ideas to inputs that are spatial, modality-specific, and often used to create further measurements. The next section makes that input contract concrete by asking what an imaging tensor would lose if it were treated as an ordinary picture.