Why medical images are not just large tensors

Tonight · ~20 min · read · energy: low · setup: none

If you point a generic vision pipeline at a CT, it will treat each slice as a picture and ignore everything that makes it a measurement. This lesson is the four things that change when the input is a medical image — and why ignoring them breaks the model in ways that look fine until deployment.

The anchor: an image is a located, acquired measurement, not a picture

A natural image is an array of RGB values that mean roughly “what a camera saw”. A CT is an array of HU values — reconstructed, geometry-carrying, acquired under a specific protocol — that mean “x-ray attenuation at a point in the patient” (Ch. 1). A model that ignores that difference treats a 5 mm voxel like a 0.7 mm one, pools contrast and non-contrast scans, and augments a CT with colour jitter that destroys the HU meaning. It may still train — and still fail the moment the acquisition changes.

The four differences

Spatial and anisotropic. Medical images are 2D / 2.5D / 3D, with voxel spacing usually thicker in z. A network that ignores spacing treats a 5 mm voxel like a 0.7 mm one — its receptive field covers a different physical extent on every scan. Either resample isotropic (Ch. 2) or make the architecture spacing-aware.
Geometry is part of the input. Origin/spacing/direction (Ch. 2) are not optional metadata; lose them and augmentation, registration, and mask handling break. A flip that is correct in array space may mirror the patient.
Scarce and expensive. Tens–hundreds of patients, not millions of images. Augmentation matters more, but it must be physically plausible — respect HU meaning (no arbitrary intensity jitter that breaks the attenuation scale), respect anatomy (no rotations that produce impossible poses).
Acquisition-dependent. Reconstruction kernel, dose, contrast phase, site define the input distribution. Domain shift is the rule, not the exception (Ch. 4): a model trained on Siemens B70f lung-kernel CT may not behave the same on GE soft-kernel CT.

Why this produces “works in dev, fails in deployment”

The four differences compound. A model that ignored spacing and was trained on one site’s kernel will silently misjudge physical extent and encounter a different texture distribution at a new site. Internal metrics looked fine because the test set shared the training distribution; the failure appears only under domain shift, which is the default in medicine. This is why “we achieved AUROC 0.95 internally” is not a deployment claim — it is, at best, an internal-testing claim (Ch. 4).

Stop and think — then reveal

A colleague proposes augmenting a CT segmentation training set with random colour jitter, random 90° rotations, and random intensity scaling by ±50%, “because that is what works on ImageNet”. For each augmentation, say whether it is safe for CT and why.

Colour jitter: nonsensical — CT is a single-channel HU scale, not RGB; “colour” has no meaning and any channel manipulation corrupts the attenuation values.
Random ±50% intensity scaling: unsafe — it destroys the HU meaning (a voxel at 0 HU water becomes ±50, breaking the physical scale that thresholds, windows and features rely on). Physically implausible.
Random 90° rotations: unsafe as-stated — a rotation in array space may produce an anatomically impossible orientation or flip the patient (LPS/RAS, Ch. 2). Augmentation should be anatomically plausible (small deformations, axis-aligned flips that respect patient orientation) and preserve HU. The rule: augment in a way the physics permits, not in a way the library offers.

What to retain

A medical image is a located, acquired measurement, not a picture; treating it as a generic tensor breaks spacing, geometry, and meaning.
Four differences: spatial/anisotropic, geometry-as-input, scarce/expensive, acquisition-dependent. Address each by design.
Augmentation must be physically plausible (preserve HU, respect anatomy/orientation).
Domain shift is the rule — internal metrics are not deployment claims.

Next: the downstream task that depends on getting all this right — segmentation as infrastructure.