Synthetic imaging: what is the model inventing?

Tonight · ~25 min · read · energy: medium · setup: none

Synthetic imaging — generating an unacquired image from an acquired one — is a real, current domain. Its central discipline is not “which architecture” but a sharper question: what information is actually present in the source, and what is the model being asked to invent? This lesson is that question, the problem families it separates, and the evaluation it forces.

The anchor: same architectures, fundamentally different problems

Conditional image generation shares image-to-image methods across many problems. But those problems are not physically or clinically equivalent. The defining question:

What information is actually present in the source image, and what is the model being asked to invent?

Answer it and the problems separate cleanly:

Problem	Source → target	What the model must invent
MRI sequence synthesis	one MR contrast → another	a contrast not acquired
MRI → synthetic CT	MR anatomy → CT attenuation	electron density MR cannot show
low-dose → standard-dose CT	noisy CT → clean CT	noise removal (info present, degraded)
reconstruction / protocol harmonisation	one kernel/protocol → another	a transform between protocols
non-contrast → contrast-enhanced CT	pre- → post-contrast	contrast uptake (genuinely absent)

The key contrast: low-dose→standard mostly denoises present information; non- contrast→contrast asks the model to hallucinate enhancement that was never measured. Same architectures, fundamentally different problems — and the hallucination risk scales with how much the target is genuinely absent from the source.

Why realistic appearance is insufficient

A synthetic image can look photorealistic and still:

• violate anatomical consistency (plausible-but-wrong structure);
• drop or invent pathology (the clinically fatal failure);
• change radiomic/quantitative values invisibly;
• be useless or misleading for the downstream task.

This is why pixel-realism metrics (SSIM, PSNR, visual realism) are necessary but not sufficient. The defining risk of synthesis is hallucination / invented information: the model confidently produces a target that was never measured, and “looks right” is not “is right”.

Evaluation must go beyond pixel realism

A synthetic-imaging evaluation that stops at SSIM/PSNR/visual realism is incomplete. Demand:

• anatomical consistency — plausible-but-wrong structure flagged;
• pathology preservation — does disease survive, neither lost nor fabricated?;
• quantitative / radiomic stability — do measurements agree on synthetic vs real?;
• task utility — does a downstream model/read perform as well on synthetic?;
• uncertainty — does the model know when it is guessing?;
• external testing — not just in-distribution;
• hallucination / invented-information risk — the defining risk of synthesis.

The “RaD” result (arXiv:2412.01496, VERIFIED) makes the task-utility point concretely: upstream perceptual metrics like FID do not track downstream segmentation performance — a synthetic image can look better and serve the task worse.

Verified reading (problem classes)

A verified, representative synthetic-imaging set lives in the literature map (“Synthetic imaging”):

• MR→synthetic CT (review) — Spadea et al., Med Phys 2021, PMID 34407209 (MR→CT electron density for dose planning).
• Reconstruction / denoising — Sim et al. shoulder MRI DL-recon, Cureus 2025;17(10):e94561, PMID 41246632; Park et al. lumbar spine CT DL denoising, Eur Radiol 2025;35(12):7867-7876, PMID 40483292; Herrmann et al. elbow MRI DL-recon, Diagnostics 2023;13(17):2747, PMID 37685285.
• Task-based evaluation (perceptual realism insufficient) — “RaD,” arXiv:2412.01496.

PLAUSIBLE INFERENCE that synthesis will matter clinically; the method family is represented by the sources above. (The EuSoMII T2→T1/STIR webinar example could not be reliably identified and is preserved as explicitly unresolved in the literature map.)

Stop and think — then reveal

Two synthetic-imaging papers report identical SSIM and visual-realism scores. One does low-dose→standard-dose CT; the other does non-contrast→contrast-enhanced CT. Which should you trust less on the realism score alone, and why?

The non-contrast→contrast paper. Low-dose→standard mostly denoises information that is present (degraded but measured), so a realistic-looking output has a decent chance of being physically faithful. Non-contrast→contrast asks the model to invent enhancement that was never measured — perfusion and vascular permeability the pre-contrast scan does not contain (Ch. 8). There, a photorealistic output can be a confident hallucination: it looks like a post-contrast CT but the enhancement is fabricated. Realism scores cannot tell the two apart; only task-based and pathology-preservation evaluation can. This is the “what is the model inventing?” distinction in action.

What to retain

1. Synthetic-imaging problems share methods but differ by what the model must invent: denoising (info present) vs synthesis of absent contrast (hallucination).
2. Realistic appearance is insufficient — a synthetic image can look right and violate anatomy, drop/invent pathology, or change quantitative values invisibly.
3. Evaluate beyond pixel realism: anatomical consistency, pathology preservation, quantitative stability, task utility, uncertainty, external testing, hallucination risk.
4. The defining risk is invented information; “looks right” is not “is right”.

Next: the substrate these methods are scaling toward — foundation and multimodal models.