Skip to content
Imaging Evidence Path

Foundation and multimodal models

Tonight · ~25 min · read · energy: medium · setup: none

Foundation models, vision-language models and generalist imaging models are the medium-term substrate for segmentation infrastructure and assisted interpretation. The goal here is orientation and informed paper reading, not implementation: what is genuinely new, what stays the same, and where the evidence boundary is today.

The anchor: a progression of generality

Hold one progression in mind — each step adds generality (more tasks/modalities) and, usually, more failure surface (grounding, hallucination, provenance):

flowchart LR
    T["task-specific model"] --> P["pretrained model"]
    P --> F["foundation model"]
    F --> V["VLM (vision-language)"]
    V --> G["generalist model"]
    G --> A["tool-using / agentic workflow"]

The questions to ask of any paper on this curve are stable: training data scope, which tasks are zero-shot vs adapted, what external testing, licence/availability, and where it would break under domain shift.

Verified examples (publication status marked)

  • TotalSegmentator (PMID 37795137, peer-reviewed) — operational now; the segmentation-infra layer (lesson 4).
  • Merlin — CT vision-language foundation model. Peer-reviewed: Nature 2026, doi:10.1038/s41586-026-10181-8, PMID 41781626 (VERIFIED; earlier preprint PMID 38978576, 2024). 3D VLM for abdominal CT; tasks incl. zero-shot findings classification, phenotype classification, report generation, 3D segmentation; internal + large external validation. A reference for foundation-model evaluation at scale.
  • a2z-1 (Rajpurkar et al., arXiv:2412.12629, VERIFIED) — generalist abdomen-pelvis CT, external validation across many conditions. Preprint — strong external-validation methodology; revisit on publication.
  • OmniMRI (He et al., arXiv:2508.17524, VERIFIED) — unified vision-language foundation model for generalist MRI (reconstruction, segmentation, detection, diagnosis, report generation). Preprint (2025-08-24), MGH/Harvard.
  • Paschali et al., “Foundation Models in Radiology: What, How, Why, and Why Not,” Radiology 2025;314(2):e240597, PMID 39903075 (VERIFIED) — the orientation/review to read first for the landscape and limits.
  • ReCo (Tzanis & Klontzas, medRxiv 2026.07.14.26358025, VERIFIED preprint; github.com/eltzanis/ReCo) — self-configuring agentic framework using MCP servers, integrating Orthanc + Merlin + TotalSegmentator + DosimeTron. An example of the agentic/MCP orchestration direction — speculative without strong clinical validation yet.

Goal here: orientation and informed paper reading, not immediate implementation. When you read a foundation-model paper, ask: training data scope, which tasks are zero-shot vs adapted, what external testing, licence/availability, and where it would break under domain shift.

What is genuinely new, and what is not

  • New — scale (one model, many tasks), zero-shot transfer to new findings, natural-language grounding (report generation, VLM queries), and the orchestration of multiple tools by an agent.
  • Not new — the domain-specific hazards. Foundation models still face domain shift (Ch. 4), reconstruction dependence (lesson 5), and the imaging-specific leakage and evaluation discipline of all of Chapter 4. A bigger model is not exempt from calibration, external testing, or honest reporting.

The evidence boundary today

Most foundation/generalist imaging models are preprints or have limited external validation; the field is moving faster than peer review. Treat “generalist model that does everything” claims as orientation, not deployment evidence. The strongest current examples (Merlin, peer-reviewed with large external validation) are references for how to evaluate at scale — read them for the evaluation methodology, not just the headline tasks.

Stop and think — then reveal

A preprint announces a “generalist foundation model for medical imaging” with strong zero-shot numbers on internal data and no external test. Which three questions from this chapter’s checklist decide whether the claim is credible?

  1. Training data scope — what (and whose) data was it trained on, and does the “zero-shot” test set leak into that training pool?
  2. External testing — is there any independent site/scanner evaluation? (Here: no, so the claim is internal-only — Ch. 4.)
  3. Domain-shift and failure surface — where would it break (vendor, modality, reconstruction), and are grounding/hallucination/provenance addressed? Without these, “generalist” is an aspiration, not a demonstrated capability. The preprint status means revisiting on peer review.

What to retain

  1. Progression: task-specific → pretrained → foundation → VLM → generalist → agentic. Each step adds generality and failure surface.
  2. Stable questions for any paper on the curve: data scope, zero-shot vs adapted, external testing, licence, domain-shift limits.
  3. New = scale, zero-shot transfer, language grounding, orchestration. Not new = domain shift, reconstruction dependence, leakage/eval discipline, calibration.
  4. Most are preprints with limited external validation — read for evaluation methodology, treat “generalist” claims as orientation, not deployment evidence.

Next: whatever the model class, how to evaluate it beyond one metric.