Foundation and multimodal models

Tonight · ~25 min · read · energy: medium · setup: none

Foundation models, vision-language models and generalist imaging models are the medium-term substrate for segmentation infrastructure and assisted interpretation. The goal here is orientation and informed paper reading, not implementation: what is genuinely new, what stays the same, and where the evidence boundary is today.

The anchor: a progression of generality

These are overlapping descriptions, not steps on a universal maturity ladder. A model may fit several descriptions. Broader scope does not establish better clinical performance or utility. Evaluate the actual task, training scope, adaptation, external testing, domain shift, licence and availability, and applicable grounding, hallucination, and provenance risks.

• Task-specific model: task scope.
• Pretrained model: training history.
• Foundation model: breadth of pretraining and adaptation.
• Vision-language model (VLM): modality combination.
• Generalist model: intended task breadth.
• Tool-using / agentic workflow: workflow organization.

The questions to ask of any paper on this curve are stable: training data scope, which tasks are zero-shot vs adapted, what external testing, licence/availability, and where it would break under domain shift.

Verified examples (publication status marked)

• TotalSegmentator (PMID 37795137, peer-reviewed) — operational now; the segmentation-infra layer (lesson 4).
• Merlin — CT vision-language foundation model. Peer-reviewed: Nature 2026, doi:10.1038/s41586-026-10181-8, PMID 41781626 (VERIFIED; earlier preprint PMID 38978576, 2024). 3D VLM for abdominal CT; tasks incl. zero-shot findings classification, phenotype classification, report generation, 3D segmentation; internal + large external validation. A reference for foundation-model evaluation at scale.
• a2z-1 (Rajpurkar et al., arXiv:2412.12629, VERIFIED) — generalist abdomen-pelvis CT, external validation across many conditions. Preprint — strong external-validation methodology; revisit on publication.
• OmniMRI (He et al., arXiv:2508.17524, VERIFIED) — unified vision-language foundation model for generalist MRI (reconstruction, segmentation, detection, diagnosis, report generation). Preprint (2025-08-24), MGH/Harvard.
• Paschali et al., “Foundation Models in Radiology: What, How, Why, and Why Not,” Radiology 2025;314(2):e240597, PMID 39903075 (VERIFIED) — the orientation/review to read first for the landscape and limits.
• ReCo (Tzanis & Klontzas, medRxiv 2026.07.14.26358025, VERIFIED preprint; github.com/eltzanis/ReCo) — self-configuring agentic framework using MCP servers, integrating Orthanc + Merlin + TotalSegmentator + DosimeTron. An example of the agentic/MCP orchestration direction — speculative without strong clinical validation yet.

Goal here: orientation and informed paper reading, not immediate implementation. When you read a foundation-model paper, ask: training data scope, which tasks are zero-shot vs adapted, what external testing, licence/availability, and where it would break under domain shift.

What is genuinely new, and what is not

• New — scale (one model, many tasks), zero-shot transfer to new findings, natural-language grounding (report generation, VLM queries), and the orchestration of multiple tools by an agent.
• Not new — the domain-specific hazards. Foundation models still face domain shift (Ch. 7), reconstruction dependence (lesson 5), and the imaging-specific leakage and evaluation discipline of all of Chapter 7. A bigger model is not exempt from calibration, external testing, or honest reporting.

The evidence boundary today

Most foundation/generalist imaging models are preprints or have limited external validation; the field is moving faster than peer review. Treat “generalist model that does everything” claims as orientation, not deployment evidence. The strongest current examples (Merlin, peer-reviewed with large external validation) are references for how to evaluate at scale — read them for the evaluation methodology, not just the headline tasks.

Stop and think — then reveal

A preprint announces a “generalist foundation model for medical imaging” with strong zero-shot numbers on internal data and no external test. Which three questions from this chapter’s checklist decide whether the claim is credible?

1. Training data scope — what (and whose) data was it trained on, and does the “zero-shot” test set leak into that training pool?
2. External testing — is there any independent site/scanner evaluation? (Here: no, so the claim is internal-only — Ch. 7.)
3. Domain-shift and failure surface — where would it break (vendor, modality, reconstruction), and are grounding/hallucination/provenance addressed? Without these, “generalist” is an aspiration, not a demonstrated capability. The preprint status means revisiting on peer review.

What to retain

1. These are overlapping descriptions: task-specific, pretrained, foundation, VLM, generalist, and tool-using / agentic workflow. Broader scope does not itself establish better clinical performance or utility.
2. Stable questions for any paper on the curve: data scope, zero-shot vs adapted, external testing, licence, domain-shift limits.
3. New = scale, zero-shot transfer, language grounding, orchestration. Not new = domain shift, reconstruction dependence, leakage/eval discipline, calibration.
4. Most are preprints with limited external validation — read for evaluation methodology, treat “generalist” claims as orientation, not deployment evidence.

Next: whatever the model class, how to evaluate it beyond one metric.