Skip to content
Imaging Evidence Path

Do the metrics support the clinical claim?

Tonight · ~15 min · read · energy: low · setup: none

An honest test set and a single AUROC still do not establish a clinical claim. This lesson is the reading question that asks whether the reported metrics actually support what the authors conclude — discrimination, calibration, utility, and uncertainty, each in turn.

The anchor: a clinical claim needs more than discrimination

A paper that reports AUROC and concludes “the model is clinically useful” has skipped the steps in between. Discrimination (AUROC/C-index) is necessary; it is not the verdict. The reading question is whether the paper reports — and the claim reflects — calibration, clinical utility, and uncertainty, or whether it leaps from a ranking metric to a clinical conclusion.

The four facets to demand

(from Ch. 4, lesson 5, now as reading questions)

  • Discrimination — AUROC / C-index. Reported, but is it the only metric?
  • Calibration — do predicted risks match observed? If calibration is unreported, the predicted probabilities are not shown to be actionable. A perfectly ranking, miscalibrated model is common and clinically dangerous.
  • Clinical utility / decision-curve — does the model change decisions for the better (net benefit over a clinically relevant threshold range)? High AUROC with no net benefit is clinically pointless.
  • Uncertainty / CIs — are patient-level confidence intervals reported, or only a population point estimate? And are subgroup analyses pre-specified and powered, or post-hoc?

The leap to watch for

The most common over-leap is from discrimination straight to utility: “AUROC 0.93, therefore clinically useful”. The missing middle is calibration and decision impact. A model can rank beautifully, be miscalibrated, and offer no net benefit over current practice — three independent failures that a single AUROC hides. When you read a conclusion, map it back to the metrics: does the strongest metric reported actually support the strongest claim made?

Survival, again

For survival claims the same applies with C-index plus calibration of survival predictions, plus the incremental-value question: does imaging add value beyond the clinical baseline (Ch. 4)? A “prognostic” feature that merely proxies stage is not a contribution.

Stop and think — then reveal

A paper reports AUROC 0.94 and concludes “our model is ready for clinical deployment to triage patients”. Calibration, decision-curve analysis, and patient-level confidence intervals are absent. Which part of the conclusion is unsupported, and what would you demand?

The “ready for clinical deployment to triage” part is unsupported. AUROC 0.94 shows good discrimination only. Triage is a decision that depends on calibration (are the predicted risks at the triage threshold correct?), on net benefit (does triaging by the model improve outcomes vs current practice over the relevant threshold range?), and on uncertainty (how wide are the patient-level CIs at the threshold?). Demand a calibration plot, a decision-curve/net-benefit analysis over the triage threshold range, patient-level CIs, and — because deployment is claimed — evidence about workflow behaviour and drift (Ch. 9). None of those follow from a single AUROC.

What to retain

  1. A clinical claim needs more than discrimination: demand calibration, utility, and uncertainty alongside AUROC.
  2. The common over-leap is “high AUROC ⇒ clinically useful”; the missing middle is calibration and decision impact.
  3. Map the conclusion back to the metrics — does the strongest metric support the strongest claim?
  4. For survival, add calibration of survival predictions and the incremental-value question (beyond the clinical baseline).

Next: the words to watch for — recognise an overclaim.