Do the metrics support the claim?

Once the evaluation boundary is understood, a second trap remains: a metric can be correctly calculated and still answer a narrower question than the conclusion suggests. The reader should move from the estimand to the metric, not from the most impressive metric to the broadest claim. Discrimination, calibration, decision utility, and uncertainty describe different properties of a model and require different evidence.

Start with the estimand, then choose the metric

For a binary diagnostic task, AUROC describes discrimination: how well scores rank observations with one recorded outcome above observations with another. It does not, by itself, describe the probability of the outcome, the effect of using a threshold, or the benefit of changing clinical practice. A C-index plays a related ranking role for some time-to-event analyses, but it does not turn a prognostic association into a causal effect or a deployment result.

The interpretation also depends on the observation unit, reference standard, cohort spectrum, and test procedure established in the preceding sections. An AUROC at the slice level against a report-derived label is not interchangeable with an AUROC at the patient level against an independent clinical reference. The same abbreviation can name a statistic while the estimand changes under the surface.

Discrimination and calibration answer different questions

Calibration asks whether predicted risks correspond to observed event frequencies under a defined population, outcome, and time horizon. A model can rank patients in the right order while systematically assigning risks that are too high or too low. If a clinician acts on a predicted 30 percent risk, the meaning of that probability matters; a ranking statistic does not establish it.

Imagine two models with AUROC 0.90 on the same constructed test sample. Model A assigns risks that are close to the observed event frequencies. Model B preserves the same ranking but is miscalibrated: a group assigned 20 percent risk experiences the outcome at a substantially different rate. Their AUROCs can be identical while their probabilities have different meanings. Calibration plots, calibration summaries, and appropriate uncertainty around them address a question that the ROC curve leaves open.

This is a reason for caution, not a license to infer that an unreported calibration analysis would have failed. If calibration is not reported, the paper has not established that its scores are actionable probabilities. The reader should preserve that limitation rather than replace it with either “the model is calibrated” or “the model is miscalibrated.”

Utility is a decision question

Decision-analytic utility asks what a score could contribute when used to choose an action under stated assumptions; it is not the same as observed clinical impact after deployment. Decision-curve or net-benefit analysis can compare using the model with simple strategies such as treating everyone or treating no one over a range of threshold probabilities. The result depends on the population, decision, threshold range, outcome definition, relative consequences assigned to false positives and false negatives, and the behavior that follows the score. Net benefit therefore estimates potential decision value under these assumptions, not what happens after deployment, and is not a property of AUROC alone.

A model can therefore rank well and still add little or no decision value over current practice at the thresholds that matter. Conversely, a useful decision analysis must state enough about the intended action and threshold assumptions for a reader to understand what “benefit” means. A high AUROC without such an analysis supports a discrimination statement, not a conclusion that the model improves care. Claims about deployment or improved care require separate impact evidence; decision-curve analysis alone does not provide it.

Uncertainty has more than one target

An interval around a population metric describes uncertainty in that summary under a stated sampling or resampling procedure. It is not automatically uncertainty about an individual patient’s risk. Calibration has its own estimation uncertainty, and an individual prediction can be affected by measurement error, limited representation of the patient’s domain, model variation, and other sources that an aggregate AUROC interval does not express. There is no generic “patient-level confidence interval” that can be demanded without defining the quantity and method.

The reader should ask which uncertainty target was reported, what unit was resampled, and whether clustering or repeated measurements were preserved. Subgroup results need the same care. Name the quantity and comparison explicitly: subgroup-specific predictive performance or the difference in that performance between subgroups. Predictive performance heterogeneity is not the same as causal effect modification; reserve “subgroup effect” for a defined treatment or exposure contrast and interaction estimand. Report intervals and, where appropriate, a formal between-subgroup comparison; state whether the analysis was prespecified, account for multiplicity, and make the effective sample size visible. A subgroup selected after inspecting the data is not equivalent to a pre-specified subgroup, and a point estimate in a small subgroup does not establish stable performance heterogeneity. The absence of an interval does not prove instability; it limits how confidently the result can be compared or transported.

Survival and incremental value

For a survival claim, the metric and the time structure must still match the question. A C-index can describe ordering of observed survival information under the study’s censoring and analysis choices, but calibration of predicted survival at relevant horizons addresses a different property. A prognostic feature or model also needs comparison with a clinical baseline when the claim is that imaging adds information. An association that disappears after the baseline variables are included may still be scientifically interesting, but it does not establish incremental predictive value.

Prediction, association, and causation

Retrospective imaging studies commonly begin with data that already exist: images, reports, treatments, and outcomes recorded through a clinical pathway. An observed association can arise from the condition of interest, but it can also arise from selection into the dataset, a confounder, a measurement process, or a variable that occurs after the relevant exposure. A model can exploit a proxy that predicts well in the sample without identifying the mechanism that produced the outcome.

Prediction asks how well the model estimates or ranks an outcome under a specified data-generating setting. Causal inference asks what would happen under a defined change in exposure or treatment, with temporal ordering, a causal estimand, a design, and assumptions about confounding and measurement. A high AUROC, a significant association, or a stable retrospective pattern does not answer that intervention question. The paper’s verbs must therefore match its design: “associated with,” “predicted,” and “caused” are not interchangeable.

What missing evidence can and cannot tell you

Suppose a constructed paper reports AUROC 0.94 for triage but gives no calibration analysis, decision-curve analysis, individual-prediction uncertainty analysis, workflow evaluation, or drift assessment. If the test design is otherwise sound, the number can support a statement about discrimination in that specified test. It cannot, by itself, support readiness for clinical triage. The missing analyses do not prove that the model would fail; they leave the decision and deployment claims unestablished.

This asymmetry is important. A missing method is not evidence that the hidden method was wrong, and a reported method is not proof that it was implemented without error. Reporting completeness tells the reader which questions the paper permits them to inspect. It does not replace the inspection. The next step is to put the same boundary into the words used by the conclusion, so that terms such as “biomarker,” “validated,” and “clinically useful” do not quietly expand the evidence.