Why Dice is not enough

Tonight · ~25 min · read · energy: medium · setup: none

Dice (and IoU) is the default segmentation metric, and it is misleading as a sole verdict. This lesson is why: Dice is size-dependent, hides boundary errors, is contextualised by reader variability, and — most importantly — says nothing about whether the mask serves its downstream measurement. By the end, “Dice 0.91” will read to you as “one facet, pending the rest”.

The anchor: the same boundary error, very different Dice

Dice measures overlap — twice the intersection over the sum of the volumes. The trap is that the same absolute boundary error produces a very different Dice penalty depending on the structure’s size. A one-voxel boundary slip on a small tumour can crater Dice; the same slip on a large organ barely moves it. Conversely, a high Dice on a large organ can hide clinically relevant local boundary errors. Dice is a gross-overlap summary, not a boundary-quality measure.

The metrics, and what each hides

Dice / IoU (Jaccard) — overlap. Strongly size-dependent (above). A high Dice on a large structure can hide local boundary errors; a small/thin structure needs a much better mask to reach the same Dice.
Hausdorff distance (and 95th-percentile / symmetric variants) — worst-case surface error; dominated by a single outlier voxel. Robust percentiles (e.g. 95%) help tame that one-voxel tyranny.
Average / mean surface distance — typical boundary error; complements Hausdorff by describing the common case rather than the worst.
Always complement overlap with surface metrics + task-aware reading. Report per-structure metrics, and interpret small/thin-structure Dice with care.

The reference standard is itself noisy

The “ground truth” mask is a human annotation, and humans disagree. Inter-reader variability contextualises the uncertainty of the reference standard and helps define the scale at which score differences are meaningful — it is not a universal numeric “ceiling” a model can never cross. A model can agree with a consensus reference differently from any individual reader, so compare against the right reference (single reader, consensus, or adjudicated) and report reader spread. Without that, a 0.02 Dice difference between two models is uninterpretable: it may be smaller than the reader-to-reader spread.

Judge a segmentation by the downstream measurement (does the tumour volume it implies agree with a human’s?), not a single overlap number.

This is the rule from lesson 2: because the mask is infrastructure, the metric that matters is the one on the task the mask serves. A model with lower Dice but a volume that matches a human reader is the better measurement tool; a model with higher Dice but a biased volume is worse for quantification.

Stop and think — then reveal

Two liver-segmentation models both reach Dice 0.95 against the same reference. One has a 95th-percentile Hausdorff of 2 mm; the other, 18 mm. For a volumetry downstream task, which do you prefer, and what does the Dice alone hide?

Prefer the 2 mm model — but note that Dice alone hid a 9× difference in worst-case boundary error. The high-Dice/18 mm model has large local boundary excursions that the overlap summary averaged away; those excursions bias the implied volume and any boundary-sensitive feature even though “Dice looks fine”. This is exactly why overlap must be complemented by surface metrics and the downstream-task check: two masks that are “equally good” by Dice can be very different by the measure that actually matters.

What to retain

Dice is size-dependent: the same boundary error penalises small/thin structures far more than large ones, and high Dice can hide local boundary errors.
Complement overlap with surface metrics (Hausdorff, 95%-Hausdorff, mean surface distance) and read them per-structure.
The reference standard is noisy; inter-reader variability contextualises what a score difference means — it is not a hard ceiling.
The metric that matters is the downstream one: does the mask’s implied measurement agree with a human’s? Judge by that, not by Dice.

Next: the tool that made good masks cheap — why nnU-Net became such a strong baseline.