Segmentation and Task-Aware Evaluation

A mask is judged through its downstream use

Once a mask is understood as measurement infrastructure, a segmentation score can no longer be the whole conclusion. Dice measures overlap between a prediction and a reference: in set notation it is twice the intersection divided by the sum of the two mask volumes. That is a useful question, but it is only one question. The same absolute boundary displacement can have a large relative effect on a small structure and a smaller effect on a large structure. A high overlap on a large structure can also coexist with a local excursion that matters for a boundary-sensitive measurement.

S and L are synthetic reference masks of different sizes; throughout the figure, dashed outlines mark a reference (S, L, or R), while solid hatched outlines mark a prediction P (S′, L′, P1, or P2). Panel A applies the same one-pixel outward dilation with a 3×3 structuring element to both masks, and Dice is computed from the drawn masks: 32/52 ≈ 0.615 for S versus 200/244 ≈ 0.820 for L — the same boundary dilation is a large relative change for the small structure and a small one for the large. Panel B keeps one reference R and compares two predictions: P1 is the one-pixel outward dilation; P2 removes a 5×5 corner notch (25 px) and adds a 4×2 edge bump (8 px). Both score Dice 200/244 = 150/183 ≈ 0.820 — exactly equal by construction — yet HD95, the 95th percentile of pooled surface-to-surface distances (not the worst case), is 1.02 px for P1 versus 4.00 px for P2, and the implied area change ΔA = area(P) − area(R) is +44 px for P1 versus −17 px for P2. Equal overlap does not mean equal spatial error or equal downstream measurement; these synthetic numbers are illustrations, not clinical thresholds, and Dice remains useful — just not sufficient. Which difference matters depends on the downstream task.

Overlap, surface distance, and size answer different questions

IoU, also called the Jaccard index, is another overlap summary with the same general limitation: its response depends on the structure and error geometry. Surface-distance measures ask a different question about separation between boundaries. The Hausdorff family can be dominated by an extreme point, while a percentile variant reduces the influence of such an outlier. The label HD95 does not define one universal implementation, however. A study must state the surface construction, directionality, symmetry, pooling, spacing, and percentile convention before two values can be compared. Mean or average surface distance summarises a typical separation under its own convention; it is not a substitute for overlap or for the task-level consequence.

This is why per-structure reporting matters. A single aggregate can hide a failure concentrated in small or thin structures, while a single surface number can hide a broad but modest overlap loss. The metric set should be selected from the intended use, and the implementation details should be part of the reported measurement rather than inferred from a familiar metric name.

Several edge cases need their own declared conventions. If both prediction and reference are empty, if a class is absent from one or both masks, or if an expected structure is missing, state whether the case counts as agreement, is undefined, is scored as failure, or is excluded; do not let a software default silently choose the estimand. For small or thin structures, report the relevant structures separately because a boundary displacement can dominate the summary. Also state whether aggregation is per case, per structure, macro, micro, or volume-weighted, since unequal weighting changes the question being answered. For a surface-distance measure, declare the surface construction, voxel or physical units, directionality, symmetry, pooling, spacing, percentile, and empty-surface handling. Finally, name at least one task-linked quantitative endpoint, such as physical-volume error for volumetry or feature stability under a declared mask perturbation for radiomics, together with the uncertainty procedure for that endpoint. These conventions make the result interpretable; they do not make one metric or one mask universally valid.

Reference labels are measurements with their own variability

The reference mask is generally an annotation, not an infallible boundary of nature. Readers can disagree about indistinct edges, small lesions, or the appropriate extent of a region. A consensus or adjudicated reference can be useful, but it represents a particular annotation procedure and may not coincide with every individual reader. Inter-reader spread therefore provides context for interpreting a model difference; it is not a universal numerical ceiling that a model cannot cross.

The comparison must also match the intended target. A model designed to reproduce one reader should not be judged as if it were estimating an adjudicated consensus without explaining the change. Conversely, agreement with a consensus does not prove agreement with every reader or downstream measurement. A small difference between two overlap scores is uninterpretable until the reference procedure, sample, uncertainty, and task are visible.

The downstream measurement completes the evaluation

Suppose two masks have similar overlap against the same reference. For a prespecified downstream measurement, if one gives an estimate that agrees more closely with the intended measurement and its uncertainty is characterised, that mask may be more useful for that task than one with a local boundary excursion that biases the estimate. A lower Dice mask can be preferable for one measurement task, while a higher Dice mask can be preferable for another. This is not a paradox; it reflects different estimands, which must be declared before one mask is called more useful. The word “useful” is therefore a qualitative, task-specific conclusion rather than a ranking supplied by Dice alone.

The evaluation of a region-producing model should consequently connect at least four layers: overlap, spatial boundary behaviour, reference-standard variability, and the downstream task. Visual quality control can complement those analyses by checking anatomy and geometry, but a visual impression is not a numerical estimate of measurement agreement. Neither visual inspection nor a set of segmentation metrics, alone or together, certifies the downstream endpoint; that requires evaluating the declared measurement. The next section turns from judging a mask to configuring a segmentation pipeline, where the same distinction prevents a reported architectural margin from being credited automatically when it may instead reflect a change in preprocessing, spacing, or training procedure.