SEG, RTSTRUCT and binary masks are not the same

“A segmentation” sounds like one thing. It is three different data structures, each of which preserves and destroys different information. This section is about what each one actually is — what it stores, how its geometry is represented, and what that means for quantitative use later — so that when a mask goes missing or a boundary shifts you know which representation lost it.

The same tumour, three storage shapes

You segment a tumour and can save it as: a binary NumPy mask, a DICOM SEG, or an RTSTRUCT. They all describe “these voxels are tumour”, but they carry different geometry, different references, and different fidelity. Converting between them is lossy in different ways. Knowing those losses is what stops you from quietly corrupting a segmentation every time you change tools.

Binary label map (mask)

A binary label map is a discrete raster representation: a 3D array where each element holds a label (1 = tumour, 0 = background; optionally multi-label). The array sits on its own grid — a declared or externally associated origin, spacing, direction, and extent — and does not inherently share any image’s shape or geometry. It is also a final representation: a deep segmentation network typically emits per-voxel logits or probabilities, and the binary label map is what you obtain after thresholding or per-element winner selection. That final label array is what PyRadiomics consumes.

Its virtue is consumption: a mask is trivially usable once it is aligned with the analysis grid. Its weakness is provenance and fidelity. A bare array has no explicit link back to the source images unless you keep it alongside the image, and fractional boundary information is quantised away the moment a mask is binary. It also carries a critical geometry requirement, the exact subject of the shared-space section: direct voxelwise use against a reference image requires complete grid equality — the same origin, spacing, direction, and size/extent, within whatever tolerance your pipeline specifies. If the grids differ, the mask needs an explicit geometry-aware transform or resampling onto the reference grid before the two arrays are combined element by element. A mask living on another valid grid is not itself a bug; silently treating it as aligned with the image is.

DICOM SEG (Segmentation Storage)

A DICOM Segmentation object is a first-class DICOM object, and several of its features make it more than a mask array.

Its source reference can cite the CT. ReferencedSeriesSequence (PS3.3 C.12.2) lists referenced series and instances, and per frame a DerivationImageSequence with SourceImageSequence entries can carry the source SOPInstanceUIDs. These references are conditional, not guaranteed: allowed SEG encodings may omit or leave the per-frame derivation references empty (for instance when frame geometry is tied to the source series through the Frame of Reference instead), and even when present they identify the referenced objects rather than certifying that one CT is the sole origin. A SEG that carries them gives you a machine-readable starting point for “which images is this mask for?”; a SEG that lacks them has not thereby been shown to come from nowhere.

Its geometry is expressed per frame or through shared functional groups: the Segmentation functional groups (PS3.3 A.51.5) provide plane position, plane orientation, and pixel measures either per frame or shared across frames rather than independently on every frame. Which conditional groups apply depends on the encoding. Empty frames may also be omitted. The frame positions a SEG declares are its own: a SEG is not necessarily on the CT grid — the standard permits SEG frames with a different size or resolution from the referenced images — so it can describe a sparse set of labelled planes, which is why the shared-space procedure of the previous section applies to SEG objects before any overlay.

Its segment identity is explicit: SegmentSequence defines segment numbers, labels, and properties (e.g. “Tumour”, “Liver”), so a multi-structure SEG stays self-describing instead of collapsing to anonymous integers.

Its encodings differ in what they store per voxel. Segmentation Type (0062,0001) distinguishes encodings including BINARY (1 bit per voxel, in/out of the segment), FRACTIONAL (stored samples are 8-bit unsigned integers; the fractional value is recovered as the stored sample divided by Maximum Fractional Value, and Segmentation Fractional Type declares whether that ratio means PROBABILITY or OCCUPANCY — the stored integer is a scaled code, not itself a probability), and LABELMAP (integer label per voxel, 8 or 16-bit, encoding multiple non-overlapping segments in one object — the closest DICOM analogue of a multi-label mask array) (PS3.3 C.8.20.2). For pixel data, binary segmentation is packed 8 one-bit pixels per byte (PS3.5 bit-packing); fractional and labelmap encodings use byte- or word-sized integer samples and are more array-like but still need a decoder. Use highdicom, dcmqi, or TRACE-CT’s decode_seg_file().

RTSTRUCT (RT Structure Set Storage)

The radiotherapy legacy format stores structures as contours — ordered lists of 3D patient-coordinate points — not as voxels. PS3.3 (C.8.8.6) defines the contour geometric types POINT, OPEN_PLANAR, OPEN_NONPLANAR, CLOSED_PLANAR, and CLOSEDPLANAR_XOR, so not every contour is a closed slice outline, and the optional Source Pixel Planes Characteristics attributes may record the image planes the contours were drawn on. Geometry lives in the contour coordinates, and a voxel interior only exists after a rasterisation step fills the contours on a target grid — a grid the format does not supply; you choose it. What can go wrong in that step depends on how it is done: the selected target grid and rasterisation convention, how ambiguous or malformed input is handled (nested contours that imply holes, unsupported contour types, inconsistent contour-to-plane association), and what plane association is assumed for the contours. A conformant closed-planar contour is closed by definition; the practical risk is convention ambiguity and edge cases that are invisible in the resulting mask, so RTSTRUCT↔mask conversion is tool-dependent and needs verification rather than blind trust.

Where information is silently lost

Round-tripping between representations destroys information in specific ways. Some losses are inherent to the representation change; others happen only when the conversion’s chosen policy performs them. This is the reference list to keep:

Where SEG↔mask loses information:

Inherent to the representation change:

• binary quantisation of a fractional boundary (a 0.7-occupancy voxel becomes 1) whenever a FRACTIONAL SEG is turned into a binary mask;
• loss of the source-image references when you export only a bare .nrrd mask.

Only under the chosen conversion policy:

• nearest-neighbour snapping when the SEG frame grid is finer or coarser than the CT — this occurs only if the conversion resamples onto that grid with a nearest-neighbour interpolator;
• dropped frames for segments outside the reference image extent — this occurs only if the selected extent policy discards them instead of keeping or extending the grid.

The reference loss is the easiest to incur and the hardest to notice, and what else survives depends on the export format and what it carries. A bare binary .nrrd mask exported without preserved metadata loses the DICOM-specific layer: the source-image references (the ReferencedSeriesSequence / source SOPInstanceUIDs that tied the mask to specific CT slices — the bare array no longer says which CT it belongs to) and the coded segment descriptions (the named, coded segment properties behind the labels). Neither loss is forced by the array format itself: NRRD can carry grid geometry and custom key–value metadata in its header or a sidecar, and a multi-label export keeps each segment’s integer identity. Fractional boundary detail is lost only when the export is binarised — a FRACTIONAL SEG exported with its fractional values intact keeps them; a binary export quantises them to 0/1. For computation a mask is fine; for exchange and provenance, keep the SEG — or at least record the source UIDs and segment descriptions alongside the mask.

Schematic representation, not full DICOM encoding: real SEG pixel-data encoding includes BINARY bit packing and FRACTIONAL value scaling, and the standards also define per-frame geometry, source-image references, and RTSTRUCT contour semantics. The 0.5 threshold and the way the 16-vertex contour was created are choices of this example, not universal conventions, and no representation shown is universally best.

The two conversions that matter for the CT/radiomics path are the two you will actually perform. One: decode a SEG onto a reference grid, validating that the SEG’s per-frame geometry matches the CT — or resampling it where it does not. Two: round-trip a mask (mask → SEG → mask) and know where it lost fidelity: fractional edges collapse to binary whenever the round trip binarises, and frames that sit off the target grid snap to the nearest reference voxel only if that round trip’s chosen policy resamples them nearest-neighbour — a policy choice, not a law of the format.

Both conversions run head-first into the same operation — moving voxel data between grids — and the image and the mask do not survive that move under the same rules. The next section explains why image and mask resampling must differ, and why that choice reaches every boundary-dependent feature.

Sources

• DICOM PS3.3, Segmentation Image Module (C.8.20.2) — Segmentation Type (0062,0001) values and encoding constraints (BINARY 1-bit, FRACTIONAL 8-bit samples with Segmentation Fractional Type / Maximum Fractional Value, LABELMAP 8/16-bit non-overlapping), the SegmentSequence segment descriptions, and BINARY bit-packing requirements. (current edition)
• DICOM PS3.3, Segmentation functional groups (A.51.5) — the conditional per-frame Pixel Measures, Plane Position (Patient), Plane Orientation (Patient), Derivation Image, and Source Image groups, including when they may be shared across frames or omitted. (current edition)
• DICOM PS3.3, Common Instance Reference Module (C.12.2) — ReferencedSeriesSequence and ReferencedInstanceSequence semantics. (current edition)
• DICOM PS3.5, Encoding of Pixel, Overlay and Waveform Data (Section 8) — native encoding and bit-packing of pixel data, including one-bit BINARY samples. (current edition)
• DICOM PS3.3, ROI Contour Module (C.8.8.6) — RTSTRUCT contour sequences, contour data, contour geometric types (closed/open, planar/non-planar, point), and Source Pixel Planes Characteristics. (current edition)
• highdicom documentation, Segmentation (SEG) Images — construction and decoding of BINARY, FRACTIONAL, and LABELMAP segmentations, frame geometry, and per-frame derivation sequences. (stable docs)
• dcmqi guide — segimage2itkimage / itkimage2segimage: DICOM SEG ↔ ITK/NRRD label-image conversion and the metadata the conversions require. (official guide)
• TRACE-CT companion implementation — this project’s SEG decoding workflow (decode_seg_file()), bounded to the linked commit; a companion example, not a standard. (commit)