Labels and reference standards

The phrase “ground truth” can make a label sound like a direct observation of the condition. In an imaging study, a label is usually the output of another measurement process: a reader’s interpretation, a pathology result, an algorithm applied to a report, a consensus procedure, or a clinical event record. The model’s score is conditional on that process. The first question is therefore not whether the label is metaphysically true, but what target the process measures and how it relates to the input.

The reference standard is part of the estimand

A reader looking at an MRI, a later pathology result, and an NLP label derived from a radiology report can all be useful for different questions. A model trained to reproduce a report impression has a legitimate report-reproduction target. That target should not silently be described as independent detection of the underlying disease. Conversely, a disease-oriented claim may need a reference process that is independent enough to test the intended condition, even if that process is available only for a subset.

Three properties are especially important. First, identify the relationship between the reference process and the image or other input the model sees. A label made from the same image can be dependent on the cues used by the reader; the model may learn the reporting process rather than the condition. This is a potential circularity for a disease-truth estimand, not an automatic reason to discard a report-reproduction study. Second, describe how the reference was constructed and its likely sources of error: reader expertise and instructions, retrospective documentation, NLP extraction, availability of confirmatory tests, and any selection into verification. Third, state how multiple readers’ opinions were handled and what disagreement means for the task.

Circularity depends on the target

Imagine a knee MRI model labelled from the radiology report written after a radiologist examined that same MRI. A high score against the label can mean that the model reproduces cues that lead to the report. It does not by itself show that the model identifies the tear independently of the reporting process. If the paper’s target is to automate report classification, this may be the quantity of interest. If the paper claims independent disease detection, the reference-standard dependence narrows what the result supports.

The same logic applies to a chest X-ray example in which a constructed model has AUROC 0.96 against a report-derived pneumonia label. The number is not a reported clinical result here. It illustrates that the immediate target is agreement with the report impression. An independent reference, such as a defined later adjudication or another appropriate clinical confirmation, applied according to a stated protocol would test a different target. Whether that target is feasible and appropriate depends on the disease and study question; the stronger claim cannot be obtained merely by changing the label’s name.

Verification and selected reference tests

Verification bias is a selection problem in the reference process. Suppose only people with a positive screen, a positive model, or a particularly concerning initial read receive a definitive reference test. Performance is then measured on a selected subset, not automatically on the full eligible cohort. Sensitivity, specificity, calibration, and the population to which the result applies can all be affected by that selection.

The presence of a selected reference does not make every study uninterpretable. It does mean that the paper must state who was verified, why, and what analysis was used to account for or describe the selection. If the authors intend a claim about all eligible examinations, a reference available only for a selected group leaves that claim under-supported. If the estimand is explicitly conditional on the verified group, the narrower claim should say so.

Build the labelling process explicitly

The design of a reference standard can be written as a chain:

clinical question -> annotation specification -> reader selection and blinding
  -> independent readings -> disagreement rule -> adjudication or consensus
  -> final reference -> provenance

The annotation specification says what is being labelled, at what granularity, and according to which criteria. “Tear” can refer to different findings if the reader is not told whether partial disruption, complete disruption, or any signal abnormality counts. Reader selection and blinding describe expertise and what information is available. A reader who sees the model output, another reader’s answer, a prior report, or a later clinical outcome is answering a different annotation question from a reader blinded to those signals.

Independent readings make disagreement observable. The disagreement rule then becomes part of the reference definition. A deterministic majority vote combines independently recorded opinions according to a stated rule. A discussion-based resolution lets readers or a panel review a disagreement and record a final decision. A study may assign a reader or panel a defined adjudication or arbitration role under stated criteria. Published terminology overlaps: “consensus”, “adjudication”, “arbitration”, and “majority vote” are not universal names for one operation. The paper must therefore state who read independently, what information was revealed, how disagreements were resolved, and how the final reference was formed, rather than relying on the procedure label. These procedures are not interchangeable when their operations differ. Provenance records the instructions, readers, timing, source objects, and final rule so that another reader can tell what the label means.

Disagreement is information

When two competent, blinded readers disagree on a borderline case, the disagreement is evidence about task ambiguity. It is not automatically a defect that should be removed before analysis. A model evaluated against a consensus or adjudicated label is being compared with that declared process. Agreement with one selected reader, consensus, and an independent biological reference can support different interpretations and may impose different practical limits on the attainable agreement.

Consider another constructed teaching example. Two blinded readers assess 500 studies independently and agree on 380, leaving 120 disagreements. If the readers are shown each other’s answers and only convergent cases are retained, two changes occur. The later answers are no longer independent, so anchoring can replace the original disagreement process. The retained sample also favours easier cases and changes the case spectrum. The 120 disagreements were useful information about ambiguity and about where the annotation specification or adjudication rule needed attention. They do not provide a universal numeric ceiling for every model; they characterise this task and reference procedure.

A well-described reference standard does not guarantee a valid clinical claim. It makes the outcome interpretable enough for the reader to ask the next question: did the data used to evaluate the model remain outside the fitted pipeline?