Skip to content
Imaging Evidence Path

What exactly is the unit of analysis?

Tonight · ~15 min · read · energy: low · setup: none

The Chapter 4 lesson on independence established that the split unit is usually the patient. This short lesson turns that into a reading question: what is the unit of analysis in this paper, and does the split respect it? It is the single most common place radiomics papers inflate their N.

The anchor: a paper’s N must be honest

A paper reports “AUROC on 600 images”. Are those 600 independent observations, or 30 patients × 20 slices? If the latter, the effective N for generalisation is ~30, and the 600 is a mirage. The unit of analysis decides what the model is predicting for (patient? lesion? scan? slice?), and the split unit decides how many independent tests of generalisation the paper actually ran.

The reading questions

  • What is the unit of analysis? Patient-level diagnosis, lesion-level prediction, scan-level longitudinal, or slice-level training? Each implies a different estimand.
  • Are repeated measurements handled? Multiple lesions per patient, repeated scans, bilateral organs — these are clustered, not independent. The model must account for within-patient dependence, and the split must keep a patient’s correlated data in one fold.
  • Did the split respect the patient? If slices or scans of the same patient appear in both train and test, N is inflated and performance is optimistic (Ch. 4).

Why this is the first integrity check

Because it is invisible in the abstract. “Validated on N=600” sounds strong until you learn N=600 is 30 patients’ slices. The reconstruction paragraph (lesson 1) should make the unit explicit; if the paper does not state it, treat the N as suspect. This is also a design lesson: when you report a study, state the unit of analysis and the split unit explicitly, so a reader does not have to guess whether your N is honest.

Stop and think — then reveal

A paper reports a lesion-classification model “validated on 800 lesions from 40 patients” with a random 80/20 split over the 800 lesions. What is wrong, and what is the honest reformulation?

The split was over lesions, not patients, so lesions from the same patient sit in both train and test — within-patient correlation leaks across the boundary and the 800-lesion N overstates the independent test of generalisation (the honest test N is closer to the held-out patients, ~8). The honest reformulation: split at the patient level (all of a patient’s lesions in one fold), report the unit of analysis as “lesion, clustered by patient”, and account for within-patient dependence in the analysis. The AUROC will usually drop — often substantially — when the leak is removed.

What to retain

  1. The unit of analysis (patient/lesion/scan/slice) decides the estimand; the split unit decides the honest test N.
  2. Repeated measurements (lesions, scans, bilateral organs) are clustered — handle within-patient dependence and keep a patient in one fold.
  3. “Validated on N images” is suspect unless the paper states the unit and that the split was patient-level; invisible N-inflation is the classic radiomics error.
  4. State the unit and split explicitly when you report your own study.

Next: the input the unit is labelled against — where did the labels come from?