Research methodology — reference

Reference · The reading-and-design checklist and frameworks table. Teaching versions live in the lessons.

Frameworks by intended use (choose by purpose; combine when scopes overlap)

If the study is…	Reporting lens	Risk-of-bias lens
Medical-imaging AI model	CLAIM 2024	study-type dependent; PROBAST+AI when also a clinical prediction model
Clinical prediction model	TRIPOD+AI	PROBAST+AI
Clinical radiomics study	CLEAR	CLEAR / METRICS
Radiomic feature computation	IBSI 1/2	(reference values)

CLAIM 2024 is a reporting lens, not a formal RoB tool. CLAIM did not extend to radiomics — use CLEAR for radiomics. CLAIM + TRIPOD+AI can be complementary; PROBAST+AI supplies the RoB lens. All linked in resources/standards-and-reference.md.

The reading-and-design checklist

For each item: did the study do this, do it well, and report it?

Question and design — precise clinical question; prediction vs diagnosis vs prognosis vs segmentation; retrospective vs prospective; cohort selection / inclusion-exclusion (spectrum bias).

Unit and reference standard — unit of analysis (patient/lesion/slice/scan); reference standard and its independence from the input; repeated measurements grouped by patient; verification bias.

Sample and splits — sample size / events-per-variable; leakage (patient/slice, preprocessing, feature selection, hyperparameter, segmentation); missing data imputation inside CV.

Testing terminology — internal / temporal / external (precise); “validation” is ambiguous.

Performance, honestly — discrimination; calibration; clinical utility / decision- curve; uncertainty / CIs; pre-specified vs post-hoc subgroup analysis.

Special cases — survival/censoring (Cox assumptions, C-index, calibration, incremental value beyond baseline); multiple testing; reproducibility/open science (binWidth/spacing/interpolation/preprocessing/model/code reported?).

Claims — do claims exceed the evidence? “biomarker” vs “feature”; “generalisable” vs “internally tested at one site”; “clinically useful” vs “high AUROC”.

Overclaim vocabulary

• “Biomarker” → unvalidated feature.
• “Generalisable” → check testing type and frozen-vs-retrained.
• “Clinically useful” → needs calibration + decision impact.
• “Validated” / “external validation” → check testing type, frozen weights, frozen threshold.

What usually goes wrong (in the papers you read)

• No external test; “generalisable” from single-site data.
• Feature selection / cohort-fitted preprocessing outside CV (optimism).
• AUROC reported; calibration and decision-impact not.
• “Biomarker” for an unvalidated feature; binWidth/spacing unreported (radiomics).
• Underpowered or post-hoc subgroup analyses.
• Using CLAIM as a radiomics checklist (it does not cover radiomics — use CLEAR).

TRACE-CT connection

TRACE-CT’s docs/experiment-protocol.md fixes the patient-level split, forbids cohort-fitted steps outside the CV loop, and records reconstruction variables — a concrete example of a leakage-safe design. Use it as a worked example when reading a paper that lacks one.

Go deeper

• Frameworks: CLAIM 2024, TRIPOD+AI, PROBAST+AI, CLEAR (+ CLEAR-E3 / METRICS), IBSI 1/2 — all in resources/standards-and-reference.md.
• One paper at a time from resources/literature-map.md, read with the lens above.

What to retain (chapter summary)

1. Read/design in order: question → design → cohort → unit → reference standard → splits → leakage → testing type → discrimination → calibration → utility → uncertainty → claims.
2. Use internal/external testing precisely; “validation” is ambiguous.
3. Choose the framework by study type: TRIPOD+AI/PROBAST+AI (prediction), CLAIM 2024 (imaging-AI model), CLEAR (radiomics — not CLAIM), IBSI (computation).
4. Reproducibility is decided by reporting binWidth/spacing/preprocessing; their absence is a red flag. High discrimination ≠ useful ≠ biomarker.