Research methodology — reference

Reference · The reading-and-design checklist and resource table. Teaching versions live in the chapter.

Resources by intended use (choose by purpose; combine when scopes overlap)

Resource	Primary role	Scope and limit
CLAIM 2024	Reporting guidance/checklist	Medical-imaging AI model studies within its scope; not a scoring or risk-of-bias tool, and the inspected 2024 update does not cover radiomics.
TRIPOD+AI	Reporting guidance/checklist	Clinical prediction-model studies; reporting completeness does not establish low bias or clinical utility.
PROBAST+AI	Quality, risk-of-bias, and applicability assessment	Individualized diagnostic/prognostic prediction models; not a feature-computation standard.
CLEAR	Reporting guidance/checklist	Clinical radiomics studies; checklist entries document what was reported, not whether quality or validity is established.
METRICS (source)	Methodological-quality assessment/scoring	Radiomics methodology; bias-relevant appraisal, not a reporting checklist or a guarantee of low bias or clinical utility.
IBSI 1/2	Computational standardisation and definitions	Feature/filter processing definitions and reference computations; not risk-of-bias or clinical-validity assessment.
Reference values	Computational reference benchmarking	Comparing a calculation with a reference value checks computation under a defined contract, not the validity of its clinical use.

CLAIM 2024, TRIPOD+AI, and CLEAR provide reporting guidance/checklists within their study scopes; CLAIM’s inspected 2024 update does not cover radiomics, for which CLEAR is the reporting checklist here. PROBAST+AI assesses quality, risk of bias, and applicability for relevant prediction models; METRICS assesses methodological quality in radiomics. IBSI 1/2 defines computational processing and feature/filter definitions, with reference values for computational benchmarking. These resources are complementary, not interchangeable substitutes, and no checklist or score by itself proves low bias, scientific validity, reproducibility, or clinical utility. All linked in the standards and reference desk.

The reading-and-design checklist

For each item: did the study do this, do it well, and report it?

Question and design — precise clinical question; prediction vs diagnosis vs prognosis vs segmentation; retrospective vs prospective; cohort selection / inclusion-exclusion (spectrum bias).

Unit and reference standard — unit of analysis (patient/lesion/slice/scan); reference standard and its independence from the input; repeated measurements grouped by patient; verification bias.

Sample and splits — sample size / events-per-variable; leakage (patient/slice, preprocessing, feature selection, hyperparameter, segmentation); missing data imputation inside CV.

Testing terminology — internal / temporal / external (precise); “validation” is ambiguous.

Performance, honestly — discrimination; calibration; clinical utility / decision- curve; uncertainty / CIs; pre-specified vs post-hoc subgroup analysis.

Special cases — survival/censoring (Cox assumptions, C-index, calibration, incremental value beyond baseline); multiple testing; reproducibility/open science (binWidth/spacing/interpolation/preprocessing/model/code reported?).

Claims — do claims exceed the evidence? “biomarker” vs “feature”; “generalisable” vs “internally tested at one site”; “clinically useful” vs “high AUROC”.

Overclaim vocabulary

• “Biomarker” → unvalidated feature.
• “Generalisable” → check testing type and frozen-vs-retrained.
• “Clinically useful” → needs calibration + decision impact.
• “Validated” / “external validation” → check testing type, frozen weights, frozen threshold.

What usually goes wrong (in the papers you read)

• No external test; “generalisable” from single-site data.
• Feature selection / cohort-fitted preprocessing outside CV (optimism).
• AUROC reported; calibration and decision-impact not.
• “Biomarker” for an unvalidated feature; binWidth/spacing unreported (radiomics).
• Underpowered or post-hoc subgroup analyses.
• Using CLAIM as a radiomics checklist (it does not cover radiomics — use CLEAR).

TRACE-CT connection

TRACE-CT’s experiment protocol fixes the patient-level split, forbids cohort-fitted steps outside the CV loop, and records reconstruction variables — a concrete example of a leakage-safe design; the concrete artifacts are inventoried in the TRACE-CT learning map. Use it as a worked example when reading a paper that lacks one.

Go deeper

• Frameworks: CLAIM 2024, TRIPOD+AI, PROBAST+AI, CLEAR (+ CLEAR-E3 / METRICS), IBSI 1/2 — all in the standards and reference desk.
• One paper at a time from the literature map, read with the lens above.
• Annotation-process design: when designing a reader study or annotation campaign, the design pipeline — specification, reader selection, blinding, independent readings, declared adjudication/consensus rule, provenance — and the reported disagreement rate are part of the reference standard’s definition, not project metadata. Appraise them the same way in papers that describe (or omit) them.

What to retain (chapter summary)

1. Read/design in order: question → design → cohort → unit → reference standard → splits → leakage → testing type → discrimination → calibration → utility → uncertainty → claims.
2. Use internal/external testing precisely; “validation” is ambiguous.
3. Choose the framework by study type: TRIPOD+AI/PROBAST+AI (prediction), CLAIM 2024 (imaging-AI model), CLEAR (radiomics — not CLAIM), IBSI (computation).
4. Reproducibility is decided by reporting binWidth/spacing/preprocessing; their absence is a red flag. High discrimination ≠ useful ≠ biomarker.