Design before fitting

The protocol is the paper

By the time a model is fitted, many of the choices that determine the meaning of its score have already been made. The estimand, cohort, observation unit, split, fitted-state boundary, feature definition, comparison baseline, metrics, and test setting are properties of the design. Writing them after seeing the result makes it difficult to distinguish a planned analysis from a successful search.

A protocol is therefore not administrative decoration. It is a compact description of what information may influence each choice and what the result will be allowed to mean. It also makes a negative result interpretable: a model that does not improve on the baseline under a declared design has answered a real, bounded question rather than merely failed to produce a flattering number.

The design sequence

Write the following sequence before fitting, adapting the detail to the estimand rather than treating it as a universal checklist:

1. Estimand and units. State the population, outcome or measurement target, time horizon, analysis unit, split unit, and resampling unit. Explain why repeated scans, lesions, or slices are grouped or modelled as they are.
2. Partitions and access. Specify which observations belong to development, tuning, and evaluation. Freeze the membership rule before interpreting the score and keep subjects or other clusters intact where the claim requires it.
3. Fitted-state audit. List every operation that can learn from data, including scaling, imputation, feature selection, harmonisation, threshold choice, segmentation training, and model tuning. For each, record which partition may fit it and where its state is applied.
4. High-dimensional control. Define the candidate feature set, regulariser, multiplicity treatment, and, if an inferential post-selection claim is intended, the separately justified selective-inference procedure. Record how selection instability will be reported. Keep predictive regularisation distinct from inferential evidence.
5. Performance objects. Select metrics that match the outcome and claim: discrimination, calibration, decision utility, and uncertainty are separate entries, not one composite verdict. If the target is individual risk, specify the individual-uncertainty framework or leave that item explicitly open rather than substituting a population metric interval.
6. Testing relationship. Name the internal, temporal, or external setting, the shift axes it represents, and whether the model is frozen, recalibrated, thresholded again, adapted, or retrained.
7. Baseline and comparison. Define the clinical or tabular baseline and the incremental claim. Give the baseline the same split, fitted-state discipline, tuning opportunity, and reporting depth as the proposed imaging procedure.
8. Reporting and provenance. Record the outcome construction, missingness, measurement contract, and software and data versions where they are relevant for provenance. Put volatile version details in companion material rather than making them stable methodological claims. Also record the reporting framework selected for the study. A reporting guideline helps make a design visible; it does not supply empirical validation or prove that every choice was made without looking at later results.

This order is a dependency graph. A metric cannot be interpreted before the estimand and test population are known. A feature-selection result cannot be interpreted before the split and fitted-state boundary are known. A utility claim cannot be interpreted before the policy and threshold context are known.

A bounded survival design

Consider a self-contained radiomics-and-survival question: does a declared lung-tumour measurement add prognostic information beyond stage and histology for the outcome and follow-up definition chosen by the study? The protocol would first identify one survival cohort and one row per patient for the primary analysis. The tumour region is the measurement input; it is not a second set of independent patients. If a patient contributes repeated scans or multiple regions, the protocol states whether those are aggregated, modelled as clusters, or reserved for a separate measurement question.

The evaluation could use patient-disjoint nested resampling when feature and model selection are part of the predictive procedure. Every outer test patient remains outside the inner search. A same-day test-retest cohort such as RIDER could characterize measurement repeatability under a separate declared design, but it must not be silently mixed with the survival cohort or assigned to its folds merely because both datasets contain imaging.

The image-to-feature contract is written down before fitting: image and mask provenance, geometry, resampling, intensity handling, discretisation, feature implementation, and any segmentation model. A fixed per-sample operation may be outside a validation loop when its constants and role were declared independently of the evaluation data. A robustness filter, scaling state, feature selector, imputer, threshold, or harmonisation model that is learned from data belongs in the permitted training portion. A segmentation mask described as “fixed” still needs provenance; if its model was trained on overlapping evaluation patients, the input pipeline can be contaminated.

The statistical comparison is clinical-only survival model versus clinical-plus-radiomics model, with proportional-hazards assumptions, discrimination, calibration, and any decision analysis specified in advance. The result is an estimate for this outcome, cohort, measurement contract, and validation procedure. It is not a general statement that radiomics improves survival prediction, and it does not replace a genuinely independent test when a clinical transport claim is intended.

Designing and reading are inverse operations

The same sequence can be used to reconstruct a published study. Start with the claimed estimand, identify the true observation and split units, trace every fitted state, identify what was selected and when, then ask which test setting was actually represented. Only after that should a score be connected to the paper’s wording.

This is why a single AUROC or C-index cannot rescue a design that selected features on the complete cohort, mixed repeated patients across folds, or adapted on the test rows. Conversely, a modest result from a transparent design may be more valid for its declared question than a large result whose fitted- state and testing boundaries are unknown.

The protocol-first argument now meets a less tidy reality: clinical tables often contain fields that were not recorded for every subject. The next case follows the same design sequence through missingness, imputation, and leakage-safe evaluation.

Sources

• Kocak B, et al. CLEAR radiomics reporting checklist. Insights into Imaging. 2023;14:75. doi:10.1186/s13244-023-01415-8.
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378.
• Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505.
• Bernett J, et al. Guiding questions to avoid data leakage in biological machine learning applications. Nature Methods. 2024;21:1444-1453. doi:10.1038/s41592-024-02362-y.