Paired uncertainty and reproducibility

An agreement coefficient is still only a point estimate. If a study reports a CCC from a few dozen paired scans, the natural next question is how much that estimate would vary across comparable samples. The bootstrap gives one way to approximate that sampling uncertainty without pretending that the observed cohort contains more biological information than it does.

One cohort, one statistic, resampled

A confidence interval describes variation in a statistic across hypothetical repetitions of a study. The study cannot usually be repeated many times, so the bootstrap treats the observed cohort as an empirical stand-in and simulates repetitions by sampling from it with replacement. Each replicate has the same number of observations as the original. Some subjects appear more than once and some are omitted; the replicate is a reweighting of the observed empirical distribution, not a new cohort.

For a four-subject toy cohort with values A = 10, B = 12, C = 18, and D = 20, the observed mean is 15. One possible resample is A, A, C, D, whose mean is 14.5. Another is B, C, C, D, whose mean is 17.0. A third is A, B, C, D, which reproduces the original mean. A bootstrap replicate is the draw plus a fresh calculation of the statistic. Repeating the operation produces an empirical sampling distribution from which a percentile interval can be read.

The cohort and interval are synthetic illustrations. A percentile interval is not exact or assumption-free, and the replicate count, seed, percentile rule, and validity policy are protocol choices rather than universal recommendations. A fixed seed improves replay only when the generator, software, code, data, and processing order are also fixed.

The pair is the sampling unit

Test-retest data contain one pair per subject: (scan 1 value, scan 2 value). The bootstrap must draw subjects and carry both members of each pair together. Drawing scan-1 values and scan-2 values independently creates pairs that never existed. The resulting statistic answers a different question, because it has destroyed the within-subject relationship that agreement analysis is meant to measure.

The same rule applies beyond the bootstrap. Patient-level splitting is required when baseline and follow-up observations belong to the same analysis unit, and any fitted preprocessing or selection must respect that unit. The resampling unit is a consequence of the estimand and dependence structure, not a convenient argument supplied to a library function.

Invalid replicates remain visible

A resample can fail the declared estimator’s finite-data or variation requirements. For CCC, for example, constant vectors can make the coefficient undefined; another estimator may have different validity conditions. Such a replicate is invalid for that statistic. It should be counted and reported, not replaced by 0, 1, or the point estimate. The analysis should declare the estimator’s validity rule and disclose the number of undefined or otherwise invalid replicates. Any threshold for the fraction of valid replicates, and whether an interval is reported, is an analysis-specific policy rather than a universal CCC rule. If the point estimate itself is not reportable, the bootstrap should be marked not run rather than presented as if invalid replicates had been evaluated.

This accounting is more informative than a polished interval with no record of how many computations failed. It also prevents a validity threshold from being quietly changed after the interval has been inspected. A linked laboratory case may freeze values such as a requested replicate count, a seed, percentile endpoints, and a valid-fraction gate; those values describe that dated case, not a recommendation for every CCC analysis.

What the bootstrap cannot repair

The bootstrap estimates uncertainty under the empirical distribution represented by the observed cohort. It does not add patients, remove selection bias, or make the cohort representative of people it did not sample. A narrow interval around a biased measurement is still a narrow interval around a biased measurement. Small cohorts and influential pairs can make both the estimate and its interval sensitive to the observed sample, so the interval should be accompanied by the paired differences, their distribution, a scatter plot against identity, and a difference-versus-mean plot.

CCC also depends on the between-subject range. For a simple paired-error illustration with Y = X + 1, the narrow-range vectors X = [10, 11, 12, 13] and Y = [11, 12, 13, 14] have CCC = 0.714 under 1/N moments. The wide-range vectors X = [10, 20, 30, 40] and Y = [11, 21, 31, 41] have CCC = 0.996. The absolute offset is one unit in both examples. The coefficient changes because that same offset consumes a different share of the observed variation. A low CCC can therefore signal poor relative agreement without identifying the absolute error in feature units.

A seed is provenance, not evidence

A pseudorandom generator maps an internal state to a draw and a next state. A seed chooses an initial state, so the same seed can reproduce the same sequence when the generator and environment are unchanged. Replay provenance therefore records the generator or bit-generator, seed or state as appropriate, software version, code, data, and processing order. The seed alone is not a replay contract. With NumPy, the documented compatibility guarantee applies to the PCG64 bit-generator’s integer stream under its stated conditions, while the higher-level Generator has no general version-compatibility guarantee. Even when integer draws match, platform or library differences can affect downstream floating computation and rounding.

No ordinary seed is scientifically better than another. Pre-specifying one with the replicate count and interval rule documents the analysis plan, but it does not prevent an analyst from trying other seeds. Sensitivity analysis addresses Monte Carlo instability, while governance and an audit trail address result shopping. These are distinct from replay provenance and from the statistical meaning of the seed itself.

The bootstrap has now supplied an uncertainty distribution for a paired agreement statistic. The next section asks a harder longitudinal question: what absolute error model is needed before a difference between two timepoints can be interpreted as change?

Sources

• Efron B. Bootstrap methods: another look at the jackknife. Annals of Statistics. 1979;7(1):1-26. doi:10.1214/aos/1176344552.
• NIST/SEMATECH e-Handbook of Statistical Methods — bootstrap methods and uncertainty.
• Ye P, et al. BJR Open. 2022;4(1):20210083. doi:10.1259/bjro.20210083, PMID 36452056.
• NumPy documentation, Random Generator and PCG64 — official documentation.