Worked paper dissection: Tran et al.

This is an illustrative, dated case built around the paper identified by Tran et al. (PMID 35726103). The point is not to rank the paper by its largest number. It is to keep the paper’s reported design, the interpretation of each result, and the limits of the resulting claim in separate registers. The numerical and Methods/Results details below are stated as the paper’s reported details for appraisal; they are not universal benchmarks or scientific support supplied by this authoring pass.

The paper and its question

The paper studies deep-learning detection of ACL tears on knee MRI and reports testing on external datasets. Its task can be described in two ways at once: it is a medical-imaging AI classifier, and it is presented as a diagnostic prediction model. That combination matters when choosing a reading lens. A reader would consider the imaging-AI reporting lens called CLAIM 2024 together with TRIPOD+AI for clinical prediction-model reporting, and PROBAST+AI for prediction-model risk of bias. CLEAR is a radiomics reporting lens and is not the primary fit for this classifier. The current public versions and exact scope of those resources still need to be checked when they are used as evidence.

The study as reported

The development cohort is reported as 19,765 knee MRI studies from 17,738 patients at 12 centres. The source describes multiple vendors, Philips, GE, and Siemens, and field strengths of 1, 1.5, and 3 T; it reports an ACL-tear prevalence of 15 percent. The development data were divided 70/20/10 into training, validation, and test partitions, with patients kept disjoint across the partitions. The input is the MRI examination, and the reported analysis unit is the examination rather than an independent slice count.

The reference process is also central. The paper reports labels derived from radiology reports through an NLP model, with an NLP AUC of 0.984, rather than labels established by arthroscopy. The report is itself the product of a radiologist interpreting the MRI. That does not make a report-reproduction target illegitimate, but it changes what a disease-detection claim means: the model may learn cues associated with the reporting process as well as cues associated with the tear. The source identifies report inaccuracy and possible verification limitations as concerns. A reader should carry those concerns with the label rather than treating the label as an independent truth by default.

For the internal test, the paper reports AUC 0.939. At a Youden-optimal threshold of 0.38, it reports sensitivity of 87 percent, specificity of 91 percent, and accuracy of 0.902. Those threshold-dependent quantities are not the same evidence as the threshold-independent AUC, and each depends on the denominator and threshold procedure described in the source.

The paper identifies two external datasets: Bien/MRNet from Stanford in the United States, described with GE imaging, and Stajduhar/KneeMRI from Croatia, described with Siemens 1.5 T imaging. The source presents them as independent external datasets. Their cohort construction and relationship to the development data are part of the external-testing question, not details that can be inferred from the dataset names alone.

What the external numbers mean

The paper reports one set of results for the original model tested as-is. In that frozen analysis, the Bien dataset has AUC 0.941 with an interval of 0.897-0.978, while the Stajduhar dataset has AUC 0.860 with an interval of 0.829-0.892. The contrast is useful because it prevents “external” from being read as a guarantee of identical performance. It does not, by itself, prove which vendor, prevalence, protocol, or cohort difference caused the contrast.

The paper also reports retraining or fine-tuning using each external dataset’s own training and validation split. After that adaptation, it reports AUC 0.962 with an interval of 0.930-0.988 for Bien and AUC 0.922 with an interval of 0.875-0.962 for Stajduhar. The reported thresholds are 0.37 for Bien and 0.60 for Stajduhar, rather than the development threshold of 0.38.

The interpretive distinction is the important result of the dissection. If the methods establish the frozen conditions described above, 0.941 and 0.860 answer the question “how did the original model travel to these tested cohorts?” The larger or more favourable values, 0.962 and 0.922, answer a different question: how did the method perform after seeing and adapting to the new domains? The thresholds also need to be tracked as part of the procedure. A threshold re-estimated for a new-domain development partition is an adapted operating point; a threshold selected on the final test partition is test-informed and cannot support a claim about an untouched pre-specified operating point.

Thus the phrase “multi-continental external validation” is not precise enough to carry all the results. The claim must say whether the weights and threshold were frozen, which external data were used for adaptation, and which partition produced the reported metric. Even a confirmed frozen result applies to the tested conditions, not automatically to every hospital, scanner, population, or future time period.

What the design does not establish

The source material does not provide a radiologist-alone versus radiologist-plus-AI comparison, a calibration analysis, or a clinical utility or outcome analysis. It also does not use surgical correlation as the label reference and does not separate partial from full tears in the reported limitation set. Those omissions matter because a classifier’s discrimination against a report-derived label is not evidence that clinicians make better decisions with it, that its probabilities are calibrated, or that it improves patient outcomes.

The case also does not establish a causal effect. A retrospective model can predict an observed label without showing that the image feature caused the tear, that changing the feature would change the outcome, or that deploying the model would improve care. The missing evidence narrows the claim; it does not turn the paper into a failed study. The study can still inform the narrower questions it actually tested, provided the report and the evaluation procedure are described accurately.

A stronger study is a set of explicit comparisons

The same reconstruction suggests several design improvements. An independent reference standard on a defined subset, such as an appropriate surgical or adjudicated expert reference for the clinical question, could test how much the report-derived label changes the target. Frozen external testing with a frozen threshold should be reported alongside any retrained or fine-tuned results so that raw travel is visible. Calibration and decision-curve or net-benefit analyses could address probability meaning and decisions at relevant thresholds. A reader study comparing the intended reader conditions could address a claim about clinical use. Partial and full tears could be separated if that distinction is part of the clinical question, with the subgroup rule specified before the result is selected.

These are proposed ways to strengthen particular claims, not a universal protocol. The right additions depend on the estimand, reference standard, workflow, and intended action. Their common feature is that each adds evidence for a question that AUC alone cannot answer.

The transferable reading rule

For any paper that says “external validation,” write down the target population, observation unit, label source, split unit, fitted state, threshold procedure, metric, and intended claim. Then separate two sentences: “the frozen model travelled to the tested domain” and “the method transferred after adaptation to the tested domain.” If the paper cannot support one of those sentences, do not let the other stand in for it. Inspect thresholds as carefully as weights, and inspect calibration and decision evidence before turning discrimination into a clinical-use claim.

This chapter has followed a result backwards from its claim to its cohort, label, estimand, information boundary, test procedure, metric, and reporting limits. The next chapter follows the evidence into the operational workflow, where a validated study result still has to become a traceable imaging object and a visible clinical action: Chapter 10 - Imaging Informatics.