Inter-rater reliability of surgical reviews for AREN03B2: A COG renal tumor committee study

Purpose The Children's Oncology Group (COG) renal tumor study (AREN03B2) requires real-time central review of radiology, pathology, and the surgical procedure to determine appropriate risk-based therapy. The purpose of this study was to determine the inter-rater reliability of the surgical reviews. Methods Of the first 3200 enrolled AREN03B2 patients, a sample of 100 enriched for blood vessel involvement, spill, rupture, and lymph node involvement was selected for analysis. The surgical assessment was then performed independently by two blinded surgical reviewers and compared to the original assessment, which had been completed by another of the committee surgeons. Variables assessed included surgeon-determined local tumor stage, overall disease stage, type of renal procedure performed, presence of tumor rupture, occurrence of intraoperative tumor spill, blood vessel involvement, presence of peritoneal implants, and interpretation of residual disease. Inter-rater reliability was measured using the Fleiss' Kappa statistic two-sided hypothesis tests (Kappa, p-value). Results Local tumor stage correlated in all 3 reviews except in one case (Kappa = 0.9775, p < 0.001). Similarly, overall disease stage had excellent correlation (0.9422, p < 0.001). There was strong correlation for type of renal procedure (0.8357, p < 0.001), presence of tumor rupture (0.6858, p < 0.001), intraoperative tumor spill (0.6493, p < 0.001), and blood vessel involvement (0.6470, p < 0.001). Variables that had lower correlation were determination of the presence of peritoneal implants (0.2753, p < 0.001) and interpretation of residual disease status (0.5310, p < 0.001). Conclusion The inter-rater reliability of the surgical review is high based on the great consistency in the 3 independent review results. This analysis provides validation and establishes precedent for real-time central surgical review to determine treatment assignment in a risk-based stratagem for multimodal cancer therapy.