External validation of a predictive model for reintubation after cardiac surgery: A retrospective, observational study

Study objective: Explore validation of a model to predict patients' risk of failing extubation, to help providers make informed, data-driven decisions regarding the optimal timing of extubation. Design: We performed temporal, geographic, and domain validations of a model for the risk of reintubation after cardiac surgery by assessing its performance on data sets from three academic medical centers, with temporal validation using data from the institution where the model was developed. Setting: Three academic medical centers in the United States. Patients: Adult patients arriving in the cardiac intensive care unit with an endotracheal tube in place after cardiac surgery. Interventions: Receiver operating characteristic (ROC) curves and concordance statistics were used as measures of discriminative ability, and calibration curves and Brier scores were used to assess the model's predictive ability. Measurements: Temporal validation was performed in 1642 patients with a reintubation rate of 4.8%, with the model demonstrating strong discrimination (optimism-corrected c-statistic 0.77) and low predictive error (Brier score 0.044) but poor model precision and recall (Optimal F1 score 0.29). Combined domain and geographic validation were performed in 2041 patients with a reintubation rate of 1.5%. The model displayed solid discriminative ability (optimism-corrected c-statistic = 0.73) and low predictive error (Brier score = 0.0149) but low precision and recall (Optimal F1 score = 0.13). Geographic validation was performed in 2489 patients with a reintubation rate of 1.6%, with the model displaying good discrimination (optimism-corrected c-statistic = 0.71) and predictive error (Brier score = 0.0152) but poor precision and recall (Optimal F1 score = 0.13). Main results: The reintubation model displayed strong discriminative ability and low predictive error within each validation cohort. Conclusions: Future work is needed to explore how to optimize models before local implementation.