Generative Artificial Intelligence Summaries to Facilitate Emergency Department Handoff

Background Emergency department (ED) handoff to inpatient teams is a potential source of error. Generative artificial intelligence (AI) has shown promise in succinctly summarizing large quantities of clinical data and may help improve ED handoff. Objectives Our objectives were to: (1) evaluate the accuracy, clinical utility, and safety of AI-generated ED-to-inpatient handoff summaries; (2) identify patient and visit characteristics influencing summary effectiveness; and (3) characterize potential error patterns to inform implementation strategies. Methods This exploratory study evaluated AI-generated handoff summaries at an urban academic ED (February-April 2024). A Health Insurance Portability and Accountability Act-compliant GPT-4 model generated summaries aligned with the IPASS framework; ED providers assessed summary accuracy, usefulness, and safety through on-shift surveys. Results Among 50 cases, median quality and usefulness scores were 4/5 (standard error = 0.13). Safety concerns arose in 6% of cases, with issues including data omissions and mischaracterizations. Consultation status significantly affected usefulness scores (p < 0.05). Omissions of relevant medications, laboratory results, and other essential details were noted (n = 6), and emergency medicine clinicians disagreed with some AI characterizations of patient stability, vitals, and workup (n = 8). The most common response was positive impressions of the technology incorporated into the handoff process (n = 11). Conclusion This exploratory provider-in-the-loop model demonstrated clinical acceptability and highlighted areas for refinement. Future studies should incorporate recipient perspectives and examine clinical outcomes to scale and optimize AI implementation.