ChatGPT’s adherence to otolaryngology clinical practice guidelines

Idit Tessler, Amit Wolfovitz, Eran E. Alon, Nir A. Gecel, Nir Livneh, Eyal Zimlichman, Eyal Klang

Research output: Contribution to journalArticlepeer-review

Abstract

Objectives: Large language models, including ChatGPT, has the potential to transform the way we approach medical knowledge, yet accuracy in clinical topics is critical. Here we assessed ChatGPT’s performance in adhering to the American Academy of Otolaryngology-Head and Neck Surgery guidelines. Methods: We presented ChatGPT with 24 clinical otolaryngology questions based on the guidelines of the American Academy of Otolaryngology. This was done three times (N = 72) to test the model’s consistency. Two otolaryngologists evaluated the responses for accuracy and relevance to the guidelines. Cohen’s Kappa was used to measure evaluator agreement, and Cronbach’s alpha assessed the consistency of ChatGPT’s responses. Results: The study revealed mixed results; 59.7% (43/72) of ChatGPT’s responses were highly accurate, while only 2.8% (2/72) directly contradicted the guidelines. The model showed 100% accuracy in Head and Neck, but lower accuracy in Rhinology and Otology/Neurotology (66%), Laryngology (50%), and Pediatrics (8%). The model’s responses were consistent in 17/24 (70.8%), with a Cronbach’s alpha value of 0.87, indicating a reasonable consistency across tests. Conclusions: Using a guideline-based set of structured questions, ChatGPT demonstrates consistency but variable accuracy in otolaryngology. Its lower performance in some areas, especially Pediatrics, suggests that further rigorous evaluation is needed before considering real-world clinical use.

Original languageEnglish
Pages (from-to)3829-3834
Number of pages6
JournalEuropean Archives of Oto-Rhino-Laryngology
Volume281
Issue number7
DOIs
StatePublished - Jul 2024

Keywords

  • Artificial intelligence
  • ChatGPT
  • Clinical practice guidelines
  • Medical knowledge
  • Otolaryngology

Fingerprint

Dive into the research topics of 'ChatGPT’s adherence to otolaryngology clinical practice guidelines'. Together they form a unique fingerprint.

Cite this