Replicating Current Procedural Terminology code assignment of rhinology operative notes using machine learning

Objectives: Documentation and billing are important and time-consuming parts of an otolaryngologist's work. Given advancements in machine learning (ML), we evaluated the ability of ML algorithms to use operative notes to classify rhinology procedures by Current Procedural Terminology (CPT®) code. We aimed to assess the potential for ML to replicate rhinologists' completion of their administrative tasks. Study Design: Retrospective cohort study. Setting: Urban tertiary hospital. Methods: A total of 594 operative notes from rhinological procedures across six CPT codes performed from 3/2017 to 4/2022 were collected from 22 otolaryngologists. Text was preprocessed and then vectorized using CountVectorizer (CV), term frequency-inverse document frequency, and Word2Vec. The Decision Tree, Support Vector Machine, Logistic Regression and Naïve Bayes (NB) algorithms were used to train and test models on operative notes. Model-classified CPT codes were compared to codes assigned by operating surgeons. Model performance was evaluated by area under the receiver operating characteristic curve (ROC-AUC), precision, recall, and F1-score. Results: Performance varied across vectorizers and ML algorithms. Across all performance metrics, CV and NB was most overall the best combination of vectorizer and ML algorithm across CPT codes and produced the single best AUC, 0.984. Conclusions: In otolaryngology applications, the performance of basic ML algorithms varies depending on the context in which they are used. All algorithms demonstrated their ability to classify CPT codes well as well as the potential for using ML to replicate rhinologists' completion of their administrative tasks.