Functional evaluation of out-of-the-box text-mining tools for data-mining tasks

Kenneth Jung, Paea LePendu, Srinivasan Iyer, Anna Bauer-Mehren, Bethany Percha, Nigam H. Shah

Research output: Contribution to journalArticlepeer-review

32 Scopus citations

Abstract

Objective: The trade-off between the speed and simplicity of dictionary-based term recognition and the richer linguistic information provided by more advanced natural language processing (NLP) is an area of active discussion in clinical informatics. In this paper, we quantify this trade-off among text processing systems that make different trade-offs between speed and linguistic understanding. We tested both types of systems in three clinical research tasks: phase IV safety profiling of a drug, learning adverse drug-drug interactions, and learning used-to-treat relationships between drugs and indications. Materials: We first benchmarked the accuracy of the NCBO Annotator and REVEAL in a manually annotated, publically available dataset from the 2008 i2b2 Obesity Challenge. We then applied the NCBO Annotator and REVEAL to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) and used the resulting data for three research tasks. Results: There is no significant difference between using the NCBO Annotator and REVEAL in the results of the three research tasks when using large datasets. In one subtask, REVEAL achieved higher sensitivity with smaller datasets. Conclusions: For a variety of tasks, employing simple term recognition methods instead of advanced NLP methods results in little or no impact on accuracy when using large datasets. Simpler dictionary-based methods have the advantage of scaling well to very large datasets. Promoting the use of simple, dictionary-based methods for population level analyses can advance adoption of NLP in practice.

Original languageEnglish
Pages (from-to)121-131
Number of pages11
JournalJournal of the American Medical Informatics Association : JAMIA
Volume22
Issue number1
DOIs
StatePublished - 2015
Externally publishedYes

Keywords

  • Electronic health records
  • Natural language processing
  • Text mining

Fingerprint

Dive into the research topics of 'Functional evaluation of out-of-the-box text-mining tools for data-mining tasks'. Together they form a unique fingerprint.

Cite this