TY - JOUR
T1 - Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools
AU - Li, Jimmy
AU - Kabouji, Joudy
AU - Bouhadoun, Sarah
AU - Tanveer, Sarah
AU - Filion, Kristian B.
AU - Gore, Genevieve
AU - Josephson, Colin Bruce
AU - Kwon, Churl Su
AU - Jette, Nathalie
AU - Bauer, Prisca Rachel
AU - Day, Gregory S.
AU - Subota, Ann
AU - Roberts, Jodie I.
AU - Lukmanji, Sara
AU - Sauro, Khara
AU - Ismaili, Adnane Alaoui
AU - Rahmani, Feriel
AU - Chelabi, Khadidja
AU - Kerdougli, Yasmine
AU - Seulami, Nour Meryem
AU - Soumana, Aminata
AU - Khalil, Sarah
AU - Maynard, Noémie
AU - Keezer, Mark Robert
N1 - Publisher Copyright:
© 2023 Elsevier Inc.
PY - 2023/10
Y1 - 2023/10
N2 - Objectives: To evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs). Study Design and Setting: Twenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard. Results: The mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved. Conclusion: TM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.
AB - Objectives: To evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs). Study Design and Setting: Twenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard. Results: The mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved. Conclusion: TM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.
KW - Abstrackr
KW - Artificial intelligence
KW - Diagnostic study
KW - Knowledge synthesis
KW - Machine learning
KW - Rayyan
KW - SWIFT-Review
KW - Sensitivity
KW - Specificity
UR - http://www.scopus.com/inward/record.url?scp=85170637694&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2023.07.010
DO - 10.1016/j.jclinepi.2023.07.010
M3 - Article
C2 - 37506951
AN - SCOPUS:85170637694
SN - 0895-4356
VL - 162
SP - 72
EP - 80
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -