Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth"

Samuel G. Armato; Rachael Y. Roberts; Masha Kocherginsky; Denise R. Aberle; Ella A. Kazerooni; Heber MacMahon; Edwin J.R. van Beek; David Yankelevitz; Geoffrey McLennan; Michael F. McNitt-Gray; Charles R. Meyer; Anthony P. Reeves; Philip Caligiuri; Leslie E. Quint; Baskaran Sundaram; Barbara Y. Croft; Laurence P. Clarke

doi:10.1016/j.acra.2008.05.022

Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth"

Samuel G. Armato, Rachael Y. Roberts, Masha Kocherginsky, Denise R. Aberle, Ella A. Kazerooni, Heber MacMahon, Edwin J.R. van Beek, David Yankelevitz, Geoffrey McLennan, Michael F. McNitt-Gray, Charles R. Meyer, Anthony P. Reeves, Philip Caligiuri, Leslie E. Quint, Baskaran Sundaram, Barbara Y. Croft, Laurence P. Clarke

Research output: Contribution to journal › Article › peer-review

66 Scopus citations

Abstract

Rationale and Objectives: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. Materials and Methods: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. Results: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. Conclusions: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.

Original language	English
Pages (from-to)	28-38
Number of pages	11
Journal	Academic Radiology
Volume	16
Issue number	1
DOIs	https://doi.org/10.1016/j.acra.2008.05.022
State	Published - Jan 2009
Externally published	Yes

Keywords

Lung nodule
computed tomography (CT)
computer-aided diagnosis (CAD)
interobserver variability
thoracic imaging

Access to Document

10.1016/j.acra.2008.05.022

Cite this

Armato, S. G., Roberts, R. Y., Kocherginsky, M., Aberle, D. R., Kazerooni, E. A., MacMahon, H., van Beek, E. J. R., Yankelevitz, D., McLennan, G., McNitt-Gray, M. F., Meyer, C. R., Reeves, A. P., Caligiuri, P., Quint, L. E., Sundaram, B., Croft, B. Y., & Clarke, L. P. (2009). Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth". Academic Radiology, 16(1), 28-38. https://doi.org/10.1016/j.acra.2008.05.022

@article{e167f1e87199442789438b90c21a91c1,

title = "Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of {"}Truth{"}",

abstract = "Rationale and Objectives: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the {"}truth{"}). The purpose of this study was to analyze (1) variability in the {"}truth{"} defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of {"}truth{"} in the context of lung nodule detection in computed tomographic (CT) scans. Materials and Methods: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel {"}truth{"} sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel {"}truth{"} sets. Results: The number of {"}true{"} nodules in the different panel {"}truth{"} sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel {"}truth{"} sets for different panel {"}truth{"} conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. Conclusions: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of {"}truth{"} on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the {"}truth{"} established by other experienced thoracic radiologists.",

keywords = "Lung nodule, computed tomography (CT), computer-aided diagnosis (CAD), interobserver variability, thoracic imaging",

author = "Armato, {Samuel G.} and Roberts, {Rachael Y.} and Masha Kocherginsky and Aberle, {Denise R.} and Kazerooni, {Ella A.} and Heber MacMahon and {van Beek}, {Edwin J.R.} and David Yankelevitz and Geoffrey McLennan and McNitt-Gray, {Michael F.} and Meyer, {Charles R.} and Reeves, {Anthony P.} and Philip Caligiuri and Quint, {Leslie E.} and Baskaran Sundaram and Croft, {Barbara Y.} and Clarke, {Laurence P.}",

year = "2009",

month = jan,

doi = "10.1016/j.acra.2008.05.022",

language = "English",

volume = "16",

pages = "28--38",

journal = "Academic Radiology",

issn = "1076-6332",

publisher = "Elsevier USA",

number = "1",

}

Armato, SG, Roberts, RY, Kocherginsky, M, Aberle, DR, Kazerooni, EA, MacMahon, H, van Beek, EJR, Yankelevitz, D, McLennan, G, McNitt-Gray, MF, Meyer, CR, Reeves, AP, Caligiuri, P, Quint, LE, Sundaram, B, Croft, BY & Clarke, LP 2009, 'Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth"', Academic Radiology, vol. 16, no. 1, pp. 28-38. https://doi.org/10.1016/j.acra.2008.05.022

TY - JOUR

T1 - Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth"

AU - Armato, Samuel G.

AU - Roberts, Rachael Y.

AU - Kocherginsky, Masha

AU - Aberle, Denise R.

AU - Kazerooni, Ella A.

AU - MacMahon, Heber

AU - van Beek, Edwin J.R.

AU - Yankelevitz, David

AU - McLennan, Geoffrey

AU - McNitt-Gray, Michael F.

AU - Meyer, Charles R.

AU - Reeves, Anthony P.

AU - Caligiuri, Philip

AU - Quint, Leslie E.

AU - Sundaram, Baskaran

AU - Croft, Barbara Y.

AU - Clarke, Laurence P.

PY - 2009/1

Y1 - 2009/1

N2 - Rationale and Objectives: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. Materials and Methods: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. Results: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. Conclusions: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.

AB - Rationale and Objectives: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. Materials and Methods: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. Results: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. Conclusions: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.

KW - Lung nodule

KW - computed tomography (CT)

KW - computer-aided diagnosis (CAD)

KW - interobserver variability

KW - thoracic imaging

UR - http://www.scopus.com/inward/record.url?scp=57049184118&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2008.05.022

DO - 10.1016/j.acra.2008.05.022

M3 - Article

C2 - 19064209

AN - SCOPUS:57049184118

SN - 1076-6332

VL - 16

SP - 28

EP - 38

JO - Academic Radiology

JF - Academic Radiology

IS - 1

ER -

Assessment of Radiologist Performance in the Detection of Lung Nodules. Dependence on the Definition of "Truth"

Abstract

Keywords

Access to Document

Fingerprint

Cite this