A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT

Cheng William Hong; Victoria Chernyak; Jin Young Choi; Sonia Lee; Chetan Potu; Timoteo Delgado; Tanya Wolfson; Anthony Gamst; Jason Birnbaum; Rony Kampalath; Chandana Lall; James T. Lee; Joseph W. Owen; Diego A. Aguirre; Mishal Mendiratta-Lala; Matthew S. Davenport; William Masch; Alexandra Roudenko; Sara C. Lewis; Andrea Siobhan Kierans; Elizabeth M. Hecht; Mustafa R. Bashir; Giuseppe Brancatelli; Michael L. Douek; Michael A. Ohliger; An Tang; Milena Cerny; Alice Fung; Eduardo A. Costa; Michael T. Corwin; John P. McGahan; Bobby Kalb; Khaled M. Elsayes; Venkateswar R. Surabhi; Katherine Blair; Robert M. Marks; Natally Horvat; Shaun Best; Ryan Ash; Karthik Ganesan; Christopher R. Kagay; Avinash Kambadakone; Jin Wang; Irene Cruite; Bijan Bijan; Mark Goodwin; Guilherme Moura Cunha; Dorathy Tamayo-Murillo; Kathryn J. Fowler; Claude B. Sirlin

doi:10.1148/radiol.222855

A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT

Cheng William Hong, Victoria Chernyak, Jin Young Choi, Sonia Lee, Chetan Potu, Timoteo Delgado, Tanya Wolfson, Anthony Gamst, Jason Birnbaum, Rony Kampalath, Chandana Lall, James T. Lee, Joseph W. Owen, Diego A. Aguirre, Mishal Mendiratta-Lala, Matthew S. Davenport, William Masch, Alexandra Roudenko, Sara C. Lewis, Andrea Siobhan KieransElizabeth M. Hecht, Mustafa R. Bashir, Giuseppe Brancatelli, Michael L. Douek, Michael A. Ohliger, An Tang, Milena Cerny, Alice Fung, Eduardo A. Costa, Michael T. Corwin, John P. McGahan, Bobby Kalb, Khaled M. Elsayes, Venkateswar R. Surabhi, Katherine Blair, Robert M. Marks, Natally Horvat, Shaun Best, Ryan Ash, Karthik Ganesan, Christopher R. Kagay, Avinash Kambadakone, Jin Wang, Irene Cruite, Bijan Bijan, Mark Goodwin, Guilherme Moura Cunha, Dorathy Tamayo-Murillo, Kathryn J. Fowler, Claude B. Sirlin

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

Background: Various limitations have impacted research evaluating reader agreement for Liver Imaging Reporting and Data System (LI-RADS). Purpose: To assess reader agreement of LI-RADS in an international multicenter multireader setting using scrollable images. Materials and Methods: This retrospective study used deidentified clinical multiphase CT and MRI and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 to August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS version 2018 category was computed as a rescored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1, definitely benign; LR-2, probably benign; LR-3, intermediate probability of malignancy; LR-4, probably hepatocellular carcinoma [HCC]; LR-5, definitely HCC; LR-M, probably malignant but not HCC specific; and LR-TIV, tumor in vein) was computed using intraclass correlation coefficients (ICCs). Agreement was also computed for dichotomized malignancy (LR-4, LR-5, LR-M, and LR-TIV), LR-5, and LR-M. Agreement was compared between research-versus-research reads and research-versus-clinical reads. Results: The study population consisted of 484 patients (mean age, 62 years ± 10 [SD]; 156 women; 93 CT examinations, 391 MRI examinations). ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.61, 0.73), 0.63 (95% CI: 0.55, 0.70), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs 0.62, respectively; P = .03) and for dichotomized malignancy (ICC, 0.63 vs 0.53, respectively; P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion: There was moderate agreement for LI-RADS version 2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study.

Original language	English
Article number	e222855
Journal	Radiology
Volume	307
Issue number	5
DOIs	https://doi.org/10.1148/radiol.222855
State	Published - Jun 2023

Access to Document

10.1148/radiol.222855

Cite this

Hong, C. W., Chernyak, V., Choi, J. Y., Lee, S., Potu, C., Delgado, T., Wolfson, T., Gamst, A., Birnbaum, J., Kampalath, R., Lall, C., Lee, J. T., Owen, J. W., Aguirre, D. A., Mendiratta-Lala, M., Davenport, M. S., Masch, W., Roudenko, A., Lewis, S. C., ... Sirlin, C. B. (2023). A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT. Radiology, 307(5), Article e222855. https://doi.org/10.1148/radiol.222855

@article{523887b0522548dea2b5192f41666f3a,

title = "A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT",

abstract = "Background: Various limitations have impacted research evaluating reader agreement for Liver Imaging Reporting and Data System (LI-RADS). Purpose: To assess reader agreement of LI-RADS in an international multicenter multireader setting using scrollable images. Materials and Methods: This retrospective study used deidentified clinical multiphase CT and MRI and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 to August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS version 2018 category was computed as a rescored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1, definitely benign; LR-2, probably benign; LR-3, intermediate probability of malignancy; LR-4, probably hepatocellular carcinoma [HCC]; LR-5, definitely HCC; LR-M, probably malignant but not HCC specific; and LR-TIV, tumor in vein) was computed using intraclass correlation coefficients (ICCs). Agreement was also computed for dichotomized malignancy (LR-4, LR-5, LR-M, and LR-TIV), LR-5, and LR-M. Agreement was compared between research-versus-research reads and research-versus-clinical reads. Results: The study population consisted of 484 patients (mean age, 62 years ± 10 [SD]; 156 women; 93 CT examinations, 391 MRI examinations). ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.61, 0.73), 0.63 (95% CI: 0.55, 0.70), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs 0.62, respectively; P = .03) and for dichotomized malignancy (ICC, 0.63 vs 0.53, respectively; P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion: There was moderate agreement for LI-RADS version 2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study.",

author = "Hong, {Cheng William} and Victoria Chernyak and Choi, {Jin Young} and Sonia Lee and Chetan Potu and Timoteo Delgado and Tanya Wolfson and Anthony Gamst and Jason Birnbaum and Rony Kampalath and Chandana Lall and Lee, {James T.} and Owen, {Joseph W.} and Aguirre, {Diego A.} and Mishal Mendiratta-Lala and Davenport, {Matthew S.} and William Masch and Alexandra Roudenko and Lewis, {Sara C.} and Kierans, {Andrea Siobhan} and Hecht, {Elizabeth M.} and Bashir, {Mustafa R.} and Giuseppe Brancatelli and Douek, {Michael L.} and Ohliger, {Michael A.} and An Tang and Milena Cerny and Alice Fung and Costa, {Eduardo A.} and Corwin, {Michael T.} and McGahan, {John P.} and Bobby Kalb and Elsayes, {Khaled M.} and Surabhi, {Venkateswar R.} and Katherine Blair and Marks, {Robert M.} and Natally Horvat and Shaun Best and Ryan Ash and Karthik Ganesan and Kagay, {Christopher R.} and Avinash Kambadakone and Jin Wang and Irene Cruite and Bijan Bijan and Mark Goodwin and Cunha, {Guilherme Moura} and Dorathy Tamayo-Murillo and Fowler, {Kathryn J.} and Sirlin, {Claude B.}",

note = "Publisher Copyright: {\textcopyright} RSNA, 2023.",

year = "2023",

month = jun,

doi = "10.1148/radiol.222855",

language = "English",

volume = "307",

journal = "Radiology",

issn = "0033-8419",

publisher = "Radiological Society of North America Inc.",

number = "5",

}

Hong, CW, Chernyak, V, Choi, JY, Lee, S, Potu, C, Delgado, T, Wolfson, T, Gamst, A, Birnbaum, J, Kampalath, R, Lall, C, Lee, JT, Owen, JW, Aguirre, DA, Mendiratta-Lala, M, Davenport, MS, Masch, W, Roudenko, A, Lewis, SC, Kierans, AS, Hecht, EM, Bashir, MR, Brancatelli, G, Douek, ML, Ohliger, MA, Tang, A, Cerny, M, Fung, A, Costa, EA, Corwin, MT, McGahan, JP, Kalb, B, Elsayes, KM, Surabhi, VR, Blair, K, Marks, RM, Horvat, N, Best, S, Ash, R, Ganesan, K, Kagay, CR, Kambadakone, A, Wang, J, Cruite, I, Bijan, B, Goodwin, M, Cunha, GM, Tamayo-Murillo, D, Fowler, KJ & Sirlin, CB 2023, 'A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT', Radiology, vol. 307, no. 5, e222855. https://doi.org/10.1148/radiol.222855

TY - JOUR

T1 - A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT

AU - Hong, Cheng William

AU - Chernyak, Victoria

AU - Choi, Jin Young

AU - Lee, Sonia

AU - Potu, Chetan

AU - Delgado, Timoteo

AU - Wolfson, Tanya

AU - Gamst, Anthony

AU - Birnbaum, Jason

AU - Kampalath, Rony

AU - Lall, Chandana

AU - Lee, James T.

AU - Owen, Joseph W.

AU - Aguirre, Diego A.

AU - Mendiratta-Lala, Mishal

AU - Davenport, Matthew S.

AU - Masch, William

AU - Roudenko, Alexandra

AU - Lewis, Sara C.

AU - Kierans, Andrea Siobhan

AU - Hecht, Elizabeth M.

AU - Bashir, Mustafa R.

AU - Brancatelli, Giuseppe

AU - Douek, Michael L.

AU - Ohliger, Michael A.

AU - Tang, An

AU - Cerny, Milena

AU - Fung, Alice

AU - Costa, Eduardo A.

AU - Corwin, Michael T.

AU - McGahan, John P.

AU - Kalb, Bobby

AU - Elsayes, Khaled M.

AU - Surabhi, Venkateswar R.

AU - Blair, Katherine

AU - Marks, Robert M.

AU - Horvat, Natally

AU - Best, Shaun

AU - Ash, Ryan

AU - Ganesan, Karthik

AU - Kagay, Christopher R.

AU - Kambadakone, Avinash

AU - Wang, Jin

AU - Cruite, Irene

AU - Bijan, Bijan

AU - Goodwin, Mark

AU - Cunha, Guilherme Moura

AU - Tamayo-Murillo, Dorathy

AU - Fowler, Kathryn J.

AU - Sirlin, Claude B.

PY - 2023/6

Y1 - 2023/6

N2 - Background: Various limitations have impacted research evaluating reader agreement for Liver Imaging Reporting and Data System (LI-RADS). Purpose: To assess reader agreement of LI-RADS in an international multicenter multireader setting using scrollable images. Materials and Methods: This retrospective study used deidentified clinical multiphase CT and MRI and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 to August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS version 2018 category was computed as a rescored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1, definitely benign; LR-2, probably benign; LR-3, intermediate probability of malignancy; LR-4, probably hepatocellular carcinoma [HCC]; LR-5, definitely HCC; LR-M, probably malignant but not HCC specific; and LR-TIV, tumor in vein) was computed using intraclass correlation coefficients (ICCs). Agreement was also computed for dichotomized malignancy (LR-4, LR-5, LR-M, and LR-TIV), LR-5, and LR-M. Agreement was compared between research-versus-research reads and research-versus-clinical reads. Results: The study population consisted of 484 patients (mean age, 62 years ± 10 [SD]; 156 women; 93 CT examinations, 391 MRI examinations). ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.61, 0.73), 0.63 (95% CI: 0.55, 0.70), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs 0.62, respectively; P = .03) and for dichotomized malignancy (ICC, 0.63 vs 0.53, respectively; P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion: There was moderate agreement for LI-RADS version 2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study.

AB - Background: Various limitations have impacted research evaluating reader agreement for Liver Imaging Reporting and Data System (LI-RADS). Purpose: To assess reader agreement of LI-RADS in an international multicenter multireader setting using scrollable images. Materials and Methods: This retrospective study used deidentified clinical multiphase CT and MRI and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 to August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS version 2018 category was computed as a rescored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1, definitely benign; LR-2, probably benign; LR-3, intermediate probability of malignancy; LR-4, probably hepatocellular carcinoma [HCC]; LR-5, definitely HCC; LR-M, probably malignant but not HCC specific; and LR-TIV, tumor in vein) was computed using intraclass correlation coefficients (ICCs). Agreement was also computed for dichotomized malignancy (LR-4, LR-5, LR-M, and LR-TIV), LR-5, and LR-M. Agreement was compared between research-versus-research reads and research-versus-clinical reads. Results: The study population consisted of 484 patients (mean age, 62 years ± 10 [SD]; 156 women; 93 CT examinations, 391 MRI examinations). ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.61, 0.73), 0.63 (95% CI: 0.55, 0.70), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs 0.62, respectively; P = .03) and for dichotomized malignancy (ICC, 0.63 vs 0.53, respectively; P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion: There was moderate agreement for LI-RADS version 2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study.

UR - http://www.scopus.com/inward/record.url?scp=85166215981&partnerID=8YFLogxK

U2 - 10.1148/radiol.222855

DO - 10.1148/radiol.222855

M3 - Article

C2 - 37489994

AN - SCOPUS:85166215981

SN - 0033-8419

VL - 307

JO - Radiology

JF - Radiology

IS - 5

M1 - e222855

ER -

A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT

Abstract

Access to Document

Fingerprint

Cite this