Critical assessment of automated flow cytometry data analysis techniques

Nima Aghaeepour; Greg Finak; Holger Hoos; Tim R. Mosmann; Ryan Brinkman; Raphael Gottardo; Richard H. Scheuermann; David Dougall; Alireza Hadj Khodabakhshi; Phillip Mah; Gerlinde Obermoser; Josef Spidlen; Ian Taylor; Sherry A. Wuensch; Jonathan Bramson; Connie Eaves; Andrew P. Weng; Edgardo S. Fortuno; Kevin Ho; Tobias R. Kollmann; Wade Rogers; Stephen De Rosa; Bakul Dalai; Ariful Azad; Alex Pothen; Aaron Brandes; Hannes Bretschneider; Robert Bruggner; Rachel Finck; Robin Jia; Noah Zimmerman; Michael Linderman; David Dill; Gary Nolan; Cliburn Chan; Faysal El Khettabi; Kieran O'Neill; Maria Chikina; Yongchao Ge; Stuart Sealfon; István Sugár; Arvind Gupta; Parisa Shooshtari; Habil Zare; Philip L. De Jager; Mike Jiang; Jens Keilwagen; Jose M. Maisog; George Luta; Andrea A. Barbo; Peter Májek; Jozef Vilček; Tapio Manninen; Heikki Huttunen; Pekka Ruusuvuori; Matti Nykter; Geoffrey J. McLachlan; Kui Wang; Iftekhar Naim; Gaurav Sharma; Radina Nikolic; Saumyadipta Pyne; Yu Qian; Peng Qiu; John Quinn; Andrew Roth; Pablo Meyer; Gustavo Stolovitzky; Julio Saez-Rodriguez; Raquel Norel; Madhuchhanda Bhattacharjee; Michael Biehl; Philipp Bucher; Kerstin Bunte; Barbara Di Camillo; Francesco Sambo; Tiziana Sanavia; Emanuele Trifoglio; Gianna Toffolo; S. D. Slavica Dimitrieva; Rene Dreos; Giovanna Ambrosini; Jan Grau; Ivo Grosse; Stefan Posch; Nicolas Guex; Miron Kursa; Witold Rudnicki; Bo Liu; Mark Maienschein-Cline; Petra Schneider; Michael Seifert; Marc Strickert; Jose M.G. Vilar

doi:10.1038/NMETH.2365

Critical assessment of automated flow cytometry data analysis techniques

Nima Aghaeepour, Greg Finak, Holger Hoos, Tim R. Mosmann, Ryan Brinkman, Raphael Gottardo, Richard H. Scheuermann, David Dougall, Alireza Hadj Khodabakhshi, Phillip Mah, Gerlinde Obermoser, Josef Spidlen, Ian Taylor, Sherry A. Wuensch, Jonathan Bramson, Connie Eaves, Andrew P. Weng, Edgardo S. Fortuno, Kevin Ho, Tobias R. KollmannWade Rogers, Stephen De Rosa, Bakul Dalai, Ariful Azad, Alex Pothen, Aaron Brandes, Hannes Bretschneider, Robert Bruggner, Rachel Finck, Robin Jia, Noah Zimmerman, Michael Linderman, David Dill, Gary Nolan, Cliburn Chan, Faysal El Khettabi, Kieran O'Neill, Maria Chikina, Yongchao Ge, Stuart Sealfon, István Sugár, Arvind Gupta, Parisa Shooshtari, Habil Zare, Philip L. De Jager, Mike Jiang, Jens Keilwagen, Jose M. Maisog, George Luta, Andrea A. Barbo, Peter Májek, Jozef Vilček, Tapio Manninen, Heikki Huttunen, Pekka Ruusuvuori, Matti Nykter, Geoffrey J. McLachlan, Kui Wang, Iftekhar Naim, Gaurav Sharma, Radina Nikolic, Saumyadipta Pyne, Yu Qian, Peng Qiu, John Quinn, Andrew Roth, Pablo Meyer, Gustavo Stolovitzky, Julio Saez-Rodriguez, Raquel Norel, Madhuchhanda Bhattacharjee, Michael Biehl, Philipp Bucher, Kerstin Bunte, Barbara Di Camillo, Francesco Sambo, Tiziana Sanavia, Emanuele Trifoglio, Gianna Toffolo, S. D. Slavica Dimitrieva, Rene Dreos, Giovanna Ambrosini, Jan Grau, Ivo Grosse, Stefan Posch, Nicolas Guex, Miron Kursa, Witold Rudnicki, Bo Liu, Mark Maienschein-Cline, Petra Schneider, Michael Seifert, Marc Strickert, Jose M.G. Vilar

Research output: Contribution to journal › Article › peer-review

462 Scopus citations

Abstract

Traditional methods for flow cytometry (FCM) data processing rely on subjective manual gating. Recently, several groups have developed computational methods for identifying cell populations in multidimensional FCM data. The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of these methods on two tasks: (i) mammalian cell population identification, to determine whether automated algorithms can reproduce expert manual gating and (ii) sample classification, to determine whether analysis pipelines can identify characteristics that correlate with external variables (such as clinical outcome). This analysis presents the results of the first FlowCAP challenges. Several methods performed well as compared to manual gating or external variables using statistical performance measures, which suggests that automated methods have reached a sufficient level of maturity and accuracy for reliable use in FCM data analysis.

Original language	English
Pages (from-to)	228-238
Number of pages	11
Journal	Nature Methods
Volume	10
Issue number	3
DOIs	https://doi.org/10.1038/NMETH.2365
State	Published - Mar 2013

Access to Document

10.1038/NMETH.2365

Cite this

Aghaeepour, N., Finak, G., Hoos, H., Mosmann, T. R., Brinkman, R., Gottardo, R., Scheuermann, R. H., Dougall, D., Khodabakhshi, A. H., Mah, P., Obermoser, G., Spidlen, J., Taylor, I., Wuensch, S. A., Bramson, J., Eaves, C., Weng, A. P., Fortuno, E. S., Ho, K., ... Vilar, J. M. G. (2013). Critical assessment of automated flow cytometry data analysis techniques. Nature Methods, 10(3), 228-238. https://doi.org/10.1038/NMETH.2365

@article{c890185c991d4cfe800d91215b5599a0,

title = "Critical assessment of automated flow cytometry data analysis techniques",

abstract = "Traditional methods for flow cytometry (FCM) data processing rely on subjective manual gating. Recently, several groups have developed computational methods for identifying cell populations in multidimensional FCM data. The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of these methods on two tasks: (i) mammalian cell population identification, to determine whether automated algorithms can reproduce expert manual gating and (ii) sample classification, to determine whether analysis pipelines can identify characteristics that correlate with external variables (such as clinical outcome). This analysis presents the results of the first FlowCAP challenges. Several methods performed well as compared to manual gating or external variables using statistical performance measures, which suggests that automated methods have reached a sufficient level of maturity and accuracy for reliable use in FCM data analysis.",

author = "Nima Aghaeepour and Greg Finak and Holger Hoos and Mosmann, {Tim R.} and Ryan Brinkman and Raphael Gottardo and Scheuermann, {Richard H.} and David Dougall and Khodabakhshi, {Alireza Hadj} and Phillip Mah and Gerlinde Obermoser and Josef Spidlen and Ian Taylor and Wuensch, {Sherry A.} and Jonathan Bramson and Connie Eaves and Weng, {Andrew P.} and Fortuno, {Edgardo S.} and Kevin Ho and Kollmann, {Tobias R.} and Wade Rogers and {De Rosa}, Stephen and Bakul Dalai and Ariful Azad and Alex Pothen and Aaron Brandes and Hannes Bretschneider and Robert Bruggner and Rachel Finck and Robin Jia and Noah Zimmerman and Michael Linderman and David Dill and Gary Nolan and Cliburn Chan and Khettabi, {Faysal El} and Kieran O'Neill and Maria Chikina and Yongchao Ge and Stuart Sealfon and Istv{\'a}n Sug{\'a}r and Arvind Gupta and Parisa Shooshtari and Habil Zare and {De Jager}, {Philip L.} and Mike Jiang and Jens Keilwagen and Maisog, {Jose M.} and George Luta and Barbo, {Andrea A.} and Peter M{\'a}jek and Jozef Vil{\v c}ek and Tapio Manninen and Heikki Huttunen and Pekka Ruusuvuori and Matti Nykter and McLachlan, {Geoffrey J.} and Kui Wang and Iftekhar Naim and Gaurav Sharma and Radina Nikolic and Saumyadipta Pyne and Yu Qian and Peng Qiu and John Quinn and Andrew Roth and Pablo Meyer and Gustavo Stolovitzky and Julio Saez-Rodriguez and Raquel Norel and Madhuchhanda Bhattacharjee and Michael Biehl and Philipp Bucher and Kerstin Bunte and {Di Camillo}, Barbara and Francesco Sambo and Tiziana Sanavia and Emanuele Trifoglio and Gianna Toffolo and {Slavica Dimitrieva}, {S. D.} and Rene Dreos and Giovanna Ambrosini and Jan Grau and Ivo Grosse and Stefan Posch and Nicolas Guex and Miron Kursa and Witold Rudnicki and Bo Liu and Mark Maienschein-Cline and Petra Schneider and Michael Seifert and Marc Strickert and Vilar, {Jose M.G.}",

note = "Funding Information: The FlowCAP summits—held on the US National Institutes of Health (NIH) campus, Bethesda, Maryland, USA in 2010 and 2011—were generously sponsored by the NIH and National Institute of Allergy and Infectious Diseases. This work was partially supported by the following: NIH/R01EB008400, NIH/N01AI40076, NIH/R01NS067305, NIH/RC2-GM093080, Canadian Cancer Society no. 700374, Natural Sciences and Engineering Research Council of Canada (H. Hoos), the Terry Fox Foundation and Terry Fox Research Institute (R. Brinkman). This work was partially supported by the following scholarships: Canadian Institute of Health/Michael Smith Foundation for Health Research (MSFHR) (N.A.), University of British Columbia{\textquoteright}s 4YF (N.A.), International Society for Advancement of Cytometry Scholar (N.A.), MSFHR Scholar (R. Brinkman), Harry Weaver Neurosciences Scholar (P.L.D.J.) and the Rachford and Carlota A. Harris Professorship (G.N.).",

year = "2013",

month = mar,

doi = "10.1038/NMETH.2365",

language = "English",

volume = "10",

pages = "228--238",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Nature Research",

number = "3",

}

Aghaeepour, N, Finak, G, Hoos, H, Mosmann, TR, Brinkman, R, Gottardo, R, Scheuermann, RH, Dougall, D, Khodabakhshi, AH, Mah, P, Obermoser, G, Spidlen, J, Taylor, I, Wuensch, SA, Bramson, J, Eaves, C, Weng, AP, Fortuno, ES, Ho, K, Kollmann, TR, Rogers, W, De Rosa, S, Dalai, B, Azad, A, Pothen, A, Brandes, A, Bretschneider, H, Bruggner, R, Finck, R, Jia, R, Zimmerman, N, Linderman, M, Dill, D, Nolan, G, Chan, C, Khettabi, FE, O'Neill, K, Chikina, M, Ge, Y , Sealfon, S , Sugár, I, Gupta, A, Shooshtari, P, Zare, H, De Jager, PL, Jiang, M, Keilwagen, J, Maisog, JM, Luta, G, Barbo, AA, Májek, P, Vilček, J, Manninen, T, Huttunen, H, Ruusuvuori, P, Nykter, M, McLachlan, GJ, Wang, K, Naim, I, Sharma, G, Nikolic, R, Pyne, S, Qian, Y, Qiu, P, Quinn, J, Roth, A, Meyer, P, Stolovitzky, G, Saez-Rodriguez, J, Norel, R, Bhattacharjee, M, Biehl, M, Bucher, P, Bunte, K, Di Camillo, B, Sambo, F, Sanavia, T, Trifoglio, E, Toffolo, G, Slavica Dimitrieva, SD, Dreos, R, Ambrosini, G, Grau, J, Grosse, I, Posch, S, Guex, N, Kursa, M, Rudnicki, W, Liu, B, Maienschein-Cline, M, Schneider, P, Seifert, M, Strickert, M & Vilar, JMG 2013, 'Critical assessment of automated flow cytometry data analysis techniques', Nature Methods, vol. 10, no. 3, pp. 228-238. https://doi.org/10.1038/NMETH.2365

TY - JOUR

T1 - Critical assessment of automated flow cytometry data analysis techniques

AU - Aghaeepour, Nima

AU - Finak, Greg

AU - Hoos, Holger

AU - Mosmann, Tim R.

AU - Brinkman, Ryan

AU - Gottardo, Raphael

AU - Scheuermann, Richard H.

AU - Dougall, David

AU - Khodabakhshi, Alireza Hadj

AU - Mah, Phillip

AU - Obermoser, Gerlinde

AU - Spidlen, Josef

AU - Taylor, Ian

AU - Wuensch, Sherry A.

AU - Bramson, Jonathan

AU - Eaves, Connie

AU - Weng, Andrew P.

AU - Fortuno, Edgardo S.

AU - Ho, Kevin

AU - Kollmann, Tobias R.

AU - Rogers, Wade

AU - De Rosa, Stephen

AU - Dalai, Bakul

AU - Azad, Ariful

AU - Pothen, Alex

AU - Brandes, Aaron

AU - Bretschneider, Hannes

AU - Bruggner, Robert

AU - Finck, Rachel

AU - Jia, Robin

AU - Zimmerman, Noah

AU - Linderman, Michael

AU - Dill, David

AU - Nolan, Gary

AU - Chan, Cliburn

AU - Khettabi, Faysal El

AU - O'Neill, Kieran

AU - Chikina, Maria

AU - Ge, Yongchao

AU - Sealfon, Stuart

AU - Sugár, István

AU - Gupta, Arvind

AU - Shooshtari, Parisa

AU - Zare, Habil

AU - De Jager, Philip L.

AU - Jiang, Mike

AU - Keilwagen, Jens

AU - Maisog, Jose M.

AU - Luta, George

AU - Barbo, Andrea A.

AU - Májek, Peter

AU - Vilček, Jozef

AU - Manninen, Tapio

AU - Huttunen, Heikki

AU - Ruusuvuori, Pekka

AU - Nykter, Matti

AU - McLachlan, Geoffrey J.

AU - Wang, Kui

AU - Naim, Iftekhar

AU - Sharma, Gaurav

AU - Nikolic, Radina

AU - Pyne, Saumyadipta

AU - Qian, Yu

AU - Qiu, Peng

AU - Quinn, John

AU - Roth, Andrew

AU - Meyer, Pablo

AU - Stolovitzky, Gustavo

AU - Saez-Rodriguez, Julio

AU - Norel, Raquel

AU - Bhattacharjee, Madhuchhanda

AU - Biehl, Michael

AU - Bucher, Philipp

AU - Bunte, Kerstin

AU - Di Camillo, Barbara

AU - Sambo, Francesco

AU - Sanavia, Tiziana

AU - Trifoglio, Emanuele

AU - Toffolo, Gianna

AU - Slavica Dimitrieva, S. D.

AU - Dreos, Rene

AU - Ambrosini, Giovanna

AU - Grau, Jan

AU - Grosse, Ivo

AU - Posch, Stefan

AU - Guex, Nicolas

AU - Kursa, Miron

AU - Rudnicki, Witold

AU - Liu, Bo

AU - Maienschein-Cline, Mark

AU - Schneider, Petra

AU - Seifert, Michael

AU - Strickert, Marc

AU - Vilar, Jose M.G.

N1 - Funding Information: The FlowCAP summits—held on the US National Institutes of Health (NIH) campus, Bethesda, Maryland, USA in 2010 and 2011—were generously sponsored by the NIH and National Institute of Allergy and Infectious Diseases. This work was partially supported by the following: NIH/R01EB008400, NIH/N01AI40076, NIH/R01NS067305, NIH/RC2-GM093080, Canadian Cancer Society no. 700374, Natural Sciences and Engineering Research Council of Canada (H. Hoos), the Terry Fox Foundation and Terry Fox Research Institute (R. Brinkman). This work was partially supported by the following scholarships: Canadian Institute of Health/Michael Smith Foundation for Health Research (MSFHR) (N.A.), University of British Columbia’s 4YF (N.A.), International Society for Advancement of Cytometry Scholar (N.A.), MSFHR Scholar (R. Brinkman), Harry Weaver Neurosciences Scholar (P.L.D.J.) and the Rachford and Carlota A. Harris Professorship (G.N.).

PY - 2013/3

Y1 - 2013/3

N2 - Traditional methods for flow cytometry (FCM) data processing rely on subjective manual gating. Recently, several groups have developed computational methods for identifying cell populations in multidimensional FCM data. The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of these methods on two tasks: (i) mammalian cell population identification, to determine whether automated algorithms can reproduce expert manual gating and (ii) sample classification, to determine whether analysis pipelines can identify characteristics that correlate with external variables (such as clinical outcome). This analysis presents the results of the first FlowCAP challenges. Several methods performed well as compared to manual gating or external variables using statistical performance measures, which suggests that automated methods have reached a sufficient level of maturity and accuracy for reliable use in FCM data analysis.

AB - Traditional methods for flow cytometry (FCM) data processing rely on subjective manual gating. Recently, several groups have developed computational methods for identifying cell populations in multidimensional FCM data. The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of these methods on two tasks: (i) mammalian cell population identification, to determine whether automated algorithms can reproduce expert manual gating and (ii) sample classification, to determine whether analysis pipelines can identify characteristics that correlate with external variables (such as clinical outcome). This analysis presents the results of the first FlowCAP challenges. Several methods performed well as compared to manual gating or external variables using statistical performance measures, which suggests that automated methods have reached a sufficient level of maturity and accuracy for reliable use in FCM data analysis.

UR - http://www.scopus.com/inward/record.url?scp=84874666550&partnerID=8YFLogxK

U2 - 10.1038/NMETH.2365

DO - 10.1038/NMETH.2365

M3 - Article

C2 - 23396282

AN - SCOPUS:84874666550

SN - 1548-7091

VL - 10

SP - 228

EP - 238

JO - Nature Methods

JF - Nature Methods

IS - 3

ER -

Critical assessment of automated flow cytometry data analysis techniques

Abstract

Access to Document

Fingerprint

Cite this