TY - JOUR
T1 - Reproducibility in the absence of selective reporting
T2 - An illustration from large-scale brain asymmetry research
AU - ENIGMA Laterality Working Group
AU - Kong, Xiang Zhen
AU - Francks, Clyde
AU - Mathias, Samuel R.
AU - Guadalupe, Tulio
AU - Abé, Christoph
AU - Agartz, Ingrid
AU - Akudjedu, Theophilus N.
AU - Aleman, Andre
AU - Alhusaini, Saud
AU - Allen, Nicholas B.
AU - Ames, David
AU - Andreassen, Ole A.
AU - Vasquez, Alejandro Arias
AU - Armstrong, Nicola J.
AU - Asherson, Phil
AU - Bergo, Felipe
AU - Bastin, Mark E.
AU - Batalla, Albert
AU - Bauer, Jochen
AU - Baune, Bernhard T.
AU - Baur-Streubel, Ramona
AU - Biederman, Joseph
AU - Blaine, Sara K.
AU - Boedhoe, Premika
AU - Bøen, Erlend
AU - Bose, Anushree
AU - Bralten, Janita
AU - Brandeis, Daniel
AU - Brem, Silvia
AU - Brodaty, Henry
AU - Yüksel, Dilara
AU - Brooks, Samantha J.
AU - Buitelaar, Jan
AU - Bürger, Christian
AU - Bülow, Robin
AU - Calhoun, Vince
AU - Calvo, Anna
AU - Canales-Rodríguez, Erick Jorge
AU - Cannon, Dara M.
AU - Caparelli, Elisabeth C.
AU - Castellanos, Francisco X.
AU - Cendes, Fernando
AU - Chaim-Avancini, Tiffany Moukbel
AU - Chantiluke, Kaylita
AU - Chen, Qun lin
AU - Chen, Xiayu
AU - Cheng, Yuqi
AU - Christakou, Anastasia
AU - Clark, Vincent P.
AU - Coghill, David
N1 - Publisher Copyright:
© 2020 The Authors. Human Brain Mapping published by Wiley Periodicals LLC.
PY - 2022/1
Y1 - 2022/1
N2 - The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left–right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an “ideal publishing environment,” that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.
AB - The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left–right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an “ideal publishing environment,” that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.
KW - P-hacking
KW - multisite collaboration
KW - publication bias
KW - reproducibility
KW - team science
UR - http://www.scopus.com/inward/record.url?scp=85089785260&partnerID=8YFLogxK
U2 - 10.1002/hbm.25154
DO - 10.1002/hbm.25154
M3 - Article
C2 - 32841457
AN - SCOPUS:85089785260
SN - 1065-9471
VL - 43
SP - 244
EP - 254
JO - Human Brain Mapping
JF - Human Brain Mapping
IS - 1
ER -