The reliability of replications: A study in computational reproductions

Export/share

Breznau, Nate; Mark Rinke, Eike; Wuttke, Alexander; Adem, Muna; Adriaans, Jule; Akdeniz, Esra; Alvarez-Benjumea, Amalia; Andersen, Henrik K.; Auer, Daniel; Azevedo, Flavio; Bahnsen, Oke; Bai, Ling; Balzer, Dave; Bauer, Paul C.; Bauer, Gerrit; Baumann, Markus; Baute, Sharon; Benoit, Verena; Bernauer, Julian; Berning, Carl; Berthold, Anna; Bethke, Felix S.; Biegert, Thomas; Blinzler, Katharina; Blumenberg, Johannes N.; Bobzien, Licia; Bohman, Andrea; Bol, Thijs; Bostic, Amie; Brzozowska, Zuzanna; Burgdorf, Katharina; Burger, Kaspar; Busch, Kathrin; Castillo, Juan-Carlos; Chan, Nathan; Christmann, Pablo; Connelly, Roxanne; Czymara, Christian S.; Damian, Elena; de Rooij, Eline A.; Ecker, Alejandro; Edelmann, Achim; Eder, Christina; Eger, Maureen A.; Ellerbrock, Simon; Forke, Anna; Forster, Andrea; Freire, Danilo; Gaasendam, Chris; Gavras, Konstantin; Gayle, Vernon; Gessler, Theresa; Gnambs, Timo; Godefroidt, Amélie; Grömping, Max; Groß, Martin; Gruber, Stefan; Gummer, Tobias; Hadjar, Andreas; Halbherr, Verena; Paul Heisig, Jan; Hellmeier, Sebastian; Heyne, Stefanie; Hirsch, Magdalena; Hjerm, Mikael; Hochman, Oshrat; Höffler, Jan H.; Hövermann, Andreas; Hunger, Sophia; Hunkler, Christian; Huth-Stöckle, Nora; S. Ignácz, Zsófia; Israel, Sabine; Jacobs, Laura; Jacobsen, Jannes; Jaeger, Bastian; Jungkunz, Sebastian; Jungmann, Nils; Kanjana, Jennifer; Kauff, Mathias; Khan, Salman; Khatua, Sayak; Kleinert, Manuel; Klinger, Julia; Kolb, Jan-Philipp; Kołczyńska, Marta; Kuk, John; Kunißen, Katharina; Kurti Sinatra, Dafina; Langenkamp, Alexander; Lee, Robin C.; Lersch, Philipp M.; Liu, David; Löbel, Lea-Maria; Lutscher, Philipp; Mader, Matthias; Madia, Joan E.; Malancu, Natalia; Maldonado, Luis; Marahrens, Helge; Martin, Nicole; Martinez, Paul; Mayerl, Jochen; Mayorga, Oscar J.; McDonnell, Robert; McManus, Patricia; McWagner, Kyle; Meeusen, Cecil; Meierrieks, Daniel; Mellon, Jonathan; Merhout, Friedolin; Merk, Samuel; Meyer, Daniel; Micheli, Leticia; Mijs, Jonathan; Moya, Cristóbal; Neunhoeffer, Marcel; Nüst, Daniel; Nygård, Olav; Ochsenfeld, Fabian; Otte, Gunnar; Pechenkina, Anna; Pickup, Mark; Prosser, Christopher; Raes, Louis; Ralston, Kevin; Ramos, Miguel; Reichert, Frank; Roets, Arne; Rogers, Jonathan; Ropers, Guido; Samuel, Robin; Sand, Gregor; Sanhueza Petrarca, Constanza; Schachter, Ariela; Schaeffer, Merlin; Schieferdecker, David; Schlueter, Elmar; Schmidt, Katja; Schmidt, Regine; Schmidt-Catran, Alexander; Schmiedeberg, Claudia; Schneider, Jürgen; Schoonvelde, Martijn; Schulte-Cloos, Julia; Schumann, Sandy; Schunck, Reinhard; Seuring, Julian; Silber, Henning; Sleegers, Willem; Sonntag, Nico; Staudt, Alexander; Steiber, NadiaORCID: https://orcid.org/0000-0002-9425-8840; Steiner, Nils D.; Sternberg, Sebastian; Stiers, Dieter; Stojmenovska, Dragana; Storz, Nora; Striessnig, ErichORCID: https://orcid.org/0000-0001-5419-9498; Stroppe, Anne-Kathrin; Suchow, Jordan W.; Teltemann, Janna; Tibajev, Andrey; Tung, Brian; Vagni, Giacomo; Van Assche, Jasper; van der Linden, Meta; van der Noll, Jolanda; Van Hootegem, Arno; Vogtenhuber, StefanORCID: https://orcid.org/0000-0003-0839-4481; Voicu, Bogdan; Wagemans, Fieke; Wehl, Nadja; Werner, Hannah; Wiernik, Brenton M.; Winter, Fabian; Wolf, Christof; Wu, Cary; Yamada, Yuki; Zakula, Björn; Zhang, Nan; Ziller, Conrad; Zins, Stefan; Żółtak, Tomasz and Nguyen, Hung H. V. (2025) The reliability of replications: A study in computational reproductions. Royal Society Open Science, 12 (3), 241038. https://doi.org/10.1098/rsos.241038

[thumbnail of breznau-rinke-et-al-2025-the-reliability-of-replications-a-study-in-computational-reproductions.pdf]

Preview

Text
breznau-rinke-et-al-2025-the-reliability-of-replications-a-study-in-computational-reproductions.pdf - Published Version
Available under License Creative Commons Attribution.
Download (1MB) | Preview

Official URL: https://royalsocietypublishing.org/doi/10.1098/rso...

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Item Type:	Article in Academic Journal
Additional Information (public):	© 2025 The Author(s).
Keywords:	reliability, replications, computational reproduction, social and behavioural sciences
Funders:	German Science Foundation – Deutsche Forschungsgemeinschaft (DFG) – Project Number 464546557, Chilean National Research and Development Agency – Agencia Nacional de Investigación y Desarrollo – ANID/FONDAP/1523A0005
Research Units:	Educational Structures and Educational Opportunities
Related URLs:	https://github.com/nbreznau/how_many_rep... Supplemental Material
Date Deposited:	28 Mar 2025 12:34
Last Modified:	28 Mar 2025 12:34
DOI:	10.1098/rsos.241038
ISSN:	2054-5703
URI:	https://irihs.ihs.ac.at/id/eprint/7168

Actions (login required)

: View Item

Altmetric

CORE (COnnecting REpositories)