Anđela Šoškić, Suzy J Styles, Emily S Kappenman, Vanja Ković
{"title":"Garden of forking paths in ERP research - Effects of varying pre-processing and analysis steps in an N400 experiment.","authors":"Anđela Šoškić, Suzy J Styles, Emily S Kappenman, Vanja Ković","doi":"10.1111/psyp.14628","DOIUrl":null,"url":null,"abstract":"<p><p>This study tackles the Garden of Forking Paths, as a challenge for replicability and reproducibility of ERP studies. Here, we applied a multiverse analysis to a sample ERP N400 dataset, donated by an independent research team. We analyzed this dataset using 14 pipelines selected to showcase the full range of methodological variability found in the N400 literature using systematic review approach. The selected pipelines were compared in depth by looking into statistical test outcomes, descriptive statistics, effect size, data quality, and statistical power. In this way we provide a worked example of how analytic flexibility can impact results in research fields with high dimensionality such as ERP, when analyzed using standard null-hypothesis significance testing. Out of the methodological decisions that were varied, high-pass filter cut-off, artifact removal method, baseline duration, reference, measurement latency and locations, and amplitude measure (peak vs. mean) were all shown to affect at least some of the study outcome measures. Low-pass filtering was the only step which did not notably influence any of these measures. This study shows that even some of the seemingly minor procedural deviations can influence the conclusions of an ERP study. We demonstrate the power of multiverse analysis in both identifying the most reliable effects in a given study, and for providing insights into consequences of methodological decisions.</p>","PeriodicalId":20913,"journal":{"name":"Psychophysiology","volume":null,"pages":null},"PeriodicalIF":2.9000,"publicationDate":"2024-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Psychophysiology","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1111/psyp.14628","RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"NEUROSCIENCES","Score":null,"Total":0}
引用次数: 0
Abstract
This study tackles the Garden of Forking Paths, as a challenge for replicability and reproducibility of ERP studies. Here, we applied a multiverse analysis to a sample ERP N400 dataset, donated by an independent research team. We analyzed this dataset using 14 pipelines selected to showcase the full range of methodological variability found in the N400 literature using systematic review approach. The selected pipelines were compared in depth by looking into statistical test outcomes, descriptive statistics, effect size, data quality, and statistical power. In this way we provide a worked example of how analytic flexibility can impact results in research fields with high dimensionality such as ERP, when analyzed using standard null-hypothesis significance testing. Out of the methodological decisions that were varied, high-pass filter cut-off, artifact removal method, baseline duration, reference, measurement latency and locations, and amplitude measure (peak vs. mean) were all shown to affect at least some of the study outcome measures. Low-pass filtering was the only step which did not notably influence any of these measures. This study shows that even some of the seemingly minor procedural deviations can influence the conclusions of an ERP study. We demonstrate the power of multiverse analysis in both identifying the most reliable effects in a given study, and for providing insights into consequences of methodological decisions.
期刊介绍:
Founded in 1964, Psychophysiology is the most established journal in the world specifically dedicated to the dissemination of psychophysiological science. The journal continues to play a key role in advancing human neuroscience in its many forms and methodologies (including central and peripheral measures), covering research on the interrelationships between the physiological and psychological aspects of brain and behavior. Typically, studies published in Psychophysiology include psychological independent variables and noninvasive physiological dependent variables (hemodynamic, optical, and electromagnetic brain imaging and/or peripheral measures such as respiratory sinus arrhythmia, electromyography, pupillography, and many others). The majority of studies published in the journal involve human participants, but work using animal models of such phenomena is occasionally published. Psychophysiology welcomes submissions on new theoretical, empirical, and methodological advances in: cognitive, affective, clinical and social neuroscience, psychopathology and psychiatry, health science and behavioral medicine, and biomedical engineering. The journal publishes theoretical papers, evaluative reviews of literature, empirical papers, and methodological papers, with submissions welcome from scientists in any fields mentioned above.