Open data through Registered Reports can accelerate cumulative knowledge

Abstract

The scientific ‘credibility revolution’ has, in many fields, ushered in fast-paced improvements to the way that research is conducted (Vazire 2018). Sparked by concerns regarding replication and reproducibility, open research practices including preprints, preregistration, Registered Reports, open materials, code, and data aim to change the research landscape by improving the robustness and credibility of findings (Pennington 2023). Peer Community In Registered Reports (PCI RR) is a new publishing platform that integrates all of these open science practices: researchers submit a Stage 1 Registered Report through a preprint server, and after undergoing peer-review and receiving in principle acceptance (IPA), this Stage 1 protocol is then preregistered. At Stage 2, researchers append their results and discussion to the approved protocol, along with open materials, code, and data and, upon acceptance, this final preprint is then ‘recommended’ to the research community (see Eder and Frings 2021). The aim of this modified review process is to mitigate biased research practices and publication processes and, in this respect, Registered Reports appear to be working (Chambers and Tzavella 2022). One benefit for authors submitting through the PCI RR publishing route is that they can chose to publish their work in any ‘PCI friendly’ journal without the need for additional peer review. Addiction Research & Theory is one such journal offering this publishing route, committing to accept Stage 2 manuscripts that have received a positive final recommendation through PCI RR that meet the journal’s scope and formatting requirements (see Pennington and Heim 2022). As Handling Editor, I am pleased to announce that ART has published its first Registered Report through this route. Karhulahti, Vahlo et al. (2022) assessed how ontologically diverse screening instruments for gaming-related health problems differ in identifying associated problem groups. In addition to championing the authors adherence to open science practices, the goal of this editorial is to document the value of open data that is promoted by the Registered Report publishing model. I believe strongly that it is important to document the early history of open science practices and researcher’s experiences as they navigate them, particularly to overcome some of the perceived barriers associated with them and to further encourage uptake (see Norris et al. 2022). Below I first highlight the research findings by Karhulahti and colleagues and the acceleration of recommended research directions that stemmed from this team’s adoption of open code and data, before outlining more generally the positive changes we are observing as a result of the scientific credibility revolution. In their Registered Report, Karhulahti et al. administered four central screening instruments (GAS7, IGDT10, GDT, and THL1) in gaming disorder measurement to a large, nationally representative sample of Finnish participants and showed that these instruments revealed different prevalence rates and considerable heterogeneity in group overlap. Based on these findings, they suggest that due to their foundational ontological diversity these instruments might measure different problems (or other constructs) to varying degrees. Their article concludes with recommendations for researchers to (a) define their construct of interest (e.g. whether they are measuring gaming disorder or gaming-related problems) and (b) seek evidence for good construct validity to ensure accurate measurement. By sharing their code, data, and materials on the Open Science Framework repository, an independent team of researchers were able to follow one of Karhulahti et al.’s proposed future directions for this research: ‘to chart further ontological differences and similarities between constructs and/or instruments’ using an item-based network model. Billieux and Fournier (2022a) conducted this exploratory model using all of the items from the four gaming disorder assessment tools in the original study to assess potential communalities among these items. This network analysis indicated very high density of connections among all items with the authors suggesting that ‘these instruments are not reliably distinct and that their content strongly overlaps, therefore measuring substantially homogeneous constructs after all’ (pp. 1). Despite the different findings between the two teams, the authors agreed that the screening of gaming disorder requires improvement and harmonization with regards to its measurement. Moreover, Billieux and Fournier highlighted the benefits of open science practices in driving cumulative science forward. Karhulahti, Adamkovi c et al. (2022) then reanalyzed their data, again using network analysis, and wrote a reply to Billieux and Fournier. As the original dataset al.so included measures from non-gaming constructs, Karhulahti et al. decided to further test whether network overlap might also occur with other constructs – namely anxiety, depression, and bullying – that are ontologically distinct from gaming disorder. Given that these constructs do not share conceptual origins, Karhulahti et al. theorized that there should (following Billieux and Fournier’s argument) be little overlap between the items. However, their results suggested that there was indeed notable overlap between these constructs. In a parallel analysis, they also investigated whether a singlefactor or four-factor structure was supported by this model, with the findings revealing that the optimal solution has