Registered reports in operations management: Lessons from an experimental trial

Field experiments involve the practice of conducting controlled interventions wherein researchers collaborate with practicing managers to study the effects of such interventions on a subset of subjects, processes or entities (Ibañez & Staats, 2019). In recent years, Operations Management (OM) as a field has seen significant interest to conducting field experiments as evidenced by studies in healthcare delivery (Anand et al., 2021; Staats et al., 2017), retail operations (Chuang et al., 2016; Craig et al., 2016) and recycling (McKie et al., 2024). While there are several benefits to conducting field experiment such as improved external validity, reduced observer bias and improved causal inference, field experiments require considerable relational investments, often require substantial time in data collection, and carry significant risks such as loss of access to participant sites through attrition. Given these points, OM researchers often shy away from field experiments as primary research method. By doing so, however, they miss an opportunity to ask and answer bold questions that can challenge existing OM theories and offer richer insights.

As an illustrative example, consider the age-old question on why operational excellence initiatives (e.g., Lean/Six Sigma, Process Management) fail to sustain themselves over time. There have been many studies exploring the factors that influence the adoption and use of operational excellence in a variety of industry contexts (e.g., Anand et al., 2021; Anderson & Chandrasekaran, 2024; Shah & Ward, 2003; Sterman et al., 2002). These studies have capitalized on several research methods including case studies, surveys, analytical models, and econometric methods. Yet, the explanations delivered from these studies leave important questions unanswered. One way to address this gap would involve the use of carefully constructed field experiments, with specific sets of interventions supporting operational excellence initiatives adopted by organizations or their units, with some controls for otherwise confounding factors, and with monitoring in place to observe their impacts over time. Unfortunately, the challenges of recruiting enough firms to secure an adequate sample size, controlling for potential spillover effects and attrition, and ensuring compliance in the experimental protocol, renders a potential research project with lead time that could run into years and with significant risk and uncertainty. Accordingly, given time pressures on faculty publishing, the paucity of such rich studies into these complex settings is far from surprising. Instead, we continue to make incremental knowledge creation through alternative research designs.

For this special issue, we were particularly motivated by the prospect of supporting authors interested in questions that required field experimental design, but who were otherwise worried about the risks in conducting them. To that end, we developed a process to encourage OM scholars to conduct experiments with interventions to advance our understanding of OM theories by reducing the risks and intrinsic cost of experiments through the pre-approval of their research designs.

For this pre-approved research design (PARD), we borrowed the two-stage research approach, also known as Registered Reports, common in other fields that use experimental designs – for example, healthcare's use of randomized control trials – and is emerging as a regular practice in other outlets such as Nature, PLOS One, and Academy of Management Discoveries. During the first stage, the authors were invited to submit their experimental design with specifics on their interventions for review, the so-called Stage 1 report. These designs include proposed research questions, articulations of how addressing these research questions might contribute to theory, intended interventions, and experimental sample and protocol. They would also include the nature of the experimental design (randomized control trials, pre-post), measures collected in the study, power analyses with expected attrition rates, intended managerial contributions from the work, and, if available, partner identification and confirmation. These designs underwent a full review that focused on evaluating the need for such experiments (i.e., importance of research question), factors studied and controlled in the designs, power analyses to determine appropriate sample sizes, and the relevant analyses planned for the designs. If approved, these research designs were published in the JOM website, and the publication of the final paper, irrespective of the results obtained, was guaranteed. Associated Stage 1 documentation is often published as protocol papers in respected journals (e.g., Trials, Implementation Science) before beginning their data collection.

Once a design has been approved, the authors engage on the second stage and conduct the experiment as approved. The authors then had the opportunity to submit the full paper with the pre-approved design and the results from the experiment, the Stage 2 report, which were then published in the SI after a quick round of reviews. We limited the submissions to the special issue to field experiments, as opposed to laboratory experiments, as ameliorating these risks and costs while working with a research partner, and collecting data from a real-world context, could arguably be viewed as having greater value.

The special issue, and its unusual review and publication arrangements, were designed to encourage field experiments in the OM context as it creates opportunities to improve experimental design and thus reduce the risk of experimental failures and provide an opportunity to publish their results even if hypotheses are not confirmed. However, the editorial team was aware that supporting such process would challenge to the journal's existing reviewing and editing processes and capabilities. As such, the special issue was also conceived as a trial run for identifying the stress points and potential solutions to continuously support this process. In this editorial, in addition to introducing the papers that were part of this process, we present the learnings for both the authors and the handling editors of the special issue. We conclude with our reflections on the main insights gained from this trial and the remaining challenges for deploying pre-approved research designs for Operations Management research.

The idea of PARD is somewhat new to OM. As a result, the number of papers submitted were smaller than a typical SI. Overall, we had 18 manuscripts submitted to this SI. One third of these submissions were desk rejected. Specifically, three proposals were rejected because researchers had already collected data from the experiment and were seeking for approval for the data analysis, thus defeating the purpose of the PARD. The other three proposals were rejected because experiments were testing technological options to improve processes – that is, the traditional design of experiment (DOE) approach (Montgomery, 2019) – and there was no OM theory behind the experimental design.

Of the 12 papers that were sent for review, four of them were rejected because there was not enough of a theoretical contribution. Four additional papers were rejected because of issues with the research design, that is, inability to randomize or control for sample attributes to rule out alternative explanations. These two reasons for rejection accounted for almost 50% of the submitted papers and they correspond to the main reasons we see for rejections when reviewing OM empirical work: lack of contribution or relevance, and identification challenges that prevent causal interpretation of the estimates (Cunningham, 2021). We realized that working in the field limits the researchers' ability to control or randomize the sample or manipulate the timing and intensity of the treatment – these are the realities of field experimentation. Nevertheless, being aware of the implications of these limitations is an important part of the process of deciding whether the proposed experiment will be effective in addressing the research question.

Three papers were rejected for what we called ‘inverted design’ process. These author groups attempted to leverage an existing experimental opportunity by operationalizing a treatment and constructs surrounding these events, that is, putting the experiment before the theory. Note that this is the context of a ‘natural experiment’ (Shadish et al., 2001) where researchers take advantage of a naturally occurring intervention to test elements of a theory. These proposals were rejected because there was no possibility to modify the experimental design, thus, again, defeating the purpose of the SI. Finally, one additional paper was withdrawn after the author team realized during the revision process that the theory they were attempting to test was not developed enough to have explicit causation mechanisms outlined, that is, the theory was still in its nascent or intermediate stage (Edmonson & McManus, 2007) and this ambiguity created problems in their measurement scales.

Out of these 15 rejections to the special issue, five groups of the authors were encouraged to resubmit their work as a normal JOM submission. For three of the papers the setting and the interventions were intriguing enough that they offered a possibility of insight outside of the causality testing that could be achieved through an experiment. Two other papers were thought to be a better fit to the Intervention-based Research department (Chandrasekaran et al., 2020; Oliva, 2019), as either the intervention was not detached enough or there was a lack of control group.

The three papers that survived the Stage 1 review process share the following characteristics. First, they all consider mature theories with explicit causal hypotheses and the experiment is designed to resolve paradoxes or empirical uncertainties. Second, the contributions of the research questions benefit the OM community and are not merely focused on the benefits of the individual sponsor of the research, that is, general OM theory. Third, there is a clear logical dependence where the research question is driving the research design and not the other way around. All accepted designs include a robust description of methods, protocols, measurements, and discussion of power. It was that methodological detail that allowed the review team to evaluate the research design and provide specific feedback and suggestions to improve it. Finally, all the successful author teams had the ability to work with sponsoring firms to identify proper controls and randomization of confounding factors during the review process. The three accepted designs all include randomized treatment designs with appropriate control groups. We present their designs and main findings in section 5.

Managing the review process for the special issue provided several lessons that can help move the field forward. Some of these takeaways are specific to evaluating field experiments while others are important to reflect on as we consider accepting study designs, in contrast to final research papers.

A first takeaway is that we currently have a limited resources in terms of the authors and reviewers within OM who are experienced in deploying and evaluating field experiments. That isn't surprising and it in fact helped motivate the special issue. However, any new(er) area needs development and support. Fortunately, the population of researchers is growing. Moreover, there are excellent resources to support the development of capabilities in this area. For example, the field of economics offers guides on field experiments (e.g., Levit & List, 2009; List, 2011) and both Ibañez and Staats (2019) and Gao and Li (2023) provide perspectives grounded in the field of operations. Evaluating papers appropriately means finding the right mix of reviewers with respect to methods and topics. At times this may be within the same person, however, as field experiment experience is still being built it may mean assembling a team with diverse experience and then having the editor integrate it.

This background and preparation are important as it helps reviewers to understand our second takeaway: the important role of tradeoffs in conducting field experiments. The current wealth of experimental experience in OM comes from researchers conducting laboratory experiments. These share many characteristics with field experiments but are not identical. When conducting field experiments it is often necessary to make tradeoffs between practicality and internal validity. All research involves tradeoffs, to some extent, and successfully completing studies involves evaluating them carefully with respect to a given question. Field experiments alter the environment, compared to lab studies, and the potential impact means that, at times, theoretical preferences might need to be forgone. For example, it may not be possible to control for all factors, the way that one could in a laboratory environment. In addition, unlike lab experiments, it is not possible to conduct multiple runs of data collection and therefore the design and the treatment conditions must be carefully thought through. Lastly, experimental designs may have to tradeoff sample size or mechanism identification to complete a study. This often means carefully considering giving up some internal validity in exchange for greater external validity. The authors must address problems, however, at the end of the day field experiments must be judged differently than laboratory experiments.

One key tradeoff that we came to appreciate is that of timing. To submit a field experiment for approval a site for the study must have been identified. However, once the company has been identified they often want to go ahead and run the study – not wait for a review process to be completed. As a result, the timing and commitment from companies can move faster than our review process converges. The authors must set expectations with companies, but also this means that the authors and editors must be in communication to address the reality of each, unique situation.

The third takeaway builds upon the tradeoff point, noting that the review process of a study before it runs can be a powerful aid to the authors as it presents the opportunity to consider alternative explanations and design issues while there is still time to address them. It is typically not plausible to rerun a field experiment. Reviewers are then put in a place of deciding if a completed study is good enough as-is. With pre-evaluation, reviewers can engage on this topic and determine whether the manuscript is worth publishing given the intended design and proposed theoretical contributions.

This leads to the fourth takeaway; the review process must consider something that has not been done yet and evaluate it appropriately. Any time we evaluate research we are concerned about type 1 and type 2 errors. Are we rejecting a paper that should be published or alternatively are we accepting a paper that has a flaw such that it should be rejected? The review process carefully evaluates a paper, and the team collectively does its best to address this challenge through rounds with the authors. In the Registered Report process where we have committed to publishing the final paper, the reviewers are forced to speculate about what may come out of a given study. Our concern with this process was, and to some extent still is, that we set the bar too high. The concern over publishing something that is eventually revealed to be “uninteresting” for lack of a better word, means that a review team may keep asking for more. This balance is one faced in all settings, but we encourage subsequent reviewers and editors to think carefully about this balance, avoiding unnecessarily putting the authors through the ringer and recognizing that if a design is correct then it is worth publishing.

Finally, during the SI PARD process, we specifically asked the review team and the authors to think about the benefits of conducting a study that produces a null result. That is, does the review team (and the authors) find value in (drafting) reading a manuscript that has theoretical and methodological rigor, yet yields a null result after collecting and analyzing the data. In our opinion, there is more development needed in OM to train doctoral students and scholars to appreciate the idea of learning from such results. Fields such as management have started to encourage such publications as evidenced by journals such as Academy of Management Discoveries (AMD). A key mission of AMD is to publish “research questions that address compelling and underexplored phenomena, novel or unusual contexts, and that reveal empirical patterns that cannot be explained by existing theory (https://aom.org/research/journals/discoveries).” We hope that more researchers start thinking along these lines to make our field interesting and relevant.

Though some of the author teams reported that participating in the PARD process increased their credibility and access with the sponsor firms, the introduction of a formal review after the design stage, but prior to the execution of the experiment, introduced challenges for the author teams. They often had to delay their experiments waiting for the review process or, perhaps more frustrating to their partners, change the experimental design to address concerns form the review team. In this section we summarize what the author teams have reported to be their main insights in handling the ‘shared’ design process and their relationship with sponsors.¹

First, all the authors teams made it clear that the individual experiment under consideration for the special issue was not an isolated intervention but rather part of a long-term working relationship with their sponsor. It was because of the existing relationships that the author teams were trusted by the sponsors to design a theory-driven intervention, and, when necessary, to modify the proposed research design based on the reviewers' feedback. The broader working relationship with the sponsor also created the degrees of freedom for the author teams to delay the execution of the experiment while it was going through the review process, for example, working on other aspects or objectives of the long-term relationship.

Second, all the author teams reported that the relationship with the sponsoring company required at least two levels of communication and engagement. One at the senior executive (C-) level to maintain the support for the long-term research relationship and its objectives, and one with the local managers responsible for implementing or executing the experimental treatment and measurements. They all reflected on the fact that these two conversations required different language and communication styles (frequency, mode, etc.).

Third, all the author teams defined their main concern as a balancing act to address the firms' concerns and profit objectives with the design and execution rigor required to address the research question. While holding true to a research question required explicit conversations with the C-level, they all reported a higher challenge in maintaining a balance between the ability to execute the treatments and measurement with the desired cleanliness of the research design. When it came to finding accommodation for those cross-requirements, two of the teams reported how useful it was to listen to line managers for implementation suggestions as they have a more solid understanding of the research context, and their recommendations were often counterintuitive for the researchers.

Finally, all research teams reported on the work required after the execution of the experiment to make the results meaningful and useful to the firm—for example, workshops to help interpret the meaning of coefficients, or assistance in developing guidelines based on the results—or to “make up” for disruptions caused by the intervention—for example, work to address new concerns or ideas emerging from the experimental results.

While some of these insights might in retrospect seem obvious, the consistency of insights across different interventions across vastly different geographical and cultural contexts is remarkable. More importantly, however, is the realization of all the work that happens behind the scenes prior, during, and after the experiment. Researcher's willingness to engage in this type of long-term relationships and ability to maintain them is perhaps one of the main reasons we do not see enough field experiments in our field.

Table 1 summarizes the papers that were accepted in this SI. The field experimental settings span different industry contexts that include online platforms, manufacturing and nanostore operations. The research questions poised in these studies challenged our existing theoretical understanding and the field experimental design was the appropriate methods in each of these cases. For instance, Son et al. (2024) look at an important issue of gender bias that can affect operational outcomes especially in larger online retail settings and hence required testing it in the field (when compared to a lab study). It is also encouraging to find that the experiments were conducted in regions that include Asia, South America and Europe which suggests that organizations all over the world are receptive to engaging in studies that advance OM's practical as well as theoretical knowledge.

The papers also used different manipulation approaches that were consistent with their research questions and at the same time practical to implement. For instance, it was easier for the study by Son et al. (2024) to randomize the client's questions into one of the five experimental conditions wherein the consultant's information about the gender were made available or masked to the client. In the case of Franke et al. (2024) randomization was done at the factory level wherein the workers in one of the factory received nudges about the availability of machines and interruptions through their smart watches. To avoid confounding effects, the authors selected a sister factory at a different location as a control group. In the case of Escamilla et al. (2024), the authors had two treatments (i.e., visit frequency and trade credits) with two levels (i.e., high vs. low) resulting in 4 conditions. The authors carefully randomized their assignments across these four conditions and adopted additional countermeasures to ensure compliance. For instance, the ERP system access in the stores with the low frequency condition were blocked for 1 week to ensure that the stores followed the same pattern as intended in the treatment.

In terms of the intended results, it is also important to note that not all the hypotheses proposed in the Stage 1 reports were supported in the actual experiment. For instance, Franke et al. (2024) argued for improved worker productivity when workers are nudged to take upon tasks that matched their skills. However, their experimental results suggest that nudging improves skill identification but does not influence productivity. The authors also describe the reasons for the lack of support for their hypotheses despite the right experimental design that advances our understanding on task identification and assignment literature.

Overall, the PARD approach to getting feedback on the designs before collecting data ensured that the authors developed research questions that challenged our existing theoretical understandings based on current literature.

What have we learned? Does it make sense to support registered reports in OM research? What are the challenges moving forward? From an outcome perspective, the evidence seems to be quite strong that review of Stage 1 reports is a worthwhile endeavor. All the successful author teams reported significant improvements on their designs as result of the early review process and feedback prior to the field experiment. Indeed, we were able to accept for publication all Stage 2 reports after minor revisions. A close look at the review process reveals that the editorial and review team essentially made sure that the research design (sample/treatment/controls) could answer the proposed research question while ruling out potential alternative explanations. The questions ‘how could we explain a null result on this experiment?’ and ‘could there be an alternative explanation for a positive finding?’ led the review teams throughout. The review process normally resulted in asking for higher specificity of the research question, additional controls to the experiments or randomizations of the sample, or the estimation of the power analysis of the sample. Granted, this sort of questioning is something that the author teams should/could have done for themselves or through a friendly review from colleagues. Nevertheless, we believe that the process of formalizing and documenting the experimental protocol to the point that it becomes possible to review and improve is an important part of the process that perhaps not all the author teams are willing to engage in. Although we do not have evidence on the outcomes of proposals that were rejected, we believe that the feedback provided to those proposals should have given pause to the authors to either re-design their experimental treatment or adjust the intended claims of their research. If all the pre-approval process does is to (weakly) enforce the norm to formalize and document the experimental protocol, we believe that on itself is a significant contribution.

By reducing the risk of a non-result and/or increasing the confidence that a null result will be considered, rather than ignored, by the field, we believe that the pre-approval process effectively serves the purpose stated for the special issue to encourage experiments questioning the status quo theories or proposing alternative explanations. Lower risk and higher confidence of these results should encourage more work with firms and for the field to further pursue engaged scholarship (van de Ven, 2007). Essentially, the PARD approach helped authors balance the tradeoffs between relevance and rigor (Keiser & Nicolai, 2005) while ensuring that reviewers and editors are comfortable with these choices, resulting in a manuscript that not only advances our theoretical knowledge but is also meaningful for the participating sites to learn from.

Considering the inputs to the review process, that is, submissions, we believe that, given the costs and challenges of performing the Stage 1 reviews, limiting the focus on field experiments was the appropriate call. The inability to easily re-run field experiments and the high risks involved in not having the appropriate research design does justify the extra scrutiny of the proposals. However, we do see two substantial challenges for the field moving forward with this type of reviews. First, as editors, we were surprised by the number of submissions with sparse theoretical content or contribution. We might have experienced some selection bias as the call for papers to the special issue favored proposals for field experiments, where having access to a research site is fundamental. Nevertheless, the main purpose of an experiment is to establish whether a particular causal mechanism is responsible for a result, that is, explicitly testing a hypothesis. Theory development or testing should be the main motivation to perform an experiment, with access to a site being also necessary. Yet, more than half of the submissions to the special issue ended up being rejected because of a lack of explicit linkage to a theory. We believe that these ‘atheoretical’ proposals may be the result of our current focus in our doctoral programs to train for methods at the potential cost of a deeper understanding of the underlying theories of OM or the theory development and testing processes per se. As a field, we ought to take a deep look of what it means to have a growing theory of Operations Management (Spearman & Hopp, 2021) and for the community to become theory-aware rather than just methods-savvy.

The second challenge in terms of potential inputs comes from the realization that all successful teams had their experiments amid long-term relationships with sponsoring organizations. While we do not see this as a strict requirement, we realize that the possibility to delaying execution and/or adjusting research designs after initial proposal is lot easier in the context of trust and a long-term relationship. Indeed, we lost a couple of proposals when the author teams were not able to adjust the experimental context to address reviewers' concerns. This represent a challenge for the field as current incentives in academia are not necessarily aligned to establish these long-term partnerships, nor for those that have invested to develop them to share the access. We believe that the possibility of reducing the cost and risk of executing field experiments can shift that calculus and encourage a stronger collaboration with industry.

From a process perspective, there are a couple of unresolved issues to successfully sustain the review of experimental designs. First, it is not clear what should be the ‘stop rule’ for a Stage 1 review process. Iterating until all possible design concerns are addressed by the author team, or until we realize that it is not possible to address them, seems to be too onerous of a load for reviewers and editors. As described above, we had committed to publish the results of any approved design, so in effect we had this ‘fix all or reject’ policy in place. In retrospect this might have been wasteful as we might have held the authors to a higher standard than necessary given the context of the experiment and the search for that certainty of outcome might have resulted in unnecessary iterations in the review process. What are the criteria to accept (or reject) a Stage 1 proposal is something that we need to further clarify before we can consider supporting registered reports in JOM. A related challenge is whether these Stage 1 manuscripts can standalone as an article for other researchers to replicate. While we did not publish these articles in print but rather had them as online supplements, such Stage 1 manuscripts standalone as research design pieces in other fields such as healthcare (i.e., Trials publishes such design pieces). Our humble opinion is that OM as a field is not ready for such radical changes but call on the authors and editors to think along these lines to improve replication and transparency around research designs.

Regarding the process itself, there are two design issues that need to be worked out for the process to be effective and sustainable. A Stage 1 report is not a full research article and a such needs to be reviewed through a different process and with different criteria than the processes we have in place for normal peer reviews. First, from the discussions in sections 3 and 4, is clear that review time is critical in this context; a 90-day review cycle (the current goal of JOM review turnaround) is often unacceptable. We currently lack the incentives to motivate reviewers to drop what they are doing and prioritize a review to provide feedback in days instead of weeks. Given the emergence of “fast track” submissions in our field, perhaps PARD should be thought of such type of manuscripts that may require faster feedback to ensure timely progress with the participating site.

The second challenge is how to staff this review process. Clearly some methodological expertise is required to assess the appropriateness of the research design. However, reviewers need also to be cognizant of the challenges of implementing and executing a ‘perfect’ research design in the field, and they need to possess a sense of how to rationalize the tradeoffs between the methodological desirability and practicality. That nuanced ability to handle the tradeoffs is normally acquired through experience. When we add the requirements for the reviewers to also be able to assess the theoretical contribution of the proposal, and the fact that the proposal is only evaluated from the design perspective (no data is available yet), it is clear the potential pool of reviewers for this effort is something that can only be slowly developed and through deliberate efforts.

While much work remains to be done, we believe that the upside from developing these capabilities can have a transformational impact on the field. We are motivated by the positive results emerging from this trial and the breath of lessons and insights that we gained from the experience. We thank the associated editors and reviewers that assisted us in this process and the previous Editors-in-Chief of JOM—Tyson Browning and Suzanne deTreville—for their initiative to experiment with this format. We hope that the results of this trial encourage JOM and the field to continue working to address the open issues of this promising process.