When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

PLOS BLOGS The Official PLOS Blog

The potential butterfly effect of preregistered peer-reviewed research

Preregistration has the power to level up research communication. And it’s so easy.

Written by Lindsay Morton

Preregistration is the practice of formally registering a study design before experimentation and data collection begin. When deposited in a repository, a preregistered study design functions as a statement of intent laying claim to a certain investigatory pathway. The preregistered study design functions as an everyday tool for executing, reporting and repeating high-quality confirmatory investigations, as well as a stable methodological record capable of facilitating reproduction and reuse over the longer term. When practiced at a journal, two additional elements are added: peer review feedback, to help optimize the methods, and analytical approach; and provisional acceptance of the resulting research article when the investigation is completed, whatever the outcomes may be.

“Peer review of the registered study protocol provides an opportunity to improve the study design before starting to collect the data. That’s the most constructive time for the review to happen. And a published registered protocol is a guarantee for the authors that even null results will be published.” says Veronique Kiermer, Chief Scientific Officer at PLOS. Authors of preregistered research concur. As Jacqueline Scholl, Oxford University puts it, “”One major advantage compared to standard peer-review was that when reviewers made sensible suggestions about additional control variables to collect, we could actually implement this; rather than just thinking ‘we wished we had done that’.” 

In practical terms, neither form of preregistration outlined above is all that different from a standard research workflow. When conducting confirmatory research, investigators routinely define a hypothesis, develop methods, and choose how to collect and analyze data. These study design plans are frequently formalized as part of a grant application, and may well undergo peer review during the funding process as well.

Therefore, taking the minimal extra step of depositing a study design with a repository seems like such a small thing. A few buttons clicked, a document uploaded, and a check box or two. Initiating journal peer review earlier in the process is a similarly a minor shift; a simple reorganization of existing processes into a more efficient configuration. And yet, on the individual level, and especially in aggregate, these modest tweaks to the peer review and publication workflow have the potential to exert an outsized impact on research and scientific communication.

Privately depositing a study design with a repository helps researchers stay accountable to themselves, reducing the risk of behaviors like p-hacking, selective publication, or suppressing negative and null research outcomes. Researchers who choose to make their study designs public from the time of deposition help to avoid unnecessary repeated studies, unintentional complementary research, and scooping, freeing up their fellow researchers to pursue different lines of inquiry and improving efficiency in the field. When private repository study designs become public upon publication of a research article, their existence serves to bolster the final work, demonstrating integrity on the part of the authors, and supporting trust among readers. Finally, retaining a permanent methodological record documenting the data collection and analysis as it was actually planned and performed ensures future reproducibility as almost nothing else can.

Refocusing journal peer review on the study design phase exerts more and greater downstream changes. Peer review that focuses on evaluating significance of the research question, the methods and analytical approach before work begins, has the power to shape stronger, more rigorous and more creative research. Making an editorial decision while results are still unknown minimizes the potential impacts confirmation bias and impact bias, taking science communication back to its roots, with an emphasis on quality, rigor, and a pure intellectual curiosity. As Kiermer explains, “Preregistration and peer review of the study protocol with a journal is a way to tackle publication bias. As long as the protocol is followed, or any deviations explained, it’s a guarantee for the author that the results will be published, even if they don’t confirm their hypothesis.”

In combination, all of these factors contribute to a more complete and efficient scientific record, replete with studies exploring important hypotheses, performed to the very highest technical standards, and free from the distorting influence of impact-chasing, ego, and bias. A scientific record that is both demonstrably trustworthy, and widely trusted. And with that, there is no telling where science might go, or how quickly.

  1. Re: “Privately depositing a study design with a repository helps researchers stay accountable to themselves, reducing the risk of behaviors like p-hacking, selective publication, or suppressing negative and null research outcomes.”

    This will make it almost impossible for biologically uninformed theorists to link their experimental evidence to stupid theories.

    For comparison, I’ve watched as the research of all intelligent serious scientists on light-activated microRNAs was linked from the ATP-dependent Creation of RNA to former President Trump’s claim on 4/23/20 that sunlight and humidity weaken COVID19 via more than 138,000 indexed published works that mention “microRNAs.”

    Now, others can learn about the facts that link the energy-dependent microRNA-mediated construction of the cell wall to protection from all virus-driven pathology via the STEM toy “Cellulose” for ages 14 plus. See: https://www.kickstarter.com/projects/geniusgames/cellulose-a-plant-cell-biology-game

  2. This is a very interesting proposal and approach. Two things in particular strike me. First of all, isn’t this going to overburden the reviewer pool even more? But perhaps even more important: Is a journal seriously going to guarantee publication because one has been through peer review with the proposal? Is there an “acceptance” stage, i.e. could the peer review of the proposal in principle lead to rejection? But, again, I like the vision. I’m just wondering about these points. Looking forward to your reply!
    Curt Rice, Rector, Norwegian University of Life Sciences

    1. Hi Curt, thank you for these thoughtful questions!

      Taking them in order…

      Will preregistration overburden the reviewer pool even more?

      From the reviewers’ point of view, the effort involved in assessing a preregistered research article should be similar to that of reviewing a standard research article–even though the review does take place in two discrete stages. In this model the bulk of constructive feedback takes place during the study design phase, while the review of the final research article focuses on confirming the authors followed their approved protocol, and assessing the results and conclusions for thoroughness, accuracy, and clarity.

      However, there is the possibility that, because of the longer period of time between study design and finished article as compared with the revision rounds of a standard research article submission, the same reviewers may not be available for both stages. That could make the assessment more challenging for reviewers coming to the full research article without first having seen the proposed study design. Something to watch for in the publishing system metadata.

      Do journals seriously guarantee publication based on peer review and provisional acceptance of the proposal?

      Yes! Provided that the authors adhere to the peer reviewed and approved study design, and uphold all applicable standards for ethics and reporting, they can rest assured that the article will go on to be formally accepted and published once the results are in. You can read a bit more on how the process has been implemented at PLOS ONE here.

      Importantly, preregistered studies can include post hoc analysis in addition to the analysis outlined in the original study design, if warranted.

  3. Know how the potential butterfly effect of preregistered peer-reviewed research by reading this blog. thanks for such valuable information.

  4. But I have only read one study that compared published studies to what they registered (can’t recall the reference off the top of my head), and it was not highly favourable – that published research did diverge from its original protocol and this was not always mentioned. So perhaps we need a better mechanism for an external party to actually check that the procedures, analyses etc. followed the pre-specified procedures. This seems too much to ask of voluntary article reviewers, who already seem sometimes too busy to give thorough journal reviews

    1. Great question! As you point out there isn’t yet a great deal of empirical research on Registered Reports in life sciences, so while preregistration has tremendous potential, it remains to be seen whether that potential will be fully realized. We hope more data is on the way as the format gains traction, and we do plan to share our own observations from preregistered research published in PLOS ONE and PLOS Biology.

      I wonder if you could be thinking of this study, from Royal Society Open Science in which the authors reviewed articles published in Psychological Science between 2015-2017 and found that, of the 27 articles with a preregistration badge, only 2 followed their original study design to the letter, and only one explained the deviations in detail.

      In considering the implications it can be helpful to examine the different modes of preregistration. In the above article (which may not actually be the one you had in mind!) the articles in the study first preregistered with a repository, and later submitted to and published in a journal. The editor and reviewers may have considered the preregistered study design, but it wasn’t inherently part of their peer review process, so it’s hard to know. It would be interesting to see a similar review of preregistered articles where both the study design and the final article undergo journal peer review.

      Interestingly, the authors of the above article refer to the 2015-17 articles they evaluated as “first generation” preregistrations. This begs the question–does practice lead to improvement? Seven years on, is the same cohort still practicing preregistration and if so, has their reporting improved?

      Another study published in PLOS Biology compares two different methods of repository preregistration, one structured and the other unstructured. The structured format was found to restrict what the authors describe as “opportunistic degrees of freedom aimed at obtaining statistical significance” more effectively. But neither format entirely eliminated these opportunities for manipulation.

Leave a Reply

Your email address will not be published. Required fields are marked *

Add your ORCID here. (e.g. 0000-0002-7299-680X)

Related Posts
Back to top