PLOS has posted a report, along with accompanying data, on qualitative research about how researchers assess credibility and impact of research in different contexts…
This post is written by PLOS’ Chief Scientific Officer Veronique Kiermer
Peer Review Week is a time to celebrate the outstanding work of reviewers across the scholarly ecosystem. But it also serves as a call to action to examine the systems we have in place and opportunities to improve.
This peer review week, the theme couldn’t feel more timely. Trust in science, and peer review more specifically has taken the spotlight as doctors, researchers, and policymakers around the world collaborate to rapidly expand and communicate emerging knowledge about COVID-19.
At the same time, the advent of preprints in biomedical sciences has created new challenges to evaluate findings outside the traditional framework of journal peer review, and high profile journal retractions have contributed to raise doubts about the effectiveness of pre-publication peer review. With the flood of information pouring in, public readers and researchers alike face the same question: how do I identify information I can trust?
Does Peer Review Work?
Peer review has long been a staple of the credibility of scientific findings for researchers and for the public. The process embodies the norm of organized skepticism that helps generate trust in scientific findings—by testing the rigor of thinking and experimentation, and by providing authors with expert feedback to build on. But importantly, peer review is more than the traditional view of pre-publication peer review conducted behind closed doors. Peer review does not stop with publication—in many ways, it only starts there.
Recently, in the context of the COVID-19 pandemic, we’ve seen researchers rapidly come together after the publication of high-stake results and point out serious methodological flaws which have led to retractions of articles and withdrawals of preprints. Social media has been an unconventional, yet instrumental tool in coordinated reactions. Some have argued this is a failure of peer review, but we believe it’s actually a celebration of peer review in different forms.
A broader view of peer review
While post-publication peer review remains rare, preprints offer new opportunities to leverage the community in order to address the challenges and limitations of traditional peer review. For example, reviewers and editors commonly acknowledge that it is increasingly difficult to cover all the bases of expertise required to properly analyze articles that have become much more multidisciplinary. Preprint commenting while the article is under review is one possible solution we’ve encouraged to address this gap, by opening up the opportunity for any researcher, anywhere in the world to volunteer their expertise, for consideration by the editor. While commenting on preprints is still rare, it could allow journals to channel within the traditional process the energy that the community already puts in scrutinizing new findings within their specialty before they are formally published in journals.
Preprints and preprint commenting can also help with another challenge of journal review: speed. PLOS journals offer authors the option to post preprints on their behalf at submission—after minimal checks for any content with potential to cause harm—so their findings can be disseminated while the journal curation process takes place. We also participate in initiatives like Review Commons to provide journal-agnostic reviews that can accelerate the peer review process at the author’s destination journal and alleviate the strain of repeated reviews on the peer reviewer community.
Beyond these practical issues, we must consider systemic challenges and in particular the documented biases in peer review. At PLOS we acknowledge the need to improve the diversity of our reviewer pool to help mitigate such biases and we are committed to this work.
Transparency, transparency, transparency
An important prerequisite for all peer review mechanisms to work as effectively as they can is transparency.
Transparency facilitates trust. Working to make every step of the research and publication process as Open as possible gives journals, peer reviewers, and the scholarly community more broadly, the means for scrutiny, and a clear understanding of limitations. At the same time, it gives readers deeper insight into the scientific process and the various checkpoints that uphold it.
From an author’s perspective, increasing transparency means making all relevant data, code, methodologies, and reagents available so that the scholarly community—starting with reviewers—can validate and reproduce the findings independently. A recent survey by the Center for Open Science indicated that researchers judging preprint credibility were influenced by cues related to open data, code and other openness measures, as well as independent verification of authors claims.
Publishing peer review reports as well is important as it makes evident what aspects of the article were (and were not) thoroughly scrutinized pre-publication. All PLOS journals now encourage their authors to publish their peer review history alongside the article, and the uptake is promising.
Peer review needs better recognition
Transparency in peer review is also crucial to recognizing peer review as a first-rate academic output. It is time for this activity, essential to the scientific process, to be considered a scientific output in its own right and to take its due place in the context of research and researchers assessment. With published peer review reports, individual reviewers can choose to take credit by revealing their identity. But recognizing that there are circumstances that call for anonymity, even unnamed reviewers should have this opportunity, and tools like ORCID and Publons allow them to demonstrate their contributions as part of their academic profile.
There’s always more we can do to provide the motivation and support for reviewers’ work. Peer review transparency and other visual signs of recognition are a way to start helping researchers leverage their peer review expertise to advance their careers.
Earning trust in peer review
Despite its challenges, the traditional journal peer review process offers important opportunities to ensure fairness in peer review. And journals have a collective responsibility to make use of these opportunities. First and foremost, we can ensure that every paper reporting a rigorously designed and executed study is peer reviewed—not only the high-stakes findings and the work of well known authors. We can also take proactive measures to limit bias, such as increasing the diversity of reviewer pools, and managing competing interests. We can foster consistent review criteria, and supplement the volunteer peer review reports with additional scrutiny for elements like statistics or ethical research practices.
In sum, I believe there are complementary benefits of journal-organized peer review and the process that can take place more organically in the research community post-publication—whether the publication is in a journal or on a preprint server. With more transparency in the system, we can take advantage of all mechanisms of peer review to decide what to trust, and importantly peer review itself can deserve our trust.