Community Comments Continued: An update on the preprint commenting pilot at PLOS

Around 11 months ago we announced our year-long  preprint commenting pilot to connect bioRxiv comments on preprints that are under consideration at select PLOS journals to the editors handling those articles. (The journals in the pilot are PLOS Computational Biology, PLOS Genetics, PLOS Neglected Tropical Diseases, and PLOS Pathogens.)

We have one month left on this pilot, and in the interest of making the most of this, we also want to hear from YOU, the open science community. Have you ever received public feedback on a preprint? Do you comment on preprints? Privately or publicly? Do you read preprints but feel that you can’t give feedback?

If you answered yes to any of the above, we want to hear from you.  Tell us your preprint story here.

As we anticipated, during this pilot the majority of eligible preprints haven’t received any public comments on the bioRxiv platform. Staff facilitating this pilot have observed that, while some preprints receive high quality feedback in the comments, most do not. Also, the timing of preprint commenting has not always lined up with the peer review process, so that preprints have received detailed feedback after an editorial decision had already been made. 

Pilot programs provide an opportunity to take the ongoing temperature of the community regarding innovations in scholarly communication and researcher behaviors. A 2019 survey reported that 53% of preprint posters do so in order to receive feedback. So how should we interpret this slow uptake of commenting on the bioRxiv platform?

We could easily assume that there just isn’t a lot of interest in preprint commenting. Yet a recent flash survey of reviewers across all the PLOS journals (not just the ones in this pilot) revealed that at least 11% of those surveyed had provided public feedback on a preprint. While a few reported giving private feedback, the majority of respondents told us that they read preprints but don’t comment on them. 

Furthermore, ASAPbio recently found that more than 50% of researchers indicate that the opportunity for early feedback on preprints was very beneficial, suggesting that authors could benefit from preprint commenting. 

In the early stages of this pilot we already knew (from the same 2019 survey above) that researchers have reported a preference for private feedback, and that most researchers who have received feedback have received it privately via email or conversation, or, if publicly, via Twitter. Our pilot, which only covered comments left on the bioRxiv platform, had no way of noting whether any comments were actually coming to the authors via other channels.

When we launched this pilot we also talked about why we engage with preprints. We see preprints as an important step towards opening up peer review to more voices beyond those expressly invited by the editors. Far from being a side-step around peer review, as some detractors see them, we see preprints as a way of getting even more eyes, and therefore reviews, on an article undergoing peer review. As our Chief Scientific Officer, Veronique Kiermer, wrote in a post during Peer Review Week, preprints maximize and diversify peer review in a way that the traditional process cannot, in a way that complements that traditional process and makes it even stronger. This is all “a celebration of peer review in different forms”!

Therefore, we remain committed to providing greater opportunities for research communities to participate in reviews of preprints, while promoting transparency in peer review, but we will now consider how we could refine processes around facilitating engagement with preprints during the peer review process. We believe that maximizing and diversifying the voices involved in peer review is simply a behavior we must patiently cultivate, and we will use the lessons learned from the pilot to help us.

Remember, please complete the survey! We have one month of the pilot left to go, and we would like to complement our final report with the findings from this survey.

And, please watch this space for more posts about preprints, peer review, and all the ways in which we’re working with the research community to improve trust in science.

Note: This blog was written by Harry Porter, a program assistant at PLOS.