At the heart of the Open Science movement is the conviction that Open Science is better science. More rigorous. More inclusive. More…
Let’s Get Creative With Our Peer Review!
I’ve been around the editorial block a few times now, as a volunteer editor, peer reviewer, and author/co-author. One of the most dreaded steps of the whole process concerns author-recommended peer reviewers. It can be agonizing as an author to make the selections and hard work for an editor to figure out which suggestions have a likelihood of panning out as meaningful reviews. In the various papers I’ve authored, co-authored, and edited, a few problems appear with some frequency:
- Adding Dr. McFamous to the list of reviewers. Odds are non-negligible that this person is already overtaxed with review requests and will say no. If they do accept, odds are also non-negligible that their review will be about three sentences of fairly uninformative commentary, submitted 2-3 weeks after the deadline. Unless this person is a legitimate expert who has recently published on the specific topic of the paper, find someone else who is more qualified and more likely to have the time to review the paper in the detail it deserves. I am somewhat surprised by the frequency with which authors will suggest someone simply because they are well-known, without regard for whether the person is the best expert for the job.
- Submitting a “no fly” list of reviewers that is a mile long. There are legitimate reasons to request certain researchers to not review a paper (conflict of interest, real or perceived unprofessionalism, whatever), but anytime more than two or three people are listed, that raises all sorts of red flags for an editor. Possible explanations? 1) One of the authors has ticked off 90% of their colleagues. 2) The paper is so flimsy that anyone with real expertise is going to shoot it down. 3) The authors won’t accept even constructive criticism and are going to be a pain to both reviewers and editors. Note how none of this reflects well on the authors. As an editor, I do my very best to respect authors’ wishes, but there are times where some requests really stretch credulity.
- Suggesting reviewers from the same institution or research group as one of the authors. This happens with surprising frequency; even if you are at a quite large institution or within a fairly large research group, and even if the suggested reviewer is only a vague acquaintance, it has the appearance of a fairly major conflict of interest. If at all possible (yes, it can be difficult in small subfields or for certain taxa), suggest someone else.
- Same. Old. Reviewers. This syndrome regularly affects authors and editors alike. The result is a list of overwhelmingly male suggested reviewers, overwhelmingly from a handful of large Western European and North American research institutions. It is sometimes correlated with #1 (Dr. McFamous), because said expert once published a paper on the subject in Nature back in 1998, even if their research for the past 15 years has taken an entirely different direction. In all honesty, these scientists are not the only people qualified to review their papers, or often even the best people. Some really exciting science is happening at small research institutions, community colleges, and places outside the USA, by people from exceptionally varied backgrounds. If we say we want to diversify science (and we all hear this rhetoric frequently), let’s get serious about it when it comes to suggesting and inviting reviewers!
If I had to guess, #4 (“Same. Old. Reviewers”) is the most prevalent, and I am undoubtedly guilty of it too, from both sides. That said, I’ve been making some serious efforts both as an editor and as an author to recognize and address the problem. Google Scholar is a major asset in this regard–a quick search on a topic, narrowed to the last two or three years of publications, will often bring up at least two or three reviewers with appropriate expertise. I tried the strategy with a paper I submitted recently and identified a couple of surprisingly logical and undoubtedly qualified researchers whom I had embarrassingly overlooked. The result was a deeper, more balanced suggested reviewer list and presumably a better review of my paper.
So, here’s my challenge to authors: be more creative on your list of suggested reviewers. And for editors: be more creative in whom you ask to review papers. Let’s build a better world of peer review!
[Just to reiterate, all opinions stated here are solely mine, and do not represent an official (or even unofficial) stance by any organization with which I am associated. Furthermore, the issues discussed here happen frequently enough over the years (seriously!) that it should be apparent I’m not talking about any paper in particular.]
I once suggested a reviewer who I knew would instantly reject the paper because I was the author, but I was interested in their comments and did not want them to feel they were cut out of the process because of our disagreements. I simply warned the editor that the review would most likely be hostile. It was, but the editor did not reject the paper because of it and I did incorporate the reviewers ‘non-hostile’ comments into the final draft. Risky but creative?
A clever move from Bill. I’m not sure I’d have the guts to pull that one.
Here’s a related issue. If I as an author I dis-recommend a reviewer, I have a good reason for that. As a handling editor, you should respect it by not inviting that reviewer; or if you really think there is no good alternative reviewer, just return the paper to the author and explain why.
I have heard of handling editors who deliberately invite reviewers that authors have dis-recommended; I have even heard it said of some handling editors that they boast of this behaviour (though I am pleased to say that I’ve never heard this boast at first hand). This is grossly unprofessional and a waste of everyone’s time.
My policy is now that if a handling editor does this, I will simply withdraw the submission and take my paper elsewhere. The sad fact is that there are “roadblock reviewers” out there who see their job as preventing publication — and whose only real effect is of course to cause another whole round of submission and review at a different journal. No purpose is served by letting these people play their games.
Suggested and non-preferred reviewers is an area that could do with clarification/guidance for many authors and editors – so, great to see your post.
Researchers, especially those at an early-career stage, aren’t always aware of who should be excluded from their lists of suggested reviewers. Journals could help greatly by giving authors guidance on this, but many don’t.
Journals can also help their editors by stating that the number of non-preferred reviewers put forward by authors should be limited to 2-3, and by asking that reasons be given for those requests.
Editors and editorial staff need to be trained to know how to handle both suggested and non-preferred reviewers when making reviewer selections for submissions- everything from the checks that should be carried out to the strategies that need to be considered. We know from the spate of ‘fake reviewer’ cases that started coming to light a couple of years ago
http://retractionwatch.com/category/by-reason-for-retraction/faked-emails/
that some journals haven ‘t been carrying out even the most basic checks on author-suggested reviewers, and consequently manuscripts have been sent for review to the authors themselves via the false identities and email accounts set up by them.
I like it! Indeed risky, but perhaps the most constructive way to deal with such a reviewer. This sounds like one of those cases where having a trustworthy editor who knows the personalities involved goes a long ways.
Absolutely agreed, Mike! There are only a few very limited cases, usually stipulated by journal policy, where an editor might still overrule such requests. For example, PLOS ONE has a policy to invite as a reviewer anyone whose work is being specifically called into question or disputed by a submitted manuscript. This seems fair for most situations, and as long as everyone knows this going in I think it’s a good way to deal with such potentially awkward occurrences. But, in any other case, I think it’s just the right thing to respect author wishes as much as possible.
Very good points! In terms of guidance to authors, this is part of why I wrote the post, too. Of course, as you note, official journal guidance would be even more helpful. I suspect the culture varies between journals and even fields.
Interesting! While that sounds like a good policy, I think in practice inviting such reviewers can often result in their trying to block publication of work critical of their own, so I am not sure it’s such a good idea. I know in my own case I have at least once requested such an author not to be invited, precisely because it would have caused a conflict of interests. (At the same time, I asked not to be considered as a reviewer of that researcher’s contribution to the same volume, for the same reason — I didn’t want to be put in the situation of having to recommend rejection of work that I knew in advance I was likely to disagree with).
In general, I think it’s better to let such things be reviewed by neutral third parties, and allow disputing researchers to keep slogging it out in further papers — not least because that way the whole world can see the unfolding sequence of argument, rather that it being buried away in a peer-review report that only two people ever see.
That said, Bill’s solution to this one is not bad at all. If I could get constructive comments from someone who I know disagrees with me, while knowing that person wouldn’t have the opportunity to block publication, then that would be truly useful.
[…] journals asking you to recommend reviewers? Get creative, says Andrew Farke! Well, not too creative, of […]
[…] Let’s Get Creative With Our Peer Review!. […]