Peer review… with a little help from my friends

Reviewing Papers Collectively | Hindawi Blog

I recently ran a twitter poll to gauge opinion on whether it was OK for a reviewer to discuss a manuscript under review with a third party. The results surprised me, and led me to question: Should we be encouraging or suppressing the discussion of others’ unpublished work? These are my thoughts.

The next generation

Peer reviewing is a skill, and one which is best learnt through practice. I don’t think most authors would be pleased to have PhD students — refereeing their first ever manuscript — to be the sole reviewers of a document on which their hopes, dreams, and future research grants depend. And yet there are very good reasons why PhD students and post-docs should be involved in the review process.

Early Career Researchers bring a wealth of knowledge that, though they might not admit it, Principal Investigators simply don’t possess.

Firstly, introducing junior researchers to the mechanism of peer review at an early stage is important for their education. It develops their skills as future senior researchers, who routinely peer review manuscripts (and perhaps funding applications), and encourages introspective critique of their own methods and processes. Of more immediate benefit to authors, Early Career Researchers (ECRs) bring a wealth of knowledge that, though they might not admit it, Principal Investigators (PIs) simply don’t possess. Indeed experienced PIs with deep subject knowledge may rarely step inside the research lab, keep tabs of the latest experimental practices, or run their own data analysis. Knowing how to practically run an experiment with the latest equipment and techniques is often better known by those who have hands-on, day-to-day experience of running them rather than, say, an experienced professor who is more comfortable behind a desk. After all, a reviewer’s report shouldn’t merely state whether a manuscript reaches the editorial criteria set out by the journal. Rather, regardless of this subjective verdict, it should identify its strengths, weaknesses, and opportunities, such that the authors may improve their research and strengthen the study’s conclusions. Thus fresh and imaginative approaches from emerging scientists, albeit with a pinch of naivety, are surely a valuable resource. Indeed despite our best intentions, this innovative thinking is too frequently lost to systematic approaches borne out through repetition, and from seeing every problem from the same perspective. However, inexperience also comes with pitfalls, a common trait of a rookie reviewer can be the tendency to concentrate on the minutiae of a study, whilst missing the bigger picture.  A list of very specific criticisms or preferences, which bear little impact on the study as a whole are generally not very constructive. Similarly, whilst clear writing is important, a list of typos raises questions of how much time was spent reviewing the science, rather than the grammar.

A common trait of a rookie reviewers can be the tendency to concentrate on the minutiae of a study, while missing the bigger picture.

Another issue, potentially more prevalent in ECRs, is the temptation to take the name of the authors into account when composing a report. This can go both ways. A well-known or famous researcher may get an easy ride from a junior reviewer who is overawed and hesitant to point out errors. Equally, seeing a ‘big name’ could encourage emerging researchers to level the playing field by providing an overly critical review. Of course, neither of these scenarios is exclusive to less experienced reviewers, and perhaps in itself a good reason for reports to be moderated by a colleague.

Allowing PIs and ECRs to compose a report in conjunction allows the PhD students and new post-docs an element of responsibility, whilst being under the close supervision of an experienced eye. Consider the alternative whereby at a certain point researchers were deemed ready to review manuscripts alone, having never before performed such a task. I know which I’d prefer.

Strength in numbers?

It seems obvious that having more eyes on a piece of work enables a deeper critique of the study, and is more likely to provide greater coverage of the topics under discussion. At a minimum, bouncing a reviewer’s report off someone else, or having them sense check it, can be useful in ensuring the intended messages are being delivered and that no obvious mistakes are made in the report.

...having more eyes on a piece of work enables a deeper critique of the study...

Moreover, as research — and research groups — become ever more interdisciplinary, and papers are written in cross-continental collaborations (and even massive research consortia), it would appear necessary to broaden the range of reviewers expertise dedicated to each manuscript. Yet it is apparent that academics are frequently overwhelmed with the number of requests to review new manuscripts. Regularly seeking four, or five, or even more reviewers for a particular manuscript is simply not feasible or sustainable. However, a brief discussion with knowledgeable colleagues could lead to quick, and no less informed, evaluation of concepts which are valuable but perhaps peripheral to the main thrust of the article. This both lessens the total load on the reviewer pool and allows faster feedback, which potentially shortens the decision period. A natural extension of this idea is to discuss unpublished papers as a research group, in a similar setting to a journal club. However, dangers do present from this ‘team effort’ approach to peer review.

...there may be a propensity for discussions to generate a shopping list of necessary further experiments, as each person has their own ideas for where the research should be headed.

The quality of the reviewer report is not necessarily improved by combining the views of a number of people. Though for the most part collective reviews are delivered with the best of intentions, groupthink has the potential to deliver an unfocused report where the most important points, critical for validity of the study, are lost amongst optional extensions to the study and personal preferences. Moreover, there may be a propensity for discussions to generate a shopping list of necessary further experiments, as each person has their own ideas for where the research should be headed. Indeed, many authors already feel that requests from over-zealous referees place unjustifiable hurdles before them, and it can take a strong editor to make editorial judgements on whether these are really necessary.

Increasing the number people involved in a peer review drastically increases the risk that confidentiality with the outside community is compromised. Not only are there more sources from which a leak may originate, but the communication streams between team members are also susceptible. We are all aware of embarrassing incidents where an email chain is accidentally forwarded to the wrong person, however the results of such indiscretions can in fact be very serious. I have also been made aware of cases in which unpublished manuscripts have been made publicly available by group members who erroneously assumed they were sharing documents on a secure internal server. Furthermore, the propensity for group members to discuss other researchers’ unpublished work in non-secure environments is increased once a culture of discussing manuscripts is established. The aforementioned issues are only the ones for which leaks are accidental, and more nefarious deeds are certainly possible.

...those privy to unpublished work can conceivably use their newfound knowledge for their own benefit...

Reviewers are not supposed to make use of information learned whilst reading unpublished manuscripts. It is every author’s fear that their hard-earned results are scooped, that is, very similar work from a competitive research group is published just ahead of their own. This can be highly distressing. While the work is usually still appropriate for publication somewhere, the lack of impact means that highly selective journals are unlikely to publish it. The onward effect, especially for emerging researchers, can be devastating. That said, the race to publication is generally understood to be a part of academic life, and indeed the competitive nature of academia can lead to more efficient and productive research processes. However, given the sometimes lengthy duration of the review period, those privy to unpublished work could conceivably use their newfound knowledge for their own benefit, before the work is published. Reviewers using or sharing privileged information is a real danger, and that danger is only increased when more people are involved. On the lighter — but all the same wrong — end of the scale, it has been known for PIs to circulate a manuscript amongst a research group and suggest they accelerate their efforts because their competitors are close behind. In the worst-case scenario, whole studies have been stolen and published elsewhere. Such practices could of course occur regardless of whether manuscripts are shared or not, but the risk is greatly increased when looser controls are in place.

Lastly, editors generally have very good reasons for asking a particular reviewer for a report. They value their editorial opinion, and rely upon their experience in the field to draw informed judgements. Good editors can assess a reviews’ report in the context of the reviewer’s scientific background, if they wanted advice from another perspective they’d seek it from elsewhere.

So who says what?

Regardless of the pros and cons of reviewers receiving help with their review, it is important to also consider the expectations of authors, and obligations of the publisher. Authors submitting to journals that employ the traditional single blind peer review model, do so with the expectation that their work will be kept confidential. Despite this, over half of the nearly 600 respondents in my Twitter poll felt that it was OK to discuss someone else’s unpublished work without the prior permission of the editor. That said, they clearly drew a line when it came to seeking advice further afield without permission; 9% felt that discussion with other colleagues was acceptable and just 4% thought that they could consult with a collaborator.

OK, it’s not a very scientific poll: the sample size was small and I suspect not representative of the broad scientific field. Nevertheless, expectations are clearly not uniform. I sampled a collection of publishers’ guidelines to see whether their advice presented a more consistent view. All but one of those I checked suggested either that reviewers should not discuss the manuscript at all, or that permission was first required from the journal or editor. The sole exception stated that while permission was required for advice sought from outside the research group, advice from within could be noted after the fact when the report was submitted. These policies, for the most part, adhere to the advice from the Committee on Publication Ethics (COPE).

Excerpt from the COPE Ethical Guidelines for Peer Reviewers (March 2013, v.1):

When preparing the report

Peer reviewers should:

• not involve anyone else in the review of a manuscript, including junior researchers they are mentoring, without first obtaining permission from the journal; the names of any individuals who have helped them with the review should be included with the returned review so that they are associated with the manuscript in the journal’s records and can also receive due credit for their efforts.

• keep all manuscript and review details confidential.

So according to COPE, permission from the journal is critical before reviewers discuss manuscripts under review — and your own research group is no exception.

Wrap-up

It is worth mentioning that other modes of peer review are currently being explored, and exciting innovations are being made all the time. Similarly, an increase in use of pre-print servers, albeit not uniformly across disciplines, is changing the way we think about the routes to disseminating our work. Nevertheless, single-blind peer review remains the dominant mechanism, and unless (or perhaps until) open/collaborative/post-publication review becomes the norm, it makes sense to assess the processes we currently have in place.

Anecdotally, I know that not all reviewers adhere to the policies laid down by their journals. I also know that in the vast, vast majority of cases, the practices of reviewers are performed in the best interests of the authors, and the community. Furthermore, I suspect that many may not even be aware that their practices are contrary to journal policy.

At a time when reviewers feel increasingly burdened by what many perceive to be an unpaid community service, I’m not convinced that we should be dictating how they should do their job, or putting new hurdles in their way. Moreover, the benefits of discussing manuscripts in a conscientious manner with a colleague (junior or otherwise) would appear to bring many more rewards than it does risks. But the bottom line is, ask for permission first.

The text and illustration in this blog post are by Hindawi and are distributed under the Creative Commons Attribution License (CC-BY).


Many academics and editors shared their experiences and opinions of the peer review process with me whilst I wrote this piece. While some were happy to be named, others preferred to remain anonymous. On balance, I have decided not to mention any names, and not to discuss any specific cases. Nevertheless, I thank them all for their input.