Diversity in peer review

This year’s Peer Review Week explores diversity in peer review. We know diversity can influence the outcome of peer review, for example, editors and reviewers from the same country as the authors are more favourable to those authors (an effect called homophily). How peer review is run might also affect the outcome of peer review processes, particularly due to the possibility of gender or country bias. Editors and peer reviewers can carry biases whether they are aware of them or not and this is where the debate about revealing or concealing the identity of authors and reviewers comes in.

In this blog post, I review the current diversity trends in terms of country of origin, gender, and different peer review models:

Country effects

The editorial composition of Hindawi journals, as calculated by Nigerian researchers for our journals in the ESCI, Scopus, and PubMed, is that about 3/4 of our editors are affiliated to institutions in Europe or North America and nearly 1/5 are at Asian affiliations. This compares to statistics from Publons’ recent survey showing that 96% of editors are from “established regions”, such as Western Europe, North America, and Australia.

Reviewer contributions from emerging regions, particularly China, lag behind the increase in submissions: the Publons data across publishers shows those from emerging regions contribute 0.66 reviews for every one submission versus 1.95 in established regions a ratio that is three times higher. This leads to a great burden on established reviewers and means researchers from emerging regions are under-involved in the key task of critical peer review.

Gender effects

Women make up a lower proportion of the scientific workforce in more senior positions and it is often well-established researchers who journals call on to become editors. Further, women tend to decline invitations at a higher rate than men, and women more often change their surname due to marriage, making it harder to track their publication record despite the use of ORCID. These factors can make gender balance in peer review and on editorial boards difficult to achieve.

Gender bias in peer review is often suspected, and editors at Frontiers were shown to use reviewers of the same gender more often, but several factors may limit the influence of gender: author lists are often of mixed gender; journals do not ask about gender (as opposed to affiliation); and many names may not be gender specific, especially across different cultures. Reflecting this, studies usually rely on disambiguation tools to assess authors’ gender. Unless the study is about gender, it is likely that few reviewers or editors will be conscious of the authors’ gender. A study by the AGU of their journals found that authors and editors suggest fewer women than men, but this did not affect editorial decisions, and a study of a public health journal found no gender-based differences in their review process.

However, men suggest peer reviewers more often than women do and author-suggested peer reviewers are more lenient than editor-selected reviewers: together, this will lead to men getting an easier ride. Also, men self-cite at higher rates than women and women are more willing to pad their references to please journals. However, at Hindawi we counter such trends as we don’t allow author-suggested reviewers and we monitor citation manipulation, including excessive self-citation by authors, citation stacking, and requests from reviewers and editors that authors cite their own work. Fortunately, there seems to be little difference in citation rates by author gender.

Peer review models

The different models of peer review single-blind, double-blind, open, and post-publication have been proposed to make a difference to diversity and inclusion, and were debated at the Council of Science Editors conference earlier this year (see my tweets and slides of the talks; there are many definitions of open review, and the one relevant here is naming reviewers to the authors). Hindawi’s journals use single-blind review the most common model in STM publishing though our proposal to the European Commission earlier this year included open peer review options.

Many fear that junior researchers and minorities will face hostility and retribution under open peer review, though blinding is no panacea: attempting to blind reviewers to authors’ identities is often unsuccessful as many reviewers can correctly guess the authorship. Contrary to what many might expect, a study at a computer science conference that used signed reviews found junior reviewers were harsher in their comments and received more polite reviews.

Even when successful, blinding may not conceal the country of origin, and English-language difficulties may still influence the outcome of the review process. A study of Nature journals found double-blind review to be unpopular among their authors, with no gender difference in preference but a clear country difference, particularly for China and India. If authors chose double-blind because of a perception that bias was causing their articles to be rejected, they will have been disappointed: more were desk rejected and of those sent for review fewer were accepted. Researchers from countries with low acceptance rates may need support before the peer review process, rather than tweaks to the review process.

Another effect of double-blind is removing an assessment by the reviewers of conflicts of interest (COIs), agendas, and knowledge of the lab’s abilities and previous work. Some knowledge of the authors may not be “bias” but “priors”. The importance of this is reflected in the fact that if undeclared COIs are raised on a published Hindawi article, we ask the editor to reassess it in light of that new information and sometimes formally re-review the article.

Diversity and inclusion in peer review is a burning issue with multifaceted implications that require the attention of all stakeholders involved publishers, reviewers, authors, funders, and institutions.

We welcome the Peer Review Week 2018 initiative to examine the diversity of those involved in peer review and to put this issue on the front burner in all of its different aspects. It is a work in progress, but we hope that the discussions that took place this week will help to bring about the changes needed towards a more open, more inclusive scholarly communications system.

Matt Hodgkinson oversees publication ethics at Hindawi. He was previously a Senior Editor at PLOS and BMC.

Opinions in this blog post are that of the author, and not necessarily that of Hindawi. The text in this blog post is by Matt Hodgkinson and is distributed under the Creative Commons Attribution License (CC-BY). Illustration by Hindawi and is also CC-BY.