Standardization vs Diversity: How Can We Push Peer Review Research Forward?

Karen Shashok

In This Article

Defining the Functions of Peer Review


Overbeke and Wager[1] put their fingers on one of the most important challenges in peer review research: "Given the lack of consensus about the primary function of peer review, it is perhaps not surprising that a review of the literature reveals little agreement about the outcomes that should be measured." So deciding which function of peer review is to be tested is an important step. For example, when the aim of peer review is to aid the editor in tending the gates (ie, in deciding what information "deserves" dissemination and what information is deemed not to be of sufficient "priority"), research methods from the behavioral, social, and anthropological sciences can be useful. Methods in these sciences are likely to help editors find out how peers reach a consensus about what is "relevant to the field" and about how motivation and decision making influence the outcomes of peer review. If the questions we are asking are "What makes some reviewers better than others?" or "How can I motivate reviewers to do a good job?," then advice from ethnographers (ie, scientists who study behavior in specific communities and cultures) and other experts in human behavior may well be useful. The potential contributions from the methods used in these areas will depend on the degree to which the editor relies on the reviewers' opinions in reaching a decision on imprecisely defined variables, such as "relevance," "importance," or "timeliness."


Another common function of peer review, whether intended or not, is as a "stamp of legitimacy." As a result, the fallacy that peer review guarantees the scientific validity of the material unfortunately remains widespread. The potentially disastrous consequences of this unscientific assumption for both journals and readers were pointed out forcefully in a recent Letter to the Editor in Nature .[8] This aspect of the peer review process is crying out for a closer look by sociologists, psychologists, and other scientists whose findings may well be of interest not only to editors, but also to the productivity or performance evaluators who are searching for the best criteria for rewarding certain research activities over others.

Improving the Science, the Writing, or Both?

Confusion persists over whether editing to improve the writing, reporting, or readability of texts falls under the purview of peer review -- a confusion reflected in the variety of practices across journals. It may therefore be helpful to distinguish between "reviewing interventions" (ie, review of the scientific content) and "editing interventions" (that affect the use of language but not the actual scientific content). This division allows the review process to be broken down into components that can be investigated separately (eg, blinding, types of support provided to reviewers, rewards offered for prompt or better-than-average reviewing, or various types of text editing). Focusing on specific interventions will make it easier to design studies that can be adapted to individual journal and editor practices.

For example, if a goal of peer review is to identify the "best" science, researchers should ask themselves whether this is assumed to mean:

  • Screening out unsuitable manuscripts only and eliminating them from the selection process;

  • Identifying a gradient of manuscripts potentially suitable for publication after varying degrees of revision; or

  • Skimming off the cream and discarding everything else.

Each reviewing task implies different levels of expertise in critical reading skills, and as any editor knows, reviewers' performance in this area varies widely. Cohort studies, before-after or diachronic studies, and other designs that aim to produce "humble" frequency distributions may shed light on what reviewers and editors are actually doing, and how changes in editorial policies affect these outcomes.

On the other hand, if the question we wish to answer is "How can I get reviewers to perform consistently and reliably?," then observational, descriptive methods are probably a necessary first step in the attempt to identify features in their backgrounds and personalities that influence their critical reading skills and their motivation to produce rigorous, useful reviews. Given the lack of consensus on how to recruit good reviewers or how to train reviewers to perform better, it may be worthwhile here to examine the literature on education and training methods. Management science is another potentially fruitful area to investigate for research methods, given that management methods and peer review methods share some basic features: Both seek to optimize the results obtained with the analytical skills that need to be applied to practical tasks, and both are based to a large extent on decision-making and choice-making processes.