Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
94 CICHETTI: THE RELIABII .ITY OF PEER REVIEW ence reviewing manuscripts and grants over a broad spectrum of disciplines (behavioral science, medicine, biostatistics), as well as my activities on editorial hoards and grant review committees, have indicated to me that when the peer review process is working properly (i.e., reviewers are selected for their varying areas of competence and they take their reviews seriously) it is not unusual to find high levels of agreement on at least the final recommendation, if not on a number of manuscript or grant attributes as well. To clarify the relevance of Bailar's example, there is no a priori reason to believe that the cardiologist, pharmacologist, and statistician should not agree that a given clinical trial evaluating a new hypertensive drug is or is not worth supporting simply because each represents a different area of expertise. They would surely agree more than alternative reviewers selected randomly. The major disagreements I have experienced (or witnessed) among reviewers (whether for manuscripts or grants) have occurred primarily because a proper match was not made between submitters and reviewers. Although the disagreement can be occasioned by a number of factors, not least among them is a lack of sufficient expertise (or even bias) on the part of one or more of the reviewers. So, in response to both Kiesler and Bailar, I would emphasize that the proper selection of reviewers to evaluate a given submission, should, in the long run, increase both the reliability and validity of the peer review process. The sine qua non necessity of obtaining a balanced set of reviews (for both manuscript and grant submissions) is widely accepted by editors, granting officials, reviewers and authors alike. See, for example the additional comments on this important issue by Adams, Eckberg, Greene, Hargens, Kraemer, Roediger, and Strieker, as well as the recently published work of Fiske and Fogg (1990). The next major issue I discuss concerns how a given editor or program director uses the information obtained from peer reviews - quite apart from issues of reliability (or validity) - to make publication or funding decisions. 3. Use of peer reviews to improve editorial/funding decisions 3.1. The editor as final arbiter. A general concern is expressed by Fletcher about how editors and granting officials use unreliable reviewer recommendations to arrive at publication or funding decisions. Strieker talks about the importance of the "active" editor who judiciously "weighs" the information, provided by reviewers, to arrive at a thoughtful publication decision. Similarly, Rourke speaks of "fair," "judicious," and "experienced" editorial practices. The reader will also recall Kiesler's concept of the "wise" editor. These descriptions, in turn, are similar to Bailar's informative notion of the editors' integration of their own knowledge with that provided by the additional "wisdom" of members of the editorial board, as well as "special consultants," as required. Lock (see also Lock 1985) proposes a "hanging committee" to examine and help resolve questions about those manuscripts receiving "gray area" or split-review recommendations. Both Bailar and Crandall speak of the need for editors to eschew a majority-vote-of-reviewers' option, by exercising their power to override the recommendations of reviewers, whenever required. Bailar states that a neglected area of the target article is the realization that an editor's decision is based not only on the overall scientific (or "technical") merit of a given submission but also on such manuscript attributes as "originality," "importance to readership," or "succinctness." I am somewhat puzzled at how an editor accomplishes this important goal, given the known unreliability of such attributes, that is, the data shown in Table 1. Bailar's further elucidation of how he was able to accomplish this objective for J NCI would make an important contribution to the field of peer review. Tyrer discusses the important role of the editor or chairman of a granting agency in which "executive decision" is used to resolve reviewer disagreements. In a slightly different context, Armstrong & Hubbard note that at least some innovative research is published by editors of the American Psychological Association (APA) journals, despite the relatively low levels of reviewer agreement on submissions describing such valuable research. Fuller expresses concern that such high quality research will frequently be published in relatively few access journals. In a more general sense, Bailar takes the position that the target article did not adequately "pound home" the relatively major role of the editor or granting official in the entire peer review process relative to what he feels is the more minor role played by the reviewers (merely the providers of "relevant information"). Unfortunately, in so doing, the views of Bailar (consistent with those of Kiesler) create a purely artificial distinction among authors, reviewers, editors, and consumers of submitted and published papers. As noted earlier (Cicchetti 1982, p. 21), "one of the most persistent problems we still face appears to be the false dichotomy we have tended to create between those who evaluate research and those who are being evaluated. Both derive from the same research species. "Theprovocativeandimaginativecommentary by Kraemer is consistent with this view. She speaks eloquently of the conflicting roles each of us is called on to play ("submittors, " "reviewers, " "consumers" of scientific papers) and the fundamentally different standards we might invoke, depending on which peer role we are assuming at a given moment. Her emphasis on an "objective," "dispassionate," and "quantitative" approach to the study of peer review as the only hope of identifying and correcting the many shortcomings in the peer review process is to be taken seriously. Toward this important goal, it is almost axiomatic that for science to continue to operate, it requires the imagination and talent of authors, dispassionate and sensitive editors and granting officials, and, finally, interested readers (or consumers) of the published research findings, so that the cycle can continue anew. In this basic and comprehensive sense I disagree with the more narrow-focus views of both Bailar and Kiesler. Though Bailar seems to be unaware of it, data were provided in the target article, showing the positive relationship between peer reviewer recommendations and the publication decisions made for more than 1,300 manuscripts submitted between 1973 and 1978 to the general focus Journal of Abnormal Psychology (JAP). It was further noted that the results were consistent with data deriving from reviews of both the Sociological Review and the Physical Review, which indicate that re-
