Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
98 CICHETTI: THE RELIABII .ITY OF PEER REVIEW size of N. What this means is that any difference between study samples, no matter how trivial, will produce statistical significance, or a so-called "positive" result. Cohen (1988) addresses this issue directly by suggesting, in the broader context of power analysis, that: (a) as a general rule, one adopt a Type II error rate of .20, thereby producing power of .80 (i.e., power = [1 — beta error]. Because Type I (alpha error rate) is often set at the nominal or conventional p level of .05, the adoption of this strategy means, literally, that one considers a Type I error to be of the order of four times as serious as a Type II error; and (b) the substantive, practical, or clinical meaning of the group difference, above and beyond its level of statistical significance should interpret .15 as a SMALL effect, .30 as a MEDIUM effect, and .50 as a LARGE effect. (See section 1.2 of this Response to Eckbergfor the adoption of analogous guidelines for interpreting the practical significance of kappa, R,, and R values.) In the remaining parts of this section of the Response, I examine commentators' reactions to a number of further specific suggestions for improving the quality of peer reviews for both manuscript and grant submissions. These include: the role of multiple reviewers; using author anofiymity or "blind" review; revealing reviewer identity; author review of referees; rewarding referee contributions; allowing authors multiple manuscript submissions; developing peer review appeals systems; and training reviewers. 4.2. The role of multiple reviewers. This strategy received widespread endorsement from those commentators who voiced an opinion (i.e. , see specifically Cohen, Crandall, Greene, Hargens, Kraemer, Marsh & Ball, Strickerand Zentall). On the other hand, Colman was opposed to the use of multiple reviewers, which he felt would: (a) produce "social loafing," or a lessening in reviewer effort; (b) encourage a "diffusion of responsibility "; and (c) increase substantially the workload of already overburdened referees. I believe that the "social loafing" phenomenon, to the extent that it might exist among multiple peer reviewers, can be greatly attenuated or even eliminated by appropriately rewarding useful and thoughtful reviews, while eliminating reviewers who consistently produce biased and poorly reasoned reviews. I am not so concerned about the increase in reviewer workload for the reasons given in section 7.2 of the target article, namely, that there are pools of potential referees large enough to make multiple reviews possible across disciplines (see also the relevant remarks of Fletcher in this regard). 4 .3. Using author anonymity or "blind" review. A preference is expressed by Kraemer and Lock for author anonymity (blind reviews) as a strategy for improving the overall quality of peer review. Lock, Colman, and Greene mention the nonfeasibility of the practice for peer review of grants because it would minimize or eliminate the important role of the author's research "track record" in deciding on the merits of the proposed research. Zentall feels that voluntary blinding might defeat its own purpose because those most likely to benefit from their past record of research accomplishments would be the least likely to use the process. Bailar claims "substantial anecdotal evidence" to support the notion that reviewers evaluate more accurately the strong and weak qualities of a given manuscript submission when they are not blinded to authors. Lock presents some recent empirical data on this issue, which conflicts with Bailar's conjectures. Specifically, McNutt et al.'s (1990) report that blinding proved successful for 76% of reviewers, and that it also resulted in a 21% improvement in the overall quality of the reviews. On a 5 point scale for assessing the quality of peer review (in which 1 = very poor and 5 = excellent) the mean "summary grade" was significantly higher (p = .007) for the blinded over the nonblinded evaluations of the same manuscripts. Moreover, the difference in median grade was a full point. Blind reviews had a median quality of review of 4, whereas the nonblind reviews of the same submissions showed a median value of 3. Finally, based on an intriguing study by Garfunkel et al. (1990) Lock notes the urgency of studying the effects of blinding on editors themselves. 4.4. Revealing reviewer Identity. It is argued by Adams that forcing reviewers to sign their evaluations would result in more constructive criticism of an author's work. Rourke concurs and predicts that the freedom of information movement will ultimately force journals to adopt the policy of signed reviews. On the opposite side of the ledger, Kraemer is opposed to signed reviews. Consistent with the position I endorsed (sect. 7.4, target article), she opts instead for a voluntary decision for whether or not to sign. Zentall and Greene are also opposed to signed reviews, and for somewhat similar reasons. Zentall predicts that the fear of retribution would significantly reduce the likelihood of individual participation in the peer review process. Greene argues for the continued need to prevent the peer review of grants from becoming "personalized." 4.5. Author review ot referees. The responsibility of detecting and discarding poor quality peer reviews is placed directly on the editor by Kraemer. Similarly, Colman enjoins the editors periodically to "solicit" authors' evaluations of reviewers' criticisms, reviewers' replies to authors' criticisms (as required) and possibly to invoke the aid of an independent judge (or "arbiter") until a "fair" editorial decision occurs. Consistent with these views, Kiesler's "wise" editor would identify and ignore biased reviews. With respect to peer review of grants, Greene finds author review a useful screening procedure whose implementation in the Department of Veterans Affairs (DVA) will be considered as an adjunct to the present policy of inviting authors to submit the names of acceptable and nonacceptable reviewers. I agree with these recommendations, which complement those suggested in section 7.4 of the target article. 4.6. Rewarding referee contributions. It is suggested by Adams that referees be rewarded for providing quality peer reviews, without creating "undue bias" or impinging on the "freedom of scientists." I strongly agree. 1 would also add that whenever possible reviewers who consistently provide high quality evaluations (well reasoned, well documented, balanced) should be invited to serve as consulting editors, members of the boards of journals for which they review, or even associate or full editors, as appropriate. They should also be asked to
