Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
96 CICHETTI: THE RELIABII .ITY OF PEER REVIEW Table 5. The fate of Journal of Abnormal Psychology submissions receiving two editorial reviews (1973-1977) A. Considering "Accept/As Is" and "Accept/Revise" as "Accept" Reviewer Number of Editorial Decision Percentage Recommendation Manuscripts Accept Reject Accepted Accept-Accept 181 159 22 87.8 Accept-Resubmit 150 79 71 52.7 Resubmit-Resubmit 30 8 22 26.7 Accept-Reject 195 28 167 14.4 Resubmit-Reject 162 6 156 03.7 Reject-Reject 278 1 277 00.4 Total 996 281 715 28.2 B. Considering 1 — Accept; 2-3 = Reject Reviewer Number of Editorial decision Percentage Recommendation Manuscripts Accept Reject Agree Disagree Accept 353 212 141 60.1% 39.9% Reject 643 69 574 89.3% 10.7% Total 996 281 715 2% (or 6) such manuscripts and rejected the remaining 22% (or 62) manuscripts. Thus, there was a 62/6 or more than tenfold probability that an editor would reject rather than accept a manuscript receiving mixed reviews. In summary, at least for the general focus journals examined thus far, when editors are faced with splitdecisions, they tend overwhelmingly to go with the lower of the two reviewer recommendations. 3.4. The editor's use ot three reviews; "Go with the mode." Of the 1698 (6.6%) submissions to JAP (19731977), 112 received three reviews. As shown in Table 6, the disposition of these manuscripts is again closely related to reviewer recommendations. Thus, all 5 manuscripts receiving unanimous acceptance votes were accepted for publication. Analogously, the 9 submissions receiving unanimous rejection votes were rejected. The disposition of the remaining 98 (or 87.5%) manuscripts is best understood by the editor's general adoption of majority rule or applying the formula "go with the mode." Thus, 24 of 33 (or 73%) of those submissions receiving 2 "accept" recommendations were accepted, whereas 80% (or 52/65) submissions receiving two "reject" votes were rejected. Application of the McNemar test of correlated proportions indicated no significant difference favoring either the editor's acceptance of these articles (20%) with majority rejection votes or his rejection of those articles (17%) with majority acceptance votes (McNemar's chi square(d), corrected, 1 df = .41, or of zero order significance). I am unaware of comparable studies on manuscripts submitted to medical journals. Bailar noted that it was not unusual, in his role as editor of JNCI (1974-1980), however, either to reject manuscripts receiving three positive reviews or to publish submissions receiving three negative reviews. Given the importance of this phenomenon, I would invite Bailar to publish these data, because they contrast so sharply with what I have presented here for a prestigious behavioral science journal. It would be important to know; (a) precisely how frequently the phenomenon occurred; and (b) in what important respects the targeted manuscripts were dissimilar from those that were less problematic. Perhaps Bailar could provide this information in a BBS Continuing Commentary. In the field of physics, recall that journal editors in the more specific focus areas use the "single initial reviewer system"(e.g. , Hargens & Herting 1990; Lock 1985, p. 20) and so would tend not to have much data on the fate of manuscripts receiving three reviews. Although it is equally clear that the more general focus areas of physics often receive three reviews (or more), the extent to which an editor uses this information to make specific publication decisions is unknown . Despite this lack of specific information, there are some sparse data, deriving from the field of physics, that bear on the broader issue of how editors use information gained from referees to improve the quality of the editorial decision-making process. These data appeared in the Physical Review (PR) and Physical Review Letters (PRL ) (1987, p. 7) Annual Report for the previous year, 1986. The statements pertain to a change in editorial policy for manuscripts submitted from the community of particle theorists to the subfield section, Elementary Particles, which represents one of 10 PRL areas of specialization. (The remaining nine subfields, orPRLcontent areas, are: General Physics, Cross-Disciplinary Physics, Astrophysics and Geophysics, Condensed Matter (CM), Electricity, CM Mechanics, Plasma Physics, Optics and Fluids, Nuclear Physics, and Atoms and Molecules.) The statement of journal policy change bears quoting; In March, 1985, a new system of handling papers in the theory of particles and fields was introduced. The divisional Associate Editors were enlisted to work closely on the processing of these papers, with the
