Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
95 CICHETTI: THE RELIABII .ITY OF PEER REVIEW viewer recommendations are the major factor in the editor's decision to accept or reject (Bakanic et al. 1987; Hargens 1988). Next is an attempt to broaden our knowledge base in this important area of peer review research. The data is based on the total number of submissions to JAP between 1973 and 1977 (1,698 manuscripts). These could be classified as follows: 15 (0,9%) were withdrawn before an editorial decision could be made; 14 (0.8%) were solicited; 384 (22.6%) were reviewed by the editor alone; 175 (10.3%) were reviewed by a single referee (other than the editor); 996 (58.7%) were evaluated by two independent reviewers; 112 (6.6%) received three independent reviews; and 2 (0.1%) received four independent reviews. As reported in section 2.3 of this Response, 86.7% (333 of 384 manuscripts) were reviewed and summarily rejected by the editor alone on the basis of very poor quality, inappropriateness for the readership of the journal, or on both accounts. All 14 of the solicited manuscripts were accepted for publication. Of the two manuscripts receiving four reviews, one was accepted, the other rejected. The fate of the remaining manuscripts, namely, those with a single review, two reviews, or three reviews follows. 3.2. The editor's use ot single reviews: "Go with the flow." Table 4 shows that of the 175 manuscripts sent to a single reviewer, 58 (33.1%) were accepted by the editor and 117 (66.9%) were rejected. The full set of data (Part A ofTable 4) indicates that, as a general rule, the editor's final decision closely parallels single reviewer recommendations. Part B of the table indicates that when the reviewer recommended that the manuscript be accepted ("as is" or "subject to revision") there was an 85% likelihood of acceptance. Analogously, when the reviewer recommended either resubmission or rejection, there was a 90% probability of rejection. An inspection of discrepancies between reviewer recommendations and editorial decisions indicated that the editor was no more likely to reject manuscripts receiving an "accept/as is" or "accept/revise" recommendation (8/54 = 14.8%) than to accept manuscripts receiving a "resubmit" or "reject" recommendation (12/121 = 9.9%). Applying McNemar's (1947) statistic for correlated proportions produced a chi square(d) value of zero-order significance, that is, 0.45. 3.3. The editor's use of two reviews: "Go with the low." Results are presented in Table 5 for those 996 manuscripts receiving two reviews during the period 19731977. The data can easily be understood if one considers that: (1) Manuscripts receiving a joint reviewer recommendation of "Resubmit" had a 27% probability of being accepted for publication, which is indistinguishable from the base rate journal acceptance rate of 28%; (2) those manuscripts receiving two reviewer votes for acceptance or a split between acceptance and resubmission had a 72% probability of being published, as compared to the 72% baseline rejection rate of the journal; and (3) the remaining 635 manuscripts (65%) had only a 5.5% probability of being accepted for publication, a rate more than five times less than the journal baseline acceptance rate of 28%. These findings also have cross-disciplinary implications. Specifically, Lock (1985, pp. 20-21) presented analogous data for 282 articles, or 50% of the 564 articles that were submitted to the medical journal Thorax and evaluated independently by two referees. Thirty-eight percent (or 107) of the manuscripts received a unanimous reviewer recommendation for acceptance. All of them were accepted by the editor. The reviewers were in agreement that an additional 38% (or 107) manuscripts should be rejected. The editor rejected all these submissions. Twenty-four percent (or 68) of the submissions received a split-decision, with one reviewer recommending "accept" and the other "reject. " The editor accepted Table 4. The fate of Journal of Abnormal Psychology submissions receiving a single editorial review (1973-1977) A. Considering each reviewer recommendation Reviewer Number of Editorial decision Percentage Recommendation Manuscripts Accept Reject Accepted 1 = Accept/As Is 24 22 2 91.7 2 = Accept/Revise 30 24 6 80.0 3 = Resubmit 26 11 15 42.3 4 = Reject 95 1 94 01.1 Total 175 58 117 33.1 B. Considering 1-2 — Accept; 3-4 = Reject Reviewer Number of Editorial decision Percentage Recommendation Manuscripts Accept Reject Accepted (1-2) = Accept 54 46 8 85.2 (3-4) = Reject 121 12 109 90.1 Total 175 58 117
