Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
47 CICHETTI: THE RELIABII .ITY OF PEER REVIEW Table 5. Relationships among chance-corrected reviewer agreement levels (R,) and agreement levels for the acceptance and rejection of manuscripts submitted to major journals in behavioral and medical science Journal R, Acceptance Rejection Combined X , P "Journal Abnormal Psychology" .14 44% (462) 70% (857) 61% (1319) 83.99 .00001 Untitled Medical Specialty Journal .26 50% (289) 76% (577) 67% (866) 57.895 .00001 "Developmental Review" .27 52% (25) 74% (47) 67% (72) 3.413 .06 "American Psychologist" .45 66% (62) 78% (97) 74% (159) 2.7315 .10 Note: R, values are all statistically significant at beyond the .01 level. Table 6. Relationships among NSF and COSPUP chance-corrected agreement levels (R,) and agreement levels on htgh-(40-50) and lcnc-( 10-39) rated grant proposals in 3 areas of research specialisation Agreement on Proposals With: Area of Specialization Rj High Ratings Low Ratings All Proposals X' P Combined: .32 54% (52) 76% (98) 68% (150) 5.69 .01 chemical dynamics .16 (NS) 41% (17) 70% (33) 60% (50) 2.71 .10 solid state physics .34 60% (20) 73% (30) 68% (50) 0.46 NS economics 44 60% (15) 83% (35) 76% (50) 1.88 NS Note: The p values of R, are at greater than the .10 level for chemical dynamics; at beyond the .01 level for solid state physics; and at beyond .001 for economics. the lower chi square(d) values tend to be associated with those journals processing relatively small numbers of manuscripts. These results parallel both those reported earlier by Lock (1985) and those completed more recently for the American Sociological Review, Physiological Zoology, and Law and Society Review (Hargens & Herting 1990a). The results in Table 6 for NSF and COSPUP grant reviews closely parallel those just reported for manuscript reviews. Specifically, reviewer agreement levels for proposals with low NSF and COSPUP ratings (10-39) were consistently higher (70%-83%) than agreement levels for those with high (40-50) ratings (41%-60% agreement). Though the numbers were too small to be statistically significant for a given specialty area, the combined Table 6 (row 1) data indicate significantly more interexaminer agreement on the 98 proposals with low ratings (76%) than on the 52 proposals with high ratings (54%). Thus, on the basis of available data, it becomes clear for the first time that reviewers are indeed substantially more in agreement on which scientific documents to reject than on which to accept. Consistent with these data, it is noteworthy that editorial decisions for general journals (e.g., Journal of Abnormal Psychology ) seem to give considerably more weight to referee consensus on rejection than to referee consensus on acceptance . Of the 203 manuscripts (of 1,316 submitted) for which independent reviewers both recommended acceptance, 28 (or 14%) were rejected. In comparison, only 28 (or 5%) of the 600 manuscripts that the reviewers agreed should be rejected were in fact accepted. Here, chi square(d), 1 df = 17.94 (p < .00001). In other words, the editor was almost three times more likely to reject a manuscript that reviewers agreed should be accepted than to accept a manuscript that the reviewers agreed should be rejected. 8 In attempting to interpret this important phenomenon further, however, one must consider a number of other issues. 5. Issues of interpretation 5.1. Caveat #1: "One swallow does not a summer make." Although the findings are consistent across the types of scientific document analyzed (manuscripts, abstracts, grant proposals) and across areas of investigation (behavioral science: psychology, sociology; medicine: genera] and specialty areas; economics; and physical sciences: chemical dynamics, solid state physics), one must keep in mind that the documents investigated do not represent a broad cross-section of existing materials. Rather, the studies of peer review have been focused on a relatively small number of prestigious journals, professional organizations, and grant-reviewing agencies. More research is clearly needed to test the generality of the findings to date. For example, as correctly noted by Lock (1985), the journals investigated tend to be general ones that all share very high rejection rates. With such a focus in mind, reviewers may be more interested in determining what is wrong with a particular submission than in documenting some of its more positive attributes. It has been conjectured by workers in the field that journals that have much higher acceptance rates, such as Physical Review (between 73% and 81%, between 1969 and 1986) may display the reverse phenomenon, or "when in doubt, accept" (Zuckerman & Merton 1971, p. 78). With respect to NSF grants, Cole and colleagues reported that at the time their study was undertaken, approximately one out of every two applicants was eventually funded. Given the current relative scarcity of NSF (and other) funds, what impact will this have on
