Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

42 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW Table 1. Levels of interreferee agreement (inlraclass correlations) on various criteria applied to the evaluation of manuscripts and extended abstracts A. For manuscripts submitted to the "Journal of Abnormal Psychology" (1973-78) Evaluative Criterion Number of Manuscripts Level of Interreferee Agreement Importance 661 .23 Design 610 .32 Data Analysis 611 .22 Style and Organization 666 .22 Literature Review 660 . 26 Reader Interest 663 .19 Succinctness 658 . 30 B. For manuscripts submitted to the "Journal of Personality and Social Psychology" (Scott 1974) Importance 312 .28 Design and Analysis 574 .19 Reader Interest 312 .07 Style and Organization 574 . 25 Succinctness 574 . 31 Literature Review 458 .37 C. For manuscripts submitted to the "British Medical Journal" (1979) Importance 707 . 33 Scientific Reliability 707 . 22 Originality 707 . 21 Suitability 707 . 22 D. For abstracts submitted to "American Association for the Study of Liver Disease" (Cicchetti & Conn 1976) No. of Levels of Interreferee Composite Evaluative Criterion Abstracts Agreement Agreement A vs. B Aos. C B vs. C Importance 77 .22 .15 .31 .24 Design and Execution 77 . 28 . 21 .34 . 29 Originality 77 . 37 . 21 .32 . 30 Note: With the exception of "Reader Interest," Section B, all values arc statistically significant at or beyond the .05 level. range from essentially 0 (R = .07 for level of "reader interest") to "highs" of .37 for both "originality" and "literature reviews," which, according to guidelines representing levels of practical significance, would be considered poor (e.g., Cicchetti & Sparrow 1981; Fleiss 1981). 4.3. Reliability of manuscript reviews: Behavioral science. It can be seen from the data presented in Table 2A that the levels of chance-corrected interreviewer agreement (kappa or R, values) range between . 19 (Journal of Abnormal Psychology - Cicchetti it Eron 1979) and .54 (American Psychologist - Cicchetti 1980; Scarr & Weber 1978). It should be noted that the reviews for the American Psychologist were based on a very small number of manuscripts (N = 87), and that the level of peer-reviewer reliability could not be successfully replicated in a followup peer-review study (Cicchetti, unpublished), in which the R, value dropped from .54 (fair agreement, see Cicchetti & Sparrow 1981) to .38 (poor agreement, Cicchetti & Sparrow op. cit.

, p. 133). 4.4. Reliability of manuscript and abstract reviews: Medicine. The data based on peer-reviewer chance-corrected reliability levels for medical journals (Table 2B) are very similar to those just presented for peer reviews of behavioral science manuscripts, namely a range between .31 (Physiological Zoology - Hargens & Herting 1990a) to .37 (a major medical subspecialty journal - Cicchetti & Conn 1978). With respect to peer review of abstracts submitted to professional meetings, Cicchetti and Conn (1976) reported very similar levels of chance-corrected agreement for ratings of overall scientific merit (i.e., between . 16 and .33, with corresponding p values between .10 and .01). 4.5. Reliability of manuscript reviews: Physical sciences. As far as we are aware, no formal studies of the reliability of peer review have been undertaken for manuscript or abstract submissions to journals in the physical sciences, yet there is a prevailing belief that levels of interreferee agreement are substantially higher for journals in the physical sciences than in other areas studied. This conclu-