Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

77 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW analysis, endpoints, findings, and conclusions. Thus, in the future, after peer review of the full article, editors might like to suggest that some reports be printed in structured abstract form with the substantive report incorporated into an on-line database. Finally, in the past editors in many disciplines have been able to sleep regardless of mistaken decisions because of the concept of Western plurality: What gets rejected by the Lancet will be published by the British Medical Journal or Gut. Cicchetti shows disquietingly that this is not so for some journals in some disciplines and for research grant applications in any of them. We urgently need retrospective and prospective studies on these findings: Did the rejected ideas stand up in the light of history? Was the rejection the result of lack of rigour by the researcher, even though the original ideas were sound? Were they ever studied by anybody else? And should society see that some journal space/money is reserved for "zany" ideas in all disciplines as in David Horrobin's journal Medical hypotheses

? Some^of these studies need not be too time consuming and might make fitting contributions to the 1992 Second World Conference on Peer Review. (Details are available from Drummond Rennie, Journal of the American Medical Association, 535 N. Dearborn St., Chicago, IL 60610.) Justice, efficiency and epistemology in the peer review of scientific manuscripts Michael J. Mahoney Department of Psychology, University of North Texas, Denton, TX 76203 Cicchetti has written a valuable and comprehensive critique of the reliability of peer review for journal manuscripts and grant applications. Besides addressing some of the many subtleties, complexities, and practical issues involved in peer review, the author has identified an important and well-replicated phenomenon in this area, namely, "that reviewers are indeed substantially more in agreement on which scientific documents to reject than on which to accept.

" This may be a heartening conclusion for those evolutionary epistemologists who view selection processes as primarily negative, but the authors offer a provocative discussion of caveats in the interpretation of this pattern. I agree that the high rate of rejection for grant proposals is of greater concern than the rejection of journal manuscripts in the social and behavioral sciences. In their recent report, the National Research Council (1988) stated that basic research in these sciences merits a 30% increase in funding over current levels. Unfortunately, federal funding for such research has been declining sharply since 1983. In fact, although federal support for research in the other sciences has increased by 36% in the last 15 years (1972 to 1987), federal funding for basic research in the social and behavioral sciences has been reduced by 25% during that same interval. Needless to say, those psychologists who recognize the need for a scientific basis for their activities will now have to work even harder to reverse the trend of declining support for their research. I also agree that allowing authors to engage in multiple manuscript submissions is not a viable solution to the problem of high rejection rates from journals. Indeed, the project recently reported by Epstein (1990) illustrates some of the problems of this practice as a research strategy, let alone as standard practice for scientific authors. Epstein apparently plagiarized an article and submitted it to 146 professional journals in (or related to) the field of social work. His methodology and quantitative results were very weak, and yet he offered an interpretation of his study that was harshly critical of the professionals who had unwittingly invested perhaps a total of 1,000 hours in his project. With other scientists, I believe that studies of peer review are a priority for future science studies. In this regard, it is reassuring that the American Medical Association sponsored a special conference on the topic (Rennie 1986). "I do not believe, however, that such a compelling priority justifies violations of human rights and the professional codes of ethical conduct developed to protect them. The questions of how, when, and why we 'draw the lines' demarcating ethical and unethical conduct will remain with us, of course, and we can only hope that their challenges will teach us some important lessons about ourselves and our methods in the process (Mahoney 1990, p. 54)." Reflections on the peer review process Herbert W. Marsh* and Samuel Ball

b *School of Education, and bFaculty of Education, University of Sydney, Sydney NSW 2006 Australia The peer review process is one of the most highly regarded and frequently used procedures for evaluating the academic merit of academic manuscripts, grant proposals, tenure/promotion applications, and academic monographs and textbooks. Hence, peer review is of utmost importance to the academic community and we welcome the comprehensive review by Cicchetti. It brings together discussions of theoretical issues, methodological/statistical concerns, a diversity of empirical studies, and practical suggestions for the interpretation and application of the peer-review process. Given the scope of his review, we will limit ourselves to comments on a few specific aspects of the peer review process for academic journals. Reliability of the editor's decision. Marsh and Ball (1981; 1989) noted that low single-reviewer reliabilities should not be confused with the reliability of the decision of the review process. First, the reliability of the mean response by multiple reviewers depends on the number of reviewers; a single-reviewer reliability of .36 results in a reliability of .53 for 2 reviewers, .65 for 3, and 0.69 for 4 (using the Spearman-Brown equation). Second, the editor serves as an implicit additional reviewer, further contributing to the reliability of the final decision. Third, the editor's decision is typically based on additional factors not considered in single-reviewer reliability estimates, such as the detailed written comments provided by reviewers, author responses to reviewer criticisms in revisions and/or separate correspondence, and sometimes further reviews of the revised manuscript. Fourth, the exclusion of manuscripts judged to be grossly inappropriate further attenuates reliability estimates, in that agreement among reviewers would probably have been best for these manuscripts. Thus, the editor's decision is likely to be substantially more reliable than that of the single reviewer. Policy decisions and practices that operationally define the peer-review process. A general framework for the review of academic manuscripts is common to most journals. Editors, chosen for their broad expertise, generally screen manuscripts for appropriateness and then assign them to one or more reviewers with particular expertise in relevant areas. The reviewers are asked to provide written critiques, ratings, or recommendations on the advisability of publication. Editors, relying on these reviews and their own appraisals of the manuscripts, decide to accept (perhaps, subject to revision) or reject the manuscripts, or to seek further review. Many details, however, are left to the discretion of the editor including: (1) the criteria to be used by the reviewers; (2) the form of the review (ratings, written critiques, etc.); (3) the number of reviewers; (4) how reviewers are selected; (5) whether authors and reviewers are anonymous; (6) what is done with divergent or inconclusive reviews; (7) the extent to which the editor's decisions are