Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
EUGENE GARFIELD: Refereeing and Peer Review. Part 1. Opinion and Conjecture on the Effectiveness of Refereeing
7 GARFIELD: REFEREEING AND PEER REVIEW, PART 1 Research Council of Canada (NRCC), and editor-in-chief, NRCC Research Journals. "The merits of this system," he writes, "are that it usually provides at least one solid [report], that the two [referees] can be checked against each other, and that one referee may cover points that the other missed." 2 3 But Harnad notes that, for many journals, the "number of referees [selected for a manuscript] is an empirical matter requiring research." 2 1 BBS uses five to eight referees per paper. In Hamad's experience, such a sample is more likely to produce a balanced review. 2 4 Along with the manuscript, referees generally receive a list of instructions and a form for comments and recommendations. Routinely, referees respond within a few weeks, recommending either publication or rejection or requesting modifications; they often include specific comments for both the author and the editor. A paper is most likely to be accepted, according to Michael Gordon, research associate, Primary Communications Research Centre, University of Leicester, UK, when the referees agree that it meets three criteria. 2 5 (p. 6-8) First, it should be sound. The authorfs) should have employed reliable research techniques, drawn valid conclusions, and committed no flaws of logic. It should also be original, in the sense that ii.. Endings have never before been published. Finally, it should be significant , meaning that it should contain some new perspective or observation of potential importance. 2 5 (p. 6-8) Of course, published articles meet these criteria in varying degrees. Referees do not always agree with one another, and some authors take this as evidence that the system is unreliable or capricious. But disagreement is at the heart of scientific inquiry. Hamad says that "the current and vital ongoing aspect of science consists of an active and often heated interaction of data, ideas, and minds, in a process one might call 'creative disagreement.' " 2 6 Moreover, reviewer disagreements are not simply shrugged off; editors generally resolve each dispute on an individual basis. Gordon described some of the options open to editors for dealing with these conflicts. 2 5 (p. 20-5) When reviewer disagreements are mild, for example, editors may rely on their own judgment to resolve them —with, perhaps, some communication with the author. 2 5 (p. 21) When differences are profound, editors may reject the paper without further reviewing or they may send the manuscript out for review once again, together with the comments of the disputing referees. Editors may also ask the author to respond to the referees' observations. After the "arbitrating" referee(s) and the author have reported, editors should be in a better position to make a final judgment. When authors take exception to referees' comments and provide editors with a point-by-point refutation, editors often follow a procedure similar to the one just outlined for adjudicating disputes between referees. 2 5 (p. 22-5) Research, Pseudo-Research, or Non-Research? The results of our literature search for this essay support the view that refereeing is an issue clouded with subjectivity and emotionalism —at least for a vocal minority. The dominant vehicle of discussion in the debate about the effectiveness of refereeing has been editorials and correspondence. Some contain incisive discussions, but with little or no empirical evidence to support what amounts to a litany of opinion and anecdote. Indeed, in an endeavor such as science, which depends on dispassionate logic and systematic evidence for much of its credibility, the dearth of rigorous thinking and hard data in the correspondence of many who are critical of refereeing is remarkable. Of the relative-
