Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

51 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW is the contention of some authors that their manuscripts seem to be rejected more on the basis of reviewers' subjective criteria (such as prestige of the author's affiliation) than on the basis of overall scientific merit (e.g., see Armstrong 1982b; Benwell 1979; Ceci & Peters 1984; Gordon 1977; Patterson 1969). Opposing arguments have been advanced (e.g., by Ingelfinger 1974). More recent criticisms of "blinding" manuscripts have been summarized by Ceci and Peters (1984, p. 1492): (1) an expensive publicity stunt used to placate authors but with little effect on quality, fairness, or interreferee reliability levels (Thomas 1982); (2) a process making it possible for authors to exaggerate their publication record, presumably by referring to their supposed research without having to cite author(s), journals, and publication dates, as proof of its existence (Howe 1982; Over 1982); (3) a mechanism enabling authors to leave out crucial information required for successful replication of their work (Lazarus 1982); and (4) a process that restricts the development of a constructive relationship between authors and editors (Eight APA journals 1972). Bradley (1981) also reports the results of a poll of psychologists revealing that more than 75% of them believed that the usual way authors' names and affiliations are removed from submitted manuscripts does not prevent reviewers from identifying the authors of such articles. (One consistent example of the failure of blinding occurs when names and affiliations are removed on the face sheet, but a footnote identifying the senior author and the institution at which the research was conducted is not.) The Ceci & Peters (1984) review of the literature found no sound empirical evidence for the futility of blind review. Rather, the negative beliefs seemed to rest on the anecdotal experiences of selected authors (e.g., Machol 1981). Ceci and Peters accordingly tested hypotheses about the feasibility of blind review. They randomly selected 180 reviewers for 6 psychology journals (each covering a different area); 81% agreed to participate and 73% returned usable questionnaires. The journals were: Journal of Personality and Social Psychology, Journal of Counseling Psychology, Human Learning, Developmental Psychology, Psychological Bulletin, and Psychometrika. Although the reviewers had predicted that they could correctly identify authors of manuscripts in 72% of the cases, their actual "hit rate" was only half of that (36%). Moreover, these results were not significantly affected by either the reviewers' age or the specific journal that was represented. The authors concluded: At a time when the integrity of the peer-review process is under siege, blind review would seem to be an obvious step toward regaining authors' confidence in the editorial process. If our findings from these six journals can be generalized to the 60 or so journals in the field (out of approximately 120) that routinely use blind review, including half of those published by the APA, then we have evidence that the personal identities and institutional affiliations of authors usually do not contaminate the evaluations of reviewers who are kept blind (Ceci & Peters 1984, p. 1494). Although the results of Ceci and Peters are impressive, one must first ask whether the peer-review glass is to be perceived as 64% full or 36% empty. Moreover, further research is needed to determine: (a) whether more specialized fields of inquiry would produce different results, because of the smaller numbers of scientists working on similar problems, and (b) whether blinding raises or lowers the reliability or validity of the review. Nonetheless, the importance of these findings should not be ignored. Perhaps a compromise would be optional blind reviewing, already the policy of some editors (e.g., see Adair 1981, p. 14). It would probably make sense to leave the responsibility of blinding a given manuscript to the author who makes the request, however This strategy would be designed (a) to increase the probability of successful blinding (e.g., eliminating mechanical detection errors), since the author who made the request would presumably have a vested interest in maintaining anonymity; and (b) to free valuable time for editors and their staffs. Optional anonymity, however, might stigmatize some authors (e.g., does the author have something to hide?).

9 With respect to NSF grant reviews, Cole and Cole (1981) note that initial attempts to blind such proposals compromised the integrity of the proposal in a number of instances, to the point that the "substantive content became very unclear. Moreover, since there was substantial disagreement about what was an identifying sentence, remark, or allusion, severe blinding depended heavily upon the blinder" (op. cit., p. 11). Cole and Cole (1981, p. 12) adopted a compromise blinding procedure similar to the one used by many scientific journals. From each grant they removed the title pages, relevant author references, descriptions of research facilities, direct references to prior work of the principal investigators, and other obvious identifying information. (The data presented in Table 4 indicate no obvious differences between COSPUP "blind" and "open" reviews of the same proposals whether in chemical dynamics, solid state physics, or economics.) 7.4. Revealing reviewer identity. The call for journal editors to force referees to reveal their identity has sometimes been strident and cuts across fields of scientific inquiry (e.g., the behavioral sciences, Patterson 1969; Surwillo 1986; the medical sciences, DeBakey & DeBakey 1976; Ingelfinger 1975; Margulis 1977; Stumpf 1980; and Wright 1970; and the physical sciences, Robertson 1976). One author asks: "Why should the wish to publish a scientific paper expose one to an assassin more completely protected than members of the infamous society, the Mafia?" (Wright 1970, p. 404). Despite the legitimate concern about possible unfairness in anonymous reviews, one needs to ask what effect open review might have on the younger reviewer who legitimately criticizes the work of an established titan in the field. What protection does such a reviewer have against possible retribution? In the reverse situation, the well-established critic would be less likely to suffer retribution from his less well known critic (e.g., see Cicchetti 1982; Scarr 1982). Consistent with the view of Armstrong (1982b), a compromise solution appears appropriate. Referees ought to be encouraged to reveal their identities, but only if they so choose. Moreover, two levels of anonymity should be available as alternatives: anonymity from the