Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

52 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW author, but not from fellow reviewers, and complete anonymity from everyone but the editor (this is BBS's current policy). Ideas similar to these would also pertain to reviewers of grant proposals. 7.5. Author review of referees. To counteract possible unfairness or incompetence in refereeing. Hall (1979, p. 798) has suggested "author review." Each author of a submitted manuscript (or grant proposal) would be given the opportunity to evaluate referees on the basis of "fairness," "carefulness," "constructiveness," or whatever other factors the author (or grant applicant) deems relevant. Such information would then be recorded, filed, and periodically reviewed by the editor (or, for that matter, the program director, or executive secretary of a granting agency). Those referees receiving repeated low ratings might then be eliminated as future peer reviewers. The process of author review is practiced by BBS. Another innovative strategy, also aimed at avoiding potential referee bias (here, Mahoney's confirmatory bias) has been used by a number of journals, including the International Journal of Forecasting (Armstrong 1982b). A referee is given a note containing information about an author's research design, methodology, and data analysis but not about the results or ensuing discussion. The referee is instructed to review the paper using the note, after which he can open a sealed envelope containing the completed manuscript. Armstrong (1982c) also recommends that authors be permitted to submit names of potential reviewers as well as those who, with reasons, should not review. All this must obviously be done in a nonbinding manner. 7.6. Rewarding referee contributions. Whereas much of the literature on manuscript peer review has tended to be highly critical of the performance record of referees, the opposite sentiment has also been expressed, especially for those referees who consistently write "commendable" reviews (e.g., Armstrong 1982b; Hunt 1971; Smith 1977). Accordingly, ways of acknowledging commendable referee performance have been suggested (e.g., letters of thanks, with copies to members of the editorial board of the journal; publication oflists of especially good referees; acknowledgments in footnotes for substantial referee suggestions; or even an offer to publish such reviews, see Armstrong 1982b; Hunt 1971). Several journals have adopted this or similar strategies. For example, BBS invites reviewers to serve as commentators if the article is accepted for publication. 7.7. Allowing authors multlpla manuscript submissions. The injunction against authors submitting the same article to more than one journal is consistent across major journals in behavioral science as well as medicine (e.g., see the 1983 American Psychological Association (APA) Publication Manual for psychology journals; the American Sociological Association (ASA), policy for sociology journals, Peters 1976; and the policy for major medical journals, Relman 1978). Peters (1976) appears convinced that a policy of multiple submissions would generate healthy competition among journals, all vying for the same manuscript. He presents some cogent arguments against the rationale for the ASA policy statement, but there are a number of attendant problems to be considered. Referees often review for a number of different journals in the same general area of inquiry. Thus, when an author submits a manuscript simultaneously to several journals, a given referee may receive requests to review the same article from several different journal editors. What is the conscientious referee to do under these circumstances; Send the same review to each journal? Select one journal only? If so, which one? As another example, what happens when more than one journal accepts an author's paper? How can an editor intelligently organize an upcoming issue when faced with a sudden and unexpected withdrawal of a previously scheduled manuscript by an author receiving a second acceptance from a preferred competing journal (e.g., see Hughes 1976)? Also, what justifies the time and expense of needless multiple refereeing? More important, Lindsey (1978) predicts that if the model of multiple submissions "is adopted by journals, authors will not be guaranteed a careful and impartial review of their work. Rather they will be vying for the attention of editors to their work. In this competition, those with prestigious credentials will receive the closest attention. Rather than equalizing access, there is the danger that multiple submissions may have just the reverse consequence" (Lindsey 1978, pp. 110-11). Given these rather serious problems, none of which appears to have yet received satisfactory solutions, multiple submission does not seem to be a viable procedure. 7.8. Developing an author-to-edltor appeal process. In earlier sections (6.1, 6.2), we discussed a number of variables that have been cited in the literature that: (a) could be objectively identified; (b) were irrelevant to the publishability of a given manuscript; but (c) have nevertheless been used at times by editors to justify rejecting a given journal submission. There are eight such variables: 1. documentable factual errors made by reviewers (e.g., suggesting an invalid procedure in lieu of a valid state-of-the-art one developed by the author) 2. faulty data analyses (which are readily reparable) in a work that has otherwise received a quite praiseworthy reviewer and/or editorial evaluation 3. prior publication in nonrefereed conference proceedings 4. replication of previously published work 5. null or negative findings (despite very favorable reviews) or (6-8) without supporting arguments: 6. subject inappropriate 7. ideas insufficiently novel or original 8. insufficient space to accommodate the submission. With respect to the first five variables, it would seem that a carefully documented, dispassionate letter could result in 4 journal editor's honoring a request that the rejected manuscript be resubmitted to a new set of referees who would review it independently and without prior knowledge of its rejected status. Should this strategy fail in the more general areas of behavioral science or medicine, then the author can choose, in all likelihood, equally prestigious alternate journals with (as we shall later report) a reasonably high probability of acceptance.