Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

57 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW empirical papers that present controversial findings, we conducted a survey of 20 current or recent editors of American Psychological Association (APA) journals. The two-page questionnaire, together with a stamped, self-addressed return envelope, was mailed out in March 1990. We followed up with phone calls 10 days after the mailing. Replies were received from 16 of the 20 editors. One question asked: "To the best of your memory, during the last two years of your tenure as editor of an APA journal, did your journal publish one or more papers that were considered to be both controversial and empirical? (That is, papers that presented empirical evidence contradicting the prevailing wisdom.)" Seven editors could recall none.

2 Four said "yes" and indicated that there was one paper. Three editors replied that there was at least one. Two said that they published several such papers. It seems that controversial empirical papers do get published, but infrequently. Almost half the editors could not recall publishing such papers \n the past two years. We then asked about the peer review for the one published controversial empirical paper that they remembered most clearly. Hie question was worded: "How did the reviewers respond to this paper?" A five-point scale from "unanimously accepted" to "unanimously rejected" was provided, as well as a "don't recall" option. One of the nine respondents to this question reported unanimous acceptance, three reported "majority in favor," four reported "even split," and one answered "don't recall." In response to a question on this published paper's contribution to the discipline, one editor said "not important" four said "somewhat important," and four selected the highest rating, "important." The editors were also asked if they had rejected any papers that were controversial and empirical. Six of the editors stated that they did not receive such papers, and four said they could not recall any. The six editors who rejected papers with controversial findings did so, they said, because of poor methodology and poor supporting arguments. Of the rejected papers that the editors "remembered most clearly," only one was "unanimously rejected;" a "majority not in favor" was reported for two, an "even split" for two, and a "majority in favor" for one. Three papers were rated as "not important," and three as "somewhat important." These results suggest that one can get reviewer agreement on controversial empirical papers. Moreover, most of these papers are published without high levels of reviewer agreement. Apparently, editors do not rely solely on reviewer agreement. It is interesting that our survey found only two instances of unanimous reviewer agreement for empirical papers with controversial findings. In one case, the recommendation was to reject. In the other, it was to accept. In the case of the accepted manuscript, it should be noted that the editor had invited this submission and had selected reviewers who, he said, were sympathetic to its content. Our survey indicates that some controversial empirical papers do get published, even when there is disagreement among the reviewers. The willingness of editors to publish such papers is encouraging. On the other hand, 7 of 16 editors could recall no instances of publishing controversial empirical findings. Consequently, we consider some strategies to increase the odds of publishing this type of paper in the next section. Possible solutions. Some methods that are currently used by journals should help. 1. Some journals' editorial policies allow the author to submit a list of possible referees, one of whom would be selected. 2. Items can be included on structured rating sheets so that reviewers rate the extent to which the findings are controversial. Editors can then give such ratings more weight. 3. Additional reviews can be sought when papers are judged to contain controversial findings. (This strategy was used for only one of the nine published papers and for only one of the six rejected papers in our survey.) 4. Special appeal procedures may help for controversial papers. This might involve other members of the editorial board. 5. Controversial papers can be reviewed initially without revealing the findings. This procedure is currently used by the International Journal of Forecasting. It has not been used frequently but, when used, it has been beneficial. 6. Provide a section of the journal for "Controversial Findings. " The selection of an editor for such a section would indicate the journal's willingness to provide space for such studies. Unfortunately, the one application of this approach that we know (Armstrong 1982b) has produced only one submission, and the findings reported in that submission were not controversial, only the methods were. Rather than looking for agreement, it might be useful to seek reviewers to act as advocates. This advocacy system would be used for papers that are designated as containing controversial results. A paper could be so designated by the author, the editor, or a reviewer, after which special advocacy procedures would be used. This might include some of the above mentioned suggestions. In addition, one could use more reviewers in an effort to find an advocate. An advocate could insist on publication; a note could be included with the published paper so that reviewers are, in a sense, willing to stake their reputations on the paper.

3 Through this note, the readers would receive information about the nature of the acceptance. All referees could be given the opportunity to write peer commentary on the paper. This procedure would greatly increase the likelihood that important papers would be published. The increased effort given to reviewing might also improve quality control. Conclusions. Controversial empirical papers are expected to receive harsh treatment in peer review, but our survey indicates that such works occasionally get published, sometimes without much peer agreement. More can be done to encourage publication, however. We suggest ways to accomplish this, in particular, the use of an advocacy procedure that explicitly recognizes the need to promote this type of research. NOTES 1. It is not clear that the quality control function is performed well. About one-third of the papers in biomedical journals were found to contain citation errors, and one-third also incorrectly quoted findings from the literature (Evans et al. 1990). In addition, Hubbard and Armstrong (1990) found that 60% of published replications with extensions in three leading marketing science journals failed to support the original findings. 2. "Unfortunately," according to one respondent. Also, in followup phone calls, several editors expressed the desire that more such work be submitted. * 3. McNutt et al. (1990) found no differences in the quality of reviews based on whether or not they Were signed by the reviewer. Also, those who signed the reviews were more likely to recommewd acceptance. Reliability, fairness, objectivity and other inappropriate goals in peer review John C. Bailar Department ot Epidemiology and BiostaOstícs, McGill University School of Medicine, Montreal H3A 1A2, Canada The following remarks arc cast largely in terms of peer review of manuscripts for possible journal publication, but they also apply generally to peer review of grant proposals. Cicchetti consistently misses the mark. The purpose of peer review is not reliability, but to improve decisions concerning publication and funding, and these authors simply do not discuss this critical matter. Cicchetti fairly states the value of both redundancy and "creative" disagreement in peer review, but fails to acknowledge adequately that editors and grants managers choose (and should choose) reviewers for their different, complementary