Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
63 CICHETTI: THE RELIABII .ITY OF PEER REVIEW What should be done to Improve reviewing? Rick Crandall Journal of Social Behavior and Personality, Box 9838, San Rafael, CA 94912 The title of this commentary uses the word should rather than can because there is a professional ethical issue here that is not being faced by journals. Cicchetti's target article makes it obvious that the review process is unreliable. This conclusion should be dealt with, yet most journals blithely continue to use the process without trying to improve it. We must question the validity of the review process, and in fact the whole editorial process, for at least four reasons. First, there is no evidence to support them. The fact that very few journals are making any effort to improve the editorial process is unethical, in my opinion. (Crandall 1990). Second, low reliability can limit validity. Third, Cicchetti and numerous others have documented the existence of systematic biases in the editorial process. It is frightening that, with the lack of other data on systematic "true" variance in the review or editorial process, one must wonder whether these biases account for the little reliability we do achieve. Fourth, in most fields, the majority of papers that are turned down are ultimately published in good journals. It is clear to me that if we really wanted to, we could achieve high reviewer reliability. Although many remedies could help, the obvious one is to train reviewers. If we can train small dogs to jump up and down off moving horses at the circus, surely we can train scientists to act as reviewers, despite the complexity of the task. Thus, I believe that underlying the question of what should be done is why we haven't bothered. Although I may tend toward a moralistic interpretation of the reason, it is likely that "people" don't think better review reliability is really needed, or the payback would not justify the investment required. Our journal has a major focus on testing and improving the editorial/review process. We have some unpublished data documenting the lack of existing training for reviewers by journals. Of 76 social and behavioral sciences editors who answered the question, "Do your reviewers receive any training, besides a general instruction sheet?" only two said yes, and their training was not major. It is clear that training is totally neglected. Reviewers must be expected to learn on the job at authors' expense. I had to laugh when I saw the recent American Psychological Association announcements recruiting members of underrepresented groups to be reviewers for journals. The only qualification mentioned was that they must have published articles in peer reviewed journals, because "the experience of publishing provides a reviewer with the basis for preparing a thorough, objective evaluative review" (American Psychologist 1989). This is logically analogous to requiring people to be executed before they can become hangmen. I guess it is better than nothingl What training can be done? Some journals do some minimal training or screening of reviewers. I'm sure that editors try to weed out "bad" reviewers, so they should be gradually improving their pool. Cicchetti suggests that reviewers should be rewarded in some way. Several journals, including ours; have some explicit procedure for adding and subtracting reviewers from the published editorial panels depending on their work. The most common training approach is probably to exchange reviews after the decision so that each reviewer can learn from the other. This could be improved considerably if the editor's decision letter wer« sent to the reviewers or if they were told explicitly how good their review was and what the other reviewer did right or wrong. In other words, review the reviewsl It would be a logical next step to have training manuscripts with prototype "ideal" reviews. When you train people (or animals) to perform a task, you also have explicit training goals and criteria. These should be written down and transmitted to reviewers. It has been demonstrated that reviewers are, in fact, trainable. To use a simple, confirmed example, we have been successful in requiring reviewers to return reviews in two weeks. This was discussed and documented 20 years ago by a sociology journal editor (Rodman 1970). Another group in need of training is authors. Many manuscripts that come in to our journal would probably be better received if they were better presented. Although I have argued that efforts to get authors to "improve" (Boor 1986) amount to blaming the victim and would be better spent testing the process (Crandall 1987b), author training could improve things a bit. What may be more important than presentation variables, however, is educating all authors to the "folklore" of how to improve their chances of getting published. We have attempted to do this (Anon. 1987; Wyer et al. 1987). A number of other issues could be elaborated on in this area. I have mentioned only a few here. Elsewhere, I have suggested a number of simple standards that should be required of journals to make improvements (Crandall 1986). Among them are enlarging the pool of reviewers and editors so the excuse about being "overworked volunteers" could be eliminated, and requiring journals to make timely decisions and to commit to doing research to improve things. I have a few brief reactions to other points raised by Cicchetti. He suggests using more reviewers to increase reliability. We do that, and it also increases speed, since you don't have to wait for a slow third reviewer. Their Table 3 data on the effects of adding reviewers on acceptance rates may also be relevant here, however. Causality goes in both directions. In addition to weaker papers being sent out for more reviews, they note elsewhere that the more reviewers you have, the greater the chance that an important flaw will be caught. Unfortunately, there is also a greater chance that a negative "picky" review will encourage an editor to reject the paper. Common sense, as well as experimental evidence (Amabile 1983), tells us that when a journal (or grant) review process rejects 75% of the submissions, a reviewer will look a lot smarter erring on the side of harsh criticism than on the side of leniency. I believe that reviewers have a negative bias unless trained otherwise. A very important issue tends to be overlooked in discussions of reviewing. We have editors who are supposed to be capable of making independent decisions. With their exposure to all papers submitted, they should be "super reviewers." Yet, too many times editors may abdicate responsibility for editorial decisions. They should not Be conducting a vote and averaging reviewers opinions. They can override reviewers! They should be using the reviewers to review and making the editorial decisions themselves. Too many times, I've seen cases in which editors do not seem to behave this way. I believe that many editors do not even read the papers for which they are supposed to have editorial responsibility. If they don't read them closely, how can they be the editors? Or how can they give reviewers feedback, as discussed earlier? We need more highlighting of issues in the review process as Cicchetti et al. have done. We also need to take a broader look at the problems, and have a commitment by journals, associations, agencies, and reviewers to do a better job. I have directly challenged groups who publish journals to take responsibility and do something about these problems even more pointedly than I have here (e.g., Crandall 1987a). There has been little or no response. And I have seen little progress or commitment in the last 10 years. Why do you think this is?
