Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation
82 CICHETTI: THE RELIABII .ITY OF PEER REVIEW yields a correlation for the two highest levels of evaluation (acceptance) of —.017, whereas for the two lowest levels of evaluation (rejection) the analogous correlation is .107. The difference between R's of .107 and -.107 surely seems more modest than the difference between agreement rates of 70% and 44%, although in the same direction. Indeed, Cohen (1988) treats the former difference as "small" (q s . 12) and the latter as "medium" (h ss .53). Conclusion. In this commentary on the indices of reliability evaluated or used in the target article, three are recommended as information-efficient: Pearson R, intraclass R, and Cohen s kappa for the 2x2 case. It is further recommended that 3 others not be used: rate of agreement, kappa for tables larger than a 2X2 (with df > 1), and x 2 or any other tests of significance, because they depend on sample size as well as on reliability per se. Toward openness and fairness in the review process Byron P. Rourke Department of Psychology, University of Windsor, Windsor, Ont. N9B 3P4, Canada Cicchetti's target article is thorough and thoughtful. The issues addressed are important, and each is dealt with in a systematic manner. Suggestions for future research are clear and relevant. The overall approach to the topic is disinterested and scientific. There is little to criticize in this presentation. I would point to some issues that seem important to me, however, that either were not mentioned or did not receive the sort of emphasis I would have given them. These are as follows: 1 . Signed reviews. From my perspective, it would seem desirable for all reviews to be signed. I say this in the full realization of realization of the problem that the author mentions regarding "younger" scientists and the recriminations that they may suffer as a result of criticizing the work of more established researchers. There is also the possibility that the younger scientist may mute criticisms if forced to acknowledge their source. I would argue, however, that insisting that all reviewers acknowledge their identities is fair and just. It would be easy to implement; in any case, it will eventually become the rule rather than the exception. (Hence, with regard to this issue, I would disagree with the position taken by Cicchetti.) With respect to fairness, I would simply point out that hiding behind the cloak of anonymity opens the door to the worst sort of blackballing. Stating opinions that one must stand by — and defend, if necessary - is part and parcel of what the social dimension of science is all about. With respect to ease of implementation, I would suggest that mandating the acknowledgment of reviews would be difficult at first, but reviewers (i.e., active scientists) would soon become accustomed to the process. Finally, as is now the case for many governmental granting bodies, the freedom of information "movement" will eventually target journals to achieve the same degree of openness as now exists in the area of grants. 2. The role of the journal editor. Journal editors can do much to increase the probability of positive or negative reviews through their choice of the consulting editor(s) (CE). Experienced journal editors have sufficient knowledge about the likely reactions to particular pieces of research of many, if not most, of the members of their editorial boards. Even though CEs are retained on the editorial board because of their perspicacity as well as objectivity, some can be "depended on" to look askance at particular research designs, methods of analysis, subject populations, and any number of other important dimensions of scientific papers. Furthermore, it is my impression that negative biases will have a much greater effect than will positive biases: That is, it is likely that the negative biases of CEs will have a larger (negative) effect than will their positive biases about particular kinds of research. Indeed, it has often been my experience that CEs will bend over backward to find fault with research that is clearly similar to their own (i.e., of a sort toward which they would be expected to have some positive bias). This being the case, what is the journal editor to do? Random assignment of CEs is one alternative. In all but the most narrowly focused journals, however, this would result in many nonexperts reviewing the work in question. The only other solution is to attempt to balance the review process by choosing CEs who represent conflicting stances vis-á-vis the project in question. If this seems to call for a rather well developed sense of where CEs stand with respect to important issues in the field, I would emphasize that this is exactly what I mean to convey. There is no substitute for a fair, judicious, and experienced journal editor if the process of evaluation is to proceed in a fair and judicious manner. 3. A good article will get published somewhere; bad articles however, also tend to get published somewhere. Cicchetti does not see the high rejection of rates of, say, the New England Journal of Medicine, as much of a problem. He points out, with good reason, that the vast majority of articles rejected from the handful of very prestigious journals eventually see the light of day elsewhere. This is all well and good. What is not so good is the practice followed by some authors of resubmitting articles to any number of journals until they get lucky. For example, over the past 14 months I have had the opportunity to review an article for four different journals. I commented on the article the first time around, and recommended a number of changes in it. When I received the unamended article from a second journal, I informed the editor that I had reviewed the piece previously, and forwarded my first review to him. The third and fourth times around for the article - still in its unamended form - were handled by simply informing the editor that I had seen the article before, that I had recommended changes in it, that these had not been made, and that I did not want to go to the trouble of commenting any further on the manuscript. I should be quick to add that my experiences in this regard are not unique: I have shared similar stories with many of my neuropsychological colleagues. What usually transpires in such cases is that the author eventually finds space for the article in a low prestige journal or, worse, a high prestige generalist journal that does not have the editorial expertise available for the proper evaluation of the manuscript. What should be done about this situation? I think the answer is quite simple: Demand that authors submit a statement to the effect that their article has been submitted to journals, X, Y, orZ and that the article has been rejected. In addition, the reviews of the article could be provided to the editor of the journal that must now decide on the acceptability of the work. 4. Journal editors should provide the verbatim reports of CEs to authors. This point was not raised in the target article. Nevertheless, it is an important one. Justice and fairness require that authors see for themselves the reviews of the work submitted. A précis of CE comments simply will not do. This procedure would enhance the fairness of the process even more if the suggestion cited above regarding the acknowledgment of the identity of the reviewers were adopted. Finally, I would point out that the suggestions made by Cicchetti regarding future research in this field are quite important. These efforts will not only enhance our knowledge about decision-making in the publication of scientific articles and the allocation of grants but will also aid immeasurably in generating new (and, one hopes, fairer) modes of operation by and for editors and granting agencies. [The commentator is co-editor of Journal of Clinical and Experimental Neuropsychology and The Clinical Neuropsychologist. Ed.]
