Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement
34 GARFIELD: REFEREEING AND PEER REVIEW, PART 1 Gillespie and colleagues sent a three-page, 19-item questionnaire to 719 researchers whose proposals had been approved or rejected by the National Cancer Institute of the NIH in 1980 and 1981. The questionnaire stated that those who did not return the survey would be assumed to be satisfied with peer review, so the authors find it noteworthy that 336 (47 percent) responded—although they do not presume that satisfaction with the system was the only reason for nonresponse. It is also interesting, the authors said, that 205 (61 percent) of the responses came from scholars whose proposals had been funded, since they expected a heavier response from scholars who had been denied funding. 1 6 Because the questionnaire was sent to researchers who had recently submitted proposals for review, it could not measure the attitudes of those whose discontent with the system had led them to give up submitting proposals. As the authors expected, previous success in obtaining funding was found to be inversely proportional to a desire to change the system. Gillespie and colleagues also found that those who have been unsuccessful until very recently in obtaining funding tended to support the current process, while those who had been successful in the past but who had recently been denied funding tended to favor modifications to the system. The authors also concluded that several complaints about the peer-review system reflected a surprising ignorance of the procedures governing the operation of the system. For instance, those who believe that cronyism or old-boy networks control the process fail to take into account the limited time that an individual may serve in a review group and the NIH's strict requirements concerning the makeup of such groups, which ensure a balanced cross section of scientists that changes constantly. 1 6 Jonathan Cole points out, however, that the choice of a given individual reviewer from among a number of roughly comparable candidates "can be a function of social and intellectual ties with study-section members." 6 Flaws in the System? There may be instances in which peer review operates with unintended blind spots or unsuspected inefficiency. Alan L. Porter, School of Industrial and Systems Engineering, and Frederick A. Rossini, School of Social Sciences, Georgia Institute of Technology, studied the fate of proposals that "fall between the cracks" of the NSF's disciplinary programs. 1 7 After analyzing 257 reviews received by 38 approved, cross-disciplinary proposals in five different subject areas, they found that reviewer decisions were more favorable when the proposal fell within the reviewer's own discipline. In discussing this finding, the authors found it reasonable "for a reviewer of proposed re search to favor that which is more familiar In such a case, one is apt to understand better what is planned; one may know the researchers personally or by reputation, and hence appreciate their expertise; and one can feel more secure in making strong recommendations." 1 7 Porter and Rossini conclude that interdisciplinary research proposals should not be reviewed in the same way as disciplinary projects. A study by Anthony S. Russell, associate professor of medicine, and Michael Grace, both at the University of Alberta, Edmonton, and Bonnie D. Thorn, director of finance and administration. Arthritis Society, Toronto, Canada, supports the widespread belief that the peer-review process is unnecessarily long and complex. 1 8 Russell and colleagues examined 113 grant applications to the Arthritis Society, a national voluntary health organization, to determine whether there were any substantia] differences between the initial assessment each proposal received in-house and the detailed, out-of-house review that followed. They found that in-depth reviews had little impact on the original rating, implying that review procedures that operate in a similar, twotiered fashion could be greatly streamlined. 1 8 And in fact, in an analysis of nearly 1,400 reviews of about 200 NSF proposals, David Klahr, Carnegie-Mellon University,
