Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
ALAN L. PORTER and FREDERICK A. ROSSINI: Peer Review of Interdisciplinary Research Proposals
156 PORTER «Sc ROSSINI: INTERDISCIPLINARY RESEARCH PROPOSALS 156 One of the most striking observations about peer review of proposals is the extent to which this process is unquestioningly accepted in the absence of much empirical data on how it performs. Cole et al., in a landmark study, contrasted NSF peer ratings with independent sets of raters. 9 They found that the fate of individual grant applications was about half determined by characteristics of the proposal, about half by "noise" in the review process. We know of no empirical data, however, on such critical issues as the effect of reviewer heterogeneity 1 0 or the characteristics of the proposed research on expected ratings. The intent of our article, therefore, is to provide such evidence on one sample of crossdisciplinaty projects. Study Design This study draws on 257 sanitized reviews of 38 projects from five different National Science Foundation programs: Neurobiology, Environmental Geosciences, Archeology, Earthquake Hazard Mitigation —Societal Response, and Science and Technology to Aid the Handicapped. These encompass basic, applied, and policy research. The sampling was purposive rather than statistically representative or random. We worked with program managers to identify projects they perceived as interdisciplinary (and added a few problemoriented disciplinary projects for comparison). Roy lists four dimensions basic to considering peer review of proposals": (1) The candidate set of proposals —(a) in response to an RFP versus (b) unsolicited, with a deadline, versus |c) no "set" at all |a program considers pertinent proposals as they come in); (2) Reviewers —how many and who selects them; (3) The review medium —mail (advisory only), mail (binding], panel, site visit, or combinations; and (4) The degree of interaction with the principal investigator |PI), ranging from none to substantial. In these terms, the 38 NSF projects for which we had reviews cross all candidate set types. For some, peer evaluation incorporates explicit comparison among a set of proposals; for others, a proposal is considered alone. The evaluations range in number of reviewers from 1 to 17 (with no specific information on their selection]; the mean number of reviews per proposal varied from 5.9 to 8.1 for the five programs. These reviews incorporate mail or mail/panel combinations. In a few cases, the review process includes feedback of initial criticisms through the program managers for principal investigator response. Complementing the proposal reviews, we were able to secure information from the Pis on the nature of the actual research process." In particular, we draw on this to determine how interdisciplinary each project was, based on a weighted function of the Pis' and our own judgments, on the number of disciplines represented on the project team, on the percent of staff from outside the Pi's general disciplinary category (i.e., engineering, life sciences, physical sciences, social sciences, or professional fields), and on the range of skills used in the project. Results The proposals studied were all funded and thus had high ratings on the NSF scale of one to five (with one being excellent and five being poor]. Also, because the projects in the sample were selected for their interdisciplinary character, we originally believed that there would be few differences in the sample in terms of such characteristics as type of project or specialty of reviewer. As Table 1 indicates, we found some rather significant differences, however. These can be summarized by noting that reviewers favored basic scientific research conducted in an academic setting. The first line of Table 1 condenses information on how ratings differed among the five NSF programs under study. Each of the two Engineering Programs (Earthquake Hazard Mitigation and Science and Technology to Aid the Handicapped] averaged a rating of 2.05. Among the sciences, the single archaeology proposal for which we had numerical ratings scored most favorable at 1.14, followed by the neurobiology proposals at 1.42, and geoscience at 1.68. Engineering proposals received more reviews (mean number of reviews of each = 7.5) than did the science proposals (mean = 6.0). In a multiple regression of peer rating on the various factors examined, the program variable appeared as the strongest predictor (p < .001), dominating other factors, including which proposal was being rated. In other words, more of the variability in rating was accounted for by the program
