Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

ALAN L. PORTER and FREDERICK A. ROSSINI: Peer Review of Interdisciplinary Research Proposals

158 PORTER «Sc ROSSINI: INTERDISCIPLINARY RESEARCH PROPOSALS 158 0.41) and reviewer (r = 0.42) axe academic (i.e., the engineering programs are more likely to engage non-academics); and with whether the PI (r = 0.48) and reviewer |r = 0.52) are scientists or engineers. The key finding is that projects that are "scientific, basic research in academic units" are the highest rated, while those that are "engineering, applied or policy research in non-academic units" are rated lowest. Projects with other combinations of characteristics are rated between those extremes. The key determining factor seems to be whether or not the project has an academic PI. However, in this study the connections demonstrated among the variables influencing peer rating are correlational, not causal. Our study also does not address the issue of whether some proposals are "better" than others in a sense independent of the ratings. Stated another way, someone could argue that the proposals by the academic Pis are rated more favorably than those from nonacademics because they ate inherently superior, not because of any favoritism. The case with respect to disciplinary match between PI and reviewer is more sharply defined. As shown on the last line of Table 1, reviewers favor that which is familiar. We considered information on reviewer and PI affiliation to judge whether their disciplines were the same (e.g., both chemists), similar (e.g., physiologist and anatomist), or different (e.g., engineer and sociologist). This line collapses categories of "same" and "similar" (means of 1.68 and 1.71). To confirm this finding, we examined the similarity of genera] disciplinary category (e.g., social science, engineering) of PI and reviewer using a non-judgmental coding. Everyone was assigned a three-digit code for discipline and these codes were located in general disciplinary categories, (following the National Research Council's groupings used in the study of doctoral scientists and engineers). Where reviewer and PI were affiliated with the same general disciplinary category, peer ratings were better (mean = 1.73); where they differed, peer ratings were significantly worse (mean = 2.08; p = 0.008). The implications of this favoritism for the familiar are critical lor interdisciplinary research |IDR|. First, this effect is extremely strong for it to appear in a sample of a constricted range of ratings. Second, this effect will work against support for IDR in that such research is inherently less likely to be reviewed by persons familiar with -the full scope of the planned work. Our conclusion should caution against the generally accepted strategy of choosing review teams, each of which is familiar with only one of the aspects of the project. These results imply that such a review strategy will generate poorer ratings, on average, than would come from a review team on which each member was familiar with the whole scope of the proposed research. Our final inquiry was whether there was a correlation between project interdisciplinarity |based on a factor analysis as noted under "Study Design") and rating. This query is constrained by the sampling of predominantly IDR projects. We did observe a correlation between rating and degree of project interdisciplinarity in the direction of more interdisciplinary projects being downgraded. In a split-sample analysis, this conclusion shows a significant association for one set of 20 projects (r = 0.50, p = 0.02), but non-significant association for the other subsample (r = 0.16, p = 0.26). However, the correlation is significant for the sample as a whole (r = 0.29, p = 0.05). This partial support for the hypothesis that IDR projects tend to rate less favorably suggests further study of a representative sample of both disciplinary and interdisciplinary proposals (preferably including non-funded as well as supported proposals). Discussion Researchers engaged in certain areas, such as the neurosciences, appear to meet less resistance from their peers (proposal reviewers and professional reward evaluators| for performing interdisciplinary research than do those in more traditional disciplinary areas. Some research areas appear more "open" to using techniques, and even substantive expertise, not historically wedded to them. Of the comments which we saw, reviewers rarely criticized crossdisciplinary features of proposals. (One economist reviewer did fault a proposal for including non-economic aspects.) Mote typically, suggestions were made to add a particular skill to the project team. On occasion, reviewers would indicate reluctance to evaluate the proposal other than in their own domain of expertise. Our general sense was that neuroscientists and aicheologists, in particular, did not need to justify their inclusion of "outside" skills. The characteristics of the research problems seemed to require certain skills, and the researchers tried to provide them. On the