Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
ALAN L. PORTER and FREDERICK A. ROSSINI: Peer Review of Interdisciplinary Research Proposals
PORTER «Sc ROSSINI: INTERDISCIPLINARY RESEARCH PROPOSALS 157 Table 1. Peer Rating Contrasts More Favorable Less Favorable Statistical Dimension Type Mean (±S.D.) |N| Type Mean [ — S.D.) IN) Significance Level NSF Program Scientific 1.50 1-0.69] |102) Engineering 2.05 1-0.96) (149) 0.0001 PI Affiliation Academic 1.65 (±0.80) (165) Non-Academic 2.18 (±0.98) 1 86) 0.0001 PI Disciplinary Category Scientific 1.75 (-0.86) (189| Engineering 2.07 (-0.981 1 62| 0.02 Reviewer Disciplinary Category Scientific 1.75 (-0.87) (128) Engineering 2.12 1-0.97) I 70) 0.01 Project Type Basic Research 1.56 1-0.43) ( 17]' Applied or 2.00 (±0.50| ( 2ir 0.01 Policy Research Disciplinary Match Between Reviewer Same or and PI Similar 1.69 1-0.83) (199) Different 2.22 (±0.99) ( 69) 0.0001 Note: Scale is the NSF tating from 1 - excellent to 5 » poor. Scale is naturally compressed as al] of these proposals were funded. * Number of projects on a project-based comparison instead of a review-based one as in the other four contrasts. to which the proposal happened to be submitted than by perceived differences in merit among individual (funded] proposals. Stated another way, the funding cutoff varies significantly by program. The person submitting a crossdisciplinary proposal might do well to compare the typical peer rating profile for each program that might evaluate the proposals. Switching to stepwise regression, we considered whether other variables augment the program variable as a predictor of a proposal's peer raring. Two other variables enter the regression significantly —whether or not the PI is academic and the similarity of PI and reviewer d..eiplines. Each of these is discussed below, with academic PI being the strongest predictor. Wh . both of these variables are entered into the equation, program is no longer a significant predictor. The intercorrelation among the predictors is also discussed further. The second line of Table 1 summarizes information on ratings by organizational affiliation of the Principal Investigator. Given the peer ratings profile that favors basic scientific research conducted in academic settings, we would have suspected that proposals from Pis nested in established disciplines would have rated better than those from centers. Forty-three percent of the ratings pertained to Pis associated with academic departments. These rated quite favorably (mean = 1.73). Interestingly, proposals from academic centers or composite center/depajtment arrangements fared at least as well (mean = 1.50). To simplify further comparison, we combined all academic proposals in Table 1. Four other groups of proposals (from public or quasi-public organizations involved in funding, or in using research; from large or small contract research organizations) each averaged over 2.0. We conclude that either academic Pis know how to prepare better proposals, or reviewers favor academics, or both. Similar patterns emerge when we consider the disciplinary category of the PI or of the reviewer. "Science" rates better than "engineering." By type of project, basic research (mean rating 1.56 for 17 projects! does better than applied (mean rating 1.98 for 13 projects) or policy research (mean 2.04 for 8 projects). Applied and policy research are combined as a category in Table 1. Differences by reviewer affiliation (i.e., academic or not; researcher or research user| did not reach statistical significance. Nor did review characteristics interact significantly with corresponding PI characteristics. One might have expected academic reviewers to be more favorable toward academic Pis, and nonacademic reviewers to favor non-academic Pis. The review data did not show such inclinations. Not surprisingly, the factors discussed are heavily interrelated. Examination of correlations among these variables finds ail of them significant, ranging in magnitude from 0.22 to 0.74. "Program," in particular, associates with whether the PI (r =
