Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis

121 MITROFF & CHUBIN: PEER REVIEW AT THE NSF more than 250 specific research proposals, read all of the peer review comments on those proposals and examined all of the correspondence between the applicant and the program director ... In addition, we conducted a quantitative analysis of 1200 applicants to the NSF in the fiscal year 1975. (Roughly half of the applicants were ultimately awarded grants.) The purpose of the quantitative study was to identify those characteristics that were correlated with the receipt of a grant from the NSF.

4

2 The characteristics chosen for analysis (basically a series of multiple regressions) consist of nine 'social stratification' variables, including rank of PhD-granting department, current academic rank, and three measures of publication and citation. These variables were then correlated with the ultimate disposition of a grant proposal measured in the aggregate as 'percentage of applicants receiving grants' and 'ratings received on proposals' (trichotomized as high, medium, and low).

4

3 Taken together, analyses of these variables are intended to test the validity of two hypotheses which dominate the peer review debate. The first (the 'old-boy' hypothesis) lacks 'conceptual clarity', according to Cole. Does old-boyism refer to 'investigators with a common view of their field', 'networks of friendships', or to 'social position' ('level of eminence')?

4

4 The second (the 'rich get richer' hypothesis) stipulates that particularistic factors (that is, those unrelated to the merit of a proposal) result in an unfair advantage (for example, for the more eminent and/or those located in high ranked departments) in gaining grant approval.

4

5 Based on their quantitative analyses, Cole interprets the evidence as a refutation of both hypotheses: The overall pattern of our data suggests that scientists with an established track record, many scientific publications, a high frequency of citations, a record of having received grants from the NSF and ties to prestigious academic departments have a higher probability of receiving NSF grants than other applicants do. Nevertheless, the granting process is actually quite open and there is nothing approximating a scientific caste system.*

6 Of the variance that can be accounted for in funding decisions, the peer review rating (among the social stratification variables) is by far the best predictor.

4

7 ... a scientist's past performance as measured by citations of his work and his recent NSF funding record does lead to a very slight accumulative advantage, but his academic affiliation does not appear to give him any advantage.

4

8 Not surprisingly, in summarizing the results of their study thus far, Cole suggests that