Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis
122 MITROFF & CHUBIN: PEER REVIEW AT THE NSF the scientific enterprise is an exceedingly equitable, although highly stratified, social institution in which the individuals who produce the work that is most favourably evaluated by their colleagues receive the lion's share of the rewards. 4 9 Reactions to the Cole Study We think the Cole study begs several questions that are vital to the peer review debate. To compound this error, the authors make some definitive-sounding extrapolations that seem unfounded by their data. First, to conclude that 'the peer review rating is by far the best predictor' of the probability of receiving a grant by no means suggests that this rating is a good predictor. Indeed, '89 percent of the observed ratings is left unexplained by the nine variables.' 5 0 This would indicate that factors other than 'social stratification' variables are at work. Yet no such factors are either employed in the analyses or conjectured in discussion of those analyses. Thus the finding that 'individuals who produce the work that is most favourably evaluated by their colleagues receive the lion's share of the rewards' circumvents the questions of why the work is favourably evaluated. No measure of its significance or innovativeness is presented; we are simply asked to believe that voluminous citation of articles denotes their high quality. 5 1 (After all, proposals which seek to extend such widely-recognized work must be of sufficient merit to justify NSF's decision to fund.) If these results accurately describe peer review in the basic research programmes of NSF, then Cole must further show why this system is 'extremely equitable, although highly stratified'. They have not done this, their rhetoric notwithstanding. Second, what Cole recognizes in the data, but overlooks in the interpretation, is that evaluation of a producer is hopelessly intertwined with the evaluation of his or her product in science. If producer and product cannot be separated analytically, then we must ask: Where in the distribution of peer review ratings does particularism tend to prevail? If high consensus is achieved in both tails of the rating distribution (as the data attest), 5 2 then the critical region of peer review is in the middle. This is the grey area where particularistic factors (such as the applicant's present affiliation or institution of PhD) colour the perceived quality of the proposed research. Since performance or 'track record' (that is, reputation) is supposed to be an explicit factor in reviewing for NSF, particularism has been institutionalized as a (partial) rationale for making both favourable and unfavourable decisions to fund. The contradiction is legitimate; tension between universalism
