Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis
138 MITROFF & CHUBIN: PEER REVIEW AT THE NSF commonly portrayed in college texts and in popular accounts of science. Surely it is important to know whether a respondent holds a conventional or a radical view of the workings of science before one can properly evaluate the respondent's attitudes towards the peer review system. Surely programme directors differ in discharging their duties. Are those harbouring a less conventional view of science more critical and sceptical of the operation of NSF peer review? 54. Gordon and Morse, op. cit. note 39. 55. Ibid., 42. In the typical situation used by Gordon to measure differentiation, a person is asked to rate ten of his most immediate colleagues, friends, associates, and so on, on a ten-point scale with respect to (a) their productivity, (b) their creativity, and (c) how easy it is to get along with the individual being rated. Low differentiaiors tend to rate all ten persons identically; in other words, low differentiators make use of only a small portion of the total ten-point scale, whereas high differentiators tend to make significantly more use of the whole scale. High differentiators tend to view people as different and unique; low differentiators view them as the same. 56. Cole ct al., op. cit. note 26, 34. 57. To this end, we are surprised that the Cole study was not couched in terms of the Ortega hypothesis which the Coles (J. R. Cole and S. Cole, 'The Ortega Hypothesis', Science, Vol. 178 [27 October 1972], 368-75) investigated a few years ago. The Coles' rejection of this hypothesis (that all scientists contribute through their modest research efforts to the incremental progress of science) raises questions as to the concentration of funding support among a small portion of the research community. S. J. Turner and D. E. Chubin, in 'Another Appraisal of Ortega, the Coles and Science Policy: The Ecclesiastcs Hypothesis', Social Science Information, Vol. 15 (1976), 657-62, argue that to equate the distributions of scientific talent, productivity, and reward is little justification for a science policy that deliberately concentrates resources among the elite that populates one tail of those distributions. Rather, they question the efficiency of a policy that would waste the talents of trained personnel without modifying the organizations that train and employ them (though we realize this is far easier said than done). To sustain the research of more scientists could calculably enhance their contributions. Yet no experiments in the democratization of research allocation have been carried out. Thus the proposition remains untested, and for us at least, the Ortega hypothesis, like the 'old boy' and 'rich get richer' hypotheses to which it is intimately related, has been gratuitously laid to rest by the Coles. 58. An 'unbiased' system would obviously not be one that randomly funds proposals; rather, it would fund primarily according to merit, which might be defined as innovative, feasible, relevant, or some combination thereof. In the discussion that follows we assume that a lack of bias is both desirable and attainable. 59. We purposefully use the term 'admittedly oversimplified' because the actual situation may be too complex to admit of the two exclusive categories, 'biased' or 'unbiased'. The actual state of the system may be neither biased nor unbiased, or it may be a condition of both — that is, a complicated mixture of partially 'biased' and 'unbiased' elements. Nevertheless, for the purpose of this analysis, it suffices to consider the 'idealized' cases in Table 2. 60. Notice that we do not say that Cases II and IV necessarily represent 'incorrect cases', since the difficulty in knowing the 'true' state of the actual system also
