Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis
137 MITROFF & CHUBIN: PEER REVIEW AT THE NSF Hensler study dealing directly with the topic of 'grantsmanship', although it is implied by a few of her questions.) 41. Cole et al., op. cit. note 26. The longer version of this report had yet to appear when we were preparing the final revision of this paper. It was published while the paper was in press: see S. Cole, L. Rubin and J. R. Cole, Peer Review in the National Science Foundation: Phase One of a Study (Washington, DC: National Academy of Sciences, 1978). We have been unable to incorporate and respond to details of the published report. An earlier draft of our paper benefitted from an unpublished preliminary version of the Cole report forwarded to I.I.M. as part of his review (see note 27). However, since it was the only public version available at the time, we have taken special pains to confine our comments here to the published interim Scientific American report. 42. Ibid., 36. 43. Ibid., 37-39. 44. Ibid., 37. 45. Ibid., 38. 46. Ibid., 40 (our italics). 47. Ibid. 48. Ibid., 41. 49. Ibid. 50. Ibid., 38. 51 . The use of citations as a measure or indicator of the perceived importance or standing of a scientist within his or her domain of research seems non-problematic to Cole, despite the reservations expressed by some investigators regarding the validity of the measure. See D. E. Chubin and S. D. Moitra, 'Content Analysis of References: Adjunct or Alternative to Citation Counting?' Social Studies of Science, Vol. 5 (1975), 432-41; N. Kaplan, 'The Norms of Citation Behavior: Prologomena to the Footnote', American Documentation, Vol. 16 (1965), 179-84. Do numerous citations to the work of a scientist truly reflect the long-term importance and significance of the work, or merely its short-term popularity? Does one cite a work to support one's own, or for more critical and rhetorical reasons? Even if it is presumed that the reviewer has knowledge of citation performance, without taking into account the reasons why scientists cite others, and why variations in citation behaviour exist across disciplines and research areas, can one confidently use citation counts as a 'common denominator' predictor variable? On this score, Cole finds — interestingly enough — that past citations contribute little to the explained variance of the funding decision, suggesting to us that citation may be an irrelevant criterion of performance because most reviewers are ignorant of an applicant's citation 'performance'. 52. Cole et al., op. cit. note 26, 39. 53. This is really an expression of our dismay over Cole's decision to report only the results of quantitative analyses. Such analyses can mask individual differences manifested in anecdotal accounts, such as interviews. For example, it would be instructive, if not indispensable, to know how NSF personnel view the system of science, and the rationality of the enterprise in which they are engaged. In the study of eminent scientists who investigated the moon rocks returned by the Apollo missions (see note 39), it was found that nearly the entire sample of forty-two scoffed at, antf in the most derisive of terms, the stereotypical view of the scientist and science itself as the 'open, free, unbiased exchange of pure ideas' that is so
