Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis
109 MITROFF & CHUBIN: PEER REVIEW AT THE NSF science advising entails the disposition of scholarly work (such as grant proposals and manuscripts) by referees representing, but not representative of, the scientific community. 6 It is the linking of advice with ultimate dispositions (that is, decisions) which endows peer review with a distinctive content. Clearly, we seek to generalize in this paper about content, recognizing that the form of peer review varies. For example, whereas the National Institutes of Health use a system of study sections, NSF uses review panels. 7 Whereas some referees act as ad hoc mail reviewers, others attend periodic meetings as panel members. But what counts is that multiple judgments are solicited and weighed differentially (depending on the source) to reach a decision: to fund or not to fund. What we shall argue is that defensible decisions are not inevitably guaranteed by the peer review mechanism. Indeed, the process can be used to justify any decision. The power vested in the mechanism or process is derived, in large part, from the power (for instance, reputation) of the referee-advisor-gatekeeper, and is rationalized by the system. 8 To reiterate, it is to this content we shall generalize, though our data are of a more limited form. The philosophy underlying the mechanism of peer review (at least in the US) warrants such substantive generalization. The Debate: A Dialectical Statement of the Issues The mode of presenting the peer review debate can help to elucidate the substance of the debate itself. Insofar as a debate features arguments sampled from a continuum of opinion, those arguments can be presented in the form of a dialectic. 9 However, since this way of posing policy issues may not be as familiar as other forms, it might be helpful to offer at least a brief exposition of dialectical analysis. In recent years scholars in those diverse fields now subsumed under the 'social studies of science' rubric — particularly in history, philosophy, sociology, and management science — have advanced a series of potent theoretical arguments for a dialectical treatment of policy issues: indeed, they have argued that the conduct of science is dialectical in its basic structure. 1 0 The essence of the argument is as follows:
