Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis

115 MITROFF & CHUBIN: PEER REVIEW AT THE NSF whelmingly against Congressional review.'

2

3 (5) No one ascribed to the extreme position that 'applicants should know nothing about who reviewers were or what they said.'

2

4 Such substantial agreement on these propositions makes them no less true or false, however, than the propositions over which there is explicit disagreement. In either case, beliefs may be so deeply held by their proponents that, at least for them, they may be immutable and irrefutable — utterly impervious to evidence. For those who are not so rigidly committed to either side as to dismiss either position a priori, the question is: What evidence, if any, exists or could be produced that bears on the various propositions and could alter the beliefs of proponents and critics alike? The Evidence In reviewing the evidence marshalled in support of each side of the debate characterized above, we shall restrict our attention to two studies: one conducted by Deborah Hensler,

2

5 the other by Stephen Cole and his colleagues.

2

6 The justification for this restriction is that, based on a review of materials on the NSF peer review system,

2

7 these two contain the most comprehensive bodies of empirical evidence which speak directly to some of the contentious issues in peer review. The review of each study will consist of describing its focus, data, and principal findings, followed by our assessment of its merits and shortcomings. Finally, we shall discuss the inferences about gaps in knowledge which can be drawn from the current literature on peer review — theoretical and operational gaps which invite further imaginative study to effect closure on key issues in the debate. The Hensler Study In 1975-76, Deborah Hensler sent a questionnaire to a 5 percent random sample (n= 1552) of approximately 31,000 persons who had served as reviewers of research proposals submitted to NSF during fiscal year 1974. The identical questionnaire was also sent to a random sample of 3256 applicants for NSF grants during the period (a 16 percent sample framed by a population of 20,000 grant applica-