Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis
120 MITROFF & CHUBIN: PEER REVIEW AT THE NSF reviewers and applicants believe to be the operation of the NSF peer review system — and not with more direct evidence of how it actually operates (which the study never purported to establish). Lest our intent be misunderstood, we are not thereby disparaging the value of the study. What scientists believe about an institution that vitally affects them is key information. The fact that so many scientists would oppose the disclosure of names of reviewers is important information in its own right. It can inform those in power that strong opposition awaits if a policy of disclosure were instituted. This finding does not indicate, however, to what extent and what forms such opposition might take, or whether the benefits of a new policy might so outweigh the disadvantages that the change would be worthwhile. It is not that the Hensler study is merely a survey of beliefs, but that it fails to penetrate to the heart of the debate, and consequently, does not aid in its resolution. To facilitate movement towards resolution, at the very least, other crucial beliefs would have to be exposed; at best, there would have to exist some other method(s) for assessing the actual state of the system. As they stand, the Hensler data do not prove that the NSF peer review system is either biased or unbiased, but that there are sizeable numbers of scientists whose experiences (that is, reviewers versus successful applicants versus unsuccessful applicants) predispose them to support one side or the other of the debate. The Cole, Rubin and Cole Study The Cole, Rubin and Cole study represents the most ambitious project to date to determine the actual operation of the NSF peer review system, at least in its basic research programmes. 4 1 Commissioned by NSF on behalf of the National Academy of Sciences, the study by Cole and his colleagues (hereafter referred to as Cole) seeks to provide evidence that is independent of scientists' beliefs or attitudes about the presence or absence of biases in the NSF peer review system. The kind of evidence sought by Cole thus augments that yielded by the Hensler survey. Specifically, Cole conducted seventy in-depth interviews with scientists involved at ail levels of the peer review system, including program directors, former program directors, mail reviewers, review-panel members and supervisory-level NSF officials. We also scrutinized
