Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement

EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement Current Contents, February 2,1987 This is the conclusion to a four-part series on refereeing and peer review in science. The first two parts discussed the refereeing of scholarly articles prior to publication.'.

2 The third part focused on the mechanics of the peer-review system for the evaluation of research-grant proposals at the National Science Foundation (NSF) and the National Institutes of Health (NIH) and opinions about those systems.

3 This part examines the research on peer review and some proposed alternatives and improvements. The COSPUP Study One of the best-known and most thorough studies of peer review was conducted by sociologists Stephen Cole and Leonard Rubin, State University of New York (SUNY), Stony Brook, and Jonathan R. Cole, Columbia University, New York. At the request of the Committee on Science and Public Policy (COSPUP) of the US National Academy of Sciences (NAS), the Coles and Rubin examined the peer-review system of the NSF. Phase one of the study, a retrospective statistical analysis of "how peer review works in the day-to-day operation of the Foundation [NSF],"

4 (p. 17) was started in 1974 and completed in 1978. Phase two, coauthored by the Coles and COSPUP, reported the results of experiments designed to address the question of whether program officers influence the peer-review process through their selection of reviewers. It was started in 1978 and published in 1981.

5 In phase one, the authors interviewed 70 scientists involved in all stages of the peer-review process, including current and former NSF program directors, advisoryand review-panel members, NSF section and division heads, and the director and associate director of the NSF.

4 (p. 18) To determine the most decisive factors in securing a grant, they collected data on 1,200 applicants, half of whom had been successful. In some cases, the authors examined not only the proposal but also the reviewers' comments, correspondence, and all paperwork connected with the funding decision. Phase two was carried out in two stages. First, the Coles submitted 150 proposals previously reviewed by the NSF to a set of surrogate program directors. Half of the surrogates received proposals that had been edited in an attempt to conceal the authors' identities; the other half received copies that were exactly as they had been submitted to the NSF. The surrogate directors were asked to name a set of possible referees for the proposals, and the Coles once again attempted to conceal the identities of half the authors. None of the participants knew how the proposals had been rated by the NSF. The Coles asked them not only to evaluate the proposals but also, when applicable, to try to identify the authors. Reviewers of "blinded" proposals were also asked whether the removal of title pages, fists of references, budgets, and other identifying comments made the proposal more difficult to evaluate.

5 (P- 6-19)