Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement

33 GARFIELD: REFEREEING AND PEER REVIEW, PART 1 that "the lower the number of reviewers used to evaluate the proposal, the greater the chance for...reversals."

6 As a result, the NSF now requires a certain minimum number of reviewers for every proposal it receives. Science journalist Tineke Bőddé lists a number of other changes in the NSF system that have been made more recently.

11 For instance, the entire process has been streamlined, with a limit of 15 pages per proposal and a policy requiring a decision within nine months. Specific guidelines on conflicts of interest have been established, verbatim copies of all reviewer comments have been made available, and a system has been set up to reconsider declined proposals. Under certain circumstances, some proposals are now exempt from peer review, and program officers can extend existing grants without further review if they feel outstanding progress has been made.

1

1 Peer Review in the NIH Fourteen scientists and administrators from various agencies within the NIH were appointed to the NIH Grants Peer Review Study Team by then-acting director, Ronald W. Lamont-Havers. Chaired by Ruth L. Kirschstein, director, National Institute of General Medical Sciences, the team was charged with evaluating the NIH's peer-review system and with making, where applicable, recommendations for improvement.

1

2 In making its assessment, the team printed an open solicitation in the Federal Register

1

3 and mailed a memorandum to 30,000 individuals, asking for written comments on the peer-review system (1,500 replies were received). The team also held open public hearings for the scientific and lay communities. The team members considered everything they read and heard, according to William F. Raub, team member and deputy director, NIH, but the project was an informal survey and, ultimately, the recommendations the team made were based on a consensus' of its members' best judgments.

1

4 Virtually every recommendation made by the study team has been implemented.

14 Among these was the suggestion that guidelines on conflicts of interest and a formal appeals system for the reconsideration of rejected proposals be established. In addition as part of the appeals procedure, the teair suggested that specific criteria be established for reevaluating proposals and that an independent ombudsman be appointed to adjudicate disputes between the NIH and applicants. A change in NIH procedure that was recently instituted is the creation of two programs allowing the life of a grant to be extended for up to 10 years under certain very limited circumstances.

1

5 In connection with the report by the NIH study team, Jonathan Cole suggests that a fruitful area for research would be a rigorous comparison of the NIH study-section approach to peer review with the individual approach used by the NSF. He says thai "panels can evaluate the relative strengthof a set of proposals, but, in fact, each pane member, while voting on all, actually only reads a few. This leads potentially to an artificial consensus, where a couple of strong characters on the panel dominate the decision-making process."

6 Studies of Scholars' Attitudes Toward Peer Review Sociologist Gilbert W. Gillespie, Cornell University; Daryl E. Chubin, director, Technology and Science Policy Program, Georgia Institute of Technology, Atlanta; and physician George M. Kurzon studied the factors that help shape applicants' attitudes toward the system.

1

6 The authors expected to find that those who experienced success in obtaining funding would tend to be satisfied with the status quo and that those who failed to obtain funding would tend to blame the system.