Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement

32 GARFIELD: REFEREEING AND PEER REVIEW, PART 1 The COSPUP Findings The main conclusion of phase one is that peer review in the NSF functions fairly.

4 (p. viii-ix) The authors found a high correlation between high reviewer ratings and favorable funding decisions. They also found that an applicant's age and track record had little effect on the chances of getting a grant and that reviewers from major, "high-status" institutions treated proposals from researchers at prestigious institutions no differently than proposals from workers at less-prestigious institutions. On the whole, the results of phase two corroborate the findings of phase one: the Coles found no evidence of bias on the part of program officers in their selection of reviewers and no evidence that external criteria such as gender, age, and race had any influence on reviewer decisions.

5 (p. 4) In the matter of blinded proposals, the Coles found it difficult to conceal authorship: "To omit all possible identifiers, in addition to the name(s) of the author(s) of the proposal, made the proposal almost unreadable," said Jonathan Cole.

6 This was reflected by the opinions of the COSPUP reviewers, who felt that the blinding process severely compromised the integrity of the proposals. Nevertheless, proposals that received high ratings by NSF reviewers generally received high ratings from COSPUP reviewers as well. However, "there was a great deal of well-considered variance in opinions among equally qualified reviewers," in the words of Jonathan Cole.

6 "Thus, if we work with a small number of reviewers and a high variance in opinion, the outcome of an evaluation will depend greatly on the people selected to review the proposal.

... This is not to imply that the process is 'unfair,' but that there is a substantial level of reviewer disagreement on rational grounds, e.g., quality of past work, priority given by a particular reviewer to the subject of the proposal, the assessment of the methods to be used, etc."

6 The Coles concluded that perhaps 25 to 30 percent of NSF funding decisions would be reversed if applications were evaluated by another, equally qualified group of reviewers. In both their phase-two monograph

5 and a paper they published in Science with statistician Gary A. Simon, SUNY, Stony Brook,

7 the Coles acknowledge that some scholars, taking note of this, will feel that the complicated system of peer review "does not buy you much."

5 (p. 43) Jonathan Cole points out, however, that "there is apt to be a great deal of disagreement on the contents of proposals...at the cutting edge of scientific inquiry,...and therefore, we should not be wholly surprised at the proportion of reversals."

6 Such reversals probably indicate that no "single, agreed-upon dogma"

7 is dominant in the fields studied, and, in fact, one of the most surprising results of the COSPUP study was that the level of consensus among reviewers was no higher in physics than in the social sciences.

47 Phase one of the COSPUP study has been cited in over 74 papers since it appeared in 1978; phase two has been cited in 17. The Science article has been cited 49 times through 1986 and is one of four papers forming the core of a research front entitled "Alternatives to, arbitration in, and other aspects of peer review of scientific journals and research proposals" (#85-4243). The other three core papers include the classic study of patterns of evaluation in science by Harriet Zuckerman and Robert K. Merton, Columbia University;

8 a controversial study of bias in the journal refereeing process by Douglas P. Peters, University of North Dakota, Grand Forks, and Stephen J. Ceci, Cornell University, Ithaca, New York;

9 and a paper on the rate of agreement between reviewers by psychologist Graver J. Whitehurst, SUNY, Stony Brook.

1

0 All three papers were mentioned in Parts 1 and 2 of this essay.

12 According to Jonathan Cole, the key policy implication of the COSPUP study was