Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

IAN I. MITROFF and DARYL E. CHUBIN: Peer Review at the NSF: A Dialectical Policy Analysis

116 MITROFF & CHUBIN: PEER REVIEW AT THE NSF tions). For both the reviewer and the applicant samples, the response rate exceeded 80 percent. Both the original questionnaire items and the resultant analyses of responses were designed to attempt to address several points of contention in the debate. An initial question concerns the similarity of backgrounds of reviewers and applicants. Are applicants being evaluated by those similar to themselves or by those who are significantly different? In terms of age, geographical location, institutional affiliation, and so on, the major difference Hensler finds between the two populations is that 'applicants are somewhat more likely [at a probability level less than 0.05] than reviewers to belong to a more recent academic generation and to be currently located at a non-PhD granting institution.'

2

8 In general, applicants are being evaluated by their peers, if by 'peer' one means one who is similar in professional and demographic background. On appraising the peer review process, those participating as ad hoc mail reviewers (45 percent of the total respondents across all divisions [directorates] of NSF) saw the system as 'sound'; half saw it as an 'an acceptable peer review mechanism with some weaknesses', and only 4 percent saw it as 'a questionable peer review mechanism with many weaknesses'.

2

9 In comparison, of those participating as panel review members, 60 percent saw the system as 'sound'; 34 percent as 'acceptable with some weakness'; and 5 percent as 'questionable with many weaknesses'. Analysis of the scaled and open-ended responses generally supports the contention that 'reviewers' assessments of the peer review process based on their experience during the past two years are largely positive.'

3

0 The issue of whether particularistic factors may intervene in the evaluation process and influence the reviewer's recommendation to fund was broached by three variants on a single question in the Hensler survey: Given two equally good proposals except for one marked difference, which proposal did the respondents think had a better chance of receiving peer review recommendation to fund? In the first case, one of the proposals was from a well-known institution; the other from a lesser-known institution. In the second case, one of the proposals was submitted by a young, as-yet not established principal investigator (PI); the other, by an older, wellestablished PI. In the third case, one of the proposals featured approaches which were consistent 'with the mainstream of thought' in the discipline or research area; the other, a project which challenged the mainstream of thought. This last case, of course, concerns