Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
ALAN L. PORTER and FREDERICK A. ROSSINI: Peer Review of Interdisciplinary Research Proposals
PORTER «Sc ROSSINI: INTERDISCIPLINARY RESEARCH PROPOSALS 159 other hand, in earlier work, we encountered academic departments that presented serious obstacles to their members trying to include skill areas beyond their discipline's expertise in their research. 1 3 If these findings generalize across NSF, then some of the fears expressed formally and informally by members of the research community have merit. Brooks reports that peer evaluations in areas other than basic reseatch are much less satisfactory. 1 4 Roy cites peer review as "fundamentally and ineluctably anti-innovation." l s Grover Whitehurst notes variation across fields in implementation of peer review." Present data suggest that peer review of NSF proposals favors research that is performed by academics, in the sciences, and that falls completely within the reviewer's own domain of expertise. Well-established research areas are thus favored over nascent ones. Compensatory mechanisms to counterbalance these inclinations may be warranted. Our most intriguing finding offers a clue as to why interdisciplinary proposals are downgraded. It is reasonable for a reviewer of proposed research to favor that which is more familiar personally. In such a case, one is apt to understand bettet what is planned; one may know the researchers personally or by reputation, and hence appreciate their expertise; and one can feel more secure in making strong recommendations. One program manager summarized his experience as demonstrating that reviewers often considered proposed work from the "standpoint of only their discipline." The marked tendency of reviewers to rate proposals from Pis from the reviewer's own discipline more favorably suggests that IDR should not be reviewed the same way as disciplinary projects. In view of this serious concern, consideration should be given to alternative strategies for reviewing IDR proposals. One proposed approach of composing a review team that includes a reviewer who knows one aspect of the proposed research, another who is expert in a second area, and so on, implies a set of reviewers each unfamiliar with much of the work." According to our present findings, that means an expected downgrading of the proposal in comparison to one of equivalent merit, but for which individual reviewers can better grasp the full scope of the research involved. To give interdisciplinary research proposals the same opportunity as disciplinary proposals for a good rating, one has several choices. One option is to lower rating standards, but that is difficult to justify within programs. NSF's current policy places applied research projects in basic research programs. When basic and applied research proposals fall within one program, peer reviews are likely to favor the basic science proposals. Our results further suggest that interdisciplinary research proposals within such a program will be at a competitive disadvantage. In essence, using peer review to choose among different types or areas of research is ill-advised. Program managers, on occasion, collaborate to consider funding a proposed project that is too large or broad in scope to be supported by a single NSF program. When this happens, sometimes "turf" issues arise. The obvious solution to such issues —cross-program review —entails additional jeopardy for the proposer (e.g., even more reviewers expert in only part of the proposal, or multiple panel reviews]. Another possibility is to seek reviewers who are expert across the breadth of the proposed work. As Martha Russell notes, 'To the extent that reviewers have a holistic perspective of knowledge creation and use, they can offer their best guesses as to the success of the proposed (interdisciplinary] research and thereby assist in screening projects which are likely to be productive."" Such reviewers may be difficult to secure for "frontier" research, and there is danger of inbreeding in cases where only a small number of potential reviewers share interests fully. Panel meetings offer an advantage in allowing discussion to help each of the members understand unfamiliar aspects of a proposal under review. The remarks of some panelist reviewers support such a strategy as they report "changing their mind after discussion." However, panels are costly and must be constituted to address sets of proposals. Novel interdisciplinary research is likely to fall on the fringe of panel expertise and hence to face a poorer expected rating than more mainstream, disciplinary research, with ail other conditions being equal. We suggest that incorporation of feedback in the mail review process could help remedy the problem of restricted reviewer expertise. Roy reports that no substantial feedback mechanism is formalized in any U.S. granting agency. 1 9 We suggest extending the provision that is sometimes used of providing the PI an opportunity to respond to the concerns of reviewers before a final funding decision is reached. Explicitly, we advocate a
