Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
RUSTUM ROY: Alternatives to Review by Peers: A Contribution to the Theory of Scientific Choice
152 ROY: ALTERNATIVES TO REVIEW BY PEERS An Optimal Review of Proposals by Peers Since the smaller the change the more likely its adoption, I include for completeness an alternative to the present system, modifying it in simple ways which would eliminate some of the major objections to the present system and which would not be more costly. From the outset, it should be made clear that the system seeks to evaluate the proposer and the ideas, and that each will be weighted equally or that greater emphasis will be laid on the former. Hence, the proposal should present in detail the recent research achievements of the programme and provide a short general indication of the work to be attempted. The review form must explicitly explain the two different evaluations being requested. Two simple sets of precautions will avoid conflict of interest by selection of assessors from industry and government, as I have already recommended, for all academic applicants. One can very easily avoid manipulation of the selection of referees by the manager of the programme by using a randomised selection from a qualified set of referees designated by the National Research Council or a similar body. A simple legislative enactment would solve this problem once and for all. For judgements regarding the allocation of public funds, the proportion of peers drawn from industry, governmental and academic laboratories should reflect roughly the national percentages of working scientists in the particular discipline throughout the country. The instructions to assessors should be very clear —as they are not today. They should be designed by experts in the preparation of questionnaires. The instructions should include the probabilities of the outcome associated with each level of rating. Thus the assessor should be informed that, say, an average rating of B— results only in a 5 per cent probability of support. A — in 75 per cent and an A+ in 95 per cent probability of support. Similarly the exact weighting to be placed on the applicant's achievements in research and on his proposed research should be indicated unambiguously. Once the assessments are received, they should be sent to the applicant for technical rebuttal, to be completed within exactly two weeks and with only one cycle of such rebuttal permitted. This avoids absurd errors in the referees' judgements. 1 4 The Dutch government uses this system. It is clear that we have absolutely no guarantee that a "principal investigator" will actually do the research he has proposed and whether it will be "successful", and if so, whether it will be of any value to science. Hence, it is absurd to have a sharp disjunction between successful and unsuccessful applicants in the distribution of funds. The budgets should be 1 4 Recently one of our proposals was classified as border-line and turned down by the National Science Foundation, although one of the reviewers' main reasons for a lower rating was that in an interdisciplinary proposal we had not included Professor X., the best known person at Pennsylvania State University. The reviewer might have had a (joint had Professor X. not been dead for over two years. The National Science Foundation . while embarrassed . had no way to correct the error because the process by then had moved to the next stage.
