Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
RUSTUM ROY: Alternatives to Review by Peers: A Contribution to the Theory of Scientific Choice
151 ROY: ALTERNATIVES TO REVIEW BY PEERS who are less likely to be biased. The reviewing group should be drawn by random-number selection by computer from a qualified set selected for each subdiscipline, and should include one or two from related disciplines. The qualified sets of reviewers would be revised annually by a committee of the National Research Council. In general, it might be valuable to have small groups of individuals in roughly the same field all evaluated by the same group; this would provide a basis for comparative judgements. The best group of peers is obvious: it is the group of scientists in the same general fields in the many similar institutions in foreign countries. The data presented to the assessors would be a curriculum vitae, up-to-date bibliography, and perhaps a two-page statement by the individual being assessed summarising his or her recent achievements in research. Again, this kind of judgement is hardly alien to academic traditions. It is made daily in appointive decisions in universities and industry. The peers would rank their evaluation of the quality, quantity and the originality of the research and its value to society. The actual budget allocations could be left to the managers since this would be dependent on the subfield of study. International Evaluation by Peers of Established Groups This is a variant of the scheme for individuals applied to departments or other research units. It is in widespread use now in many parts of the world. In its ideal form, unfortunately, it cannot cope with large numbers of institutions. For example , to allocate funds among half a dozen bioengineering groups, each group would prepare a statement of recent achievements and provide the data on its productivity over the preceding five years. The international group of assessors would both evaluate each group's "productivity", compare the productivity of the groups in the set, and point out possible overlaps, gaps, etc. Ideally, the peer group would visit each of the units, assess its capacity "on the ground" and provide a much more precise recommendation regarding the relative merits of each unit working on the subject to the grant-awarding body. Whenever such assessments have been used, they have been regarded as eminently fair. Clearly it is impractical to have the same group of assessors visit 100 or even 25 institutions. A postal review alone could perhaps be done for 20 or 25 by the same set of assessors, but several sets of assessors may be needed when large sets of institutions must be evaluated. Visits to sites by an international peer group are clearly the best way to evaluate the smaller sets of specialised research institutes, such as water research institutes and materials research laboratories. It is again a mark of the conservatism of the management of American research institutions that this has never been done. It is a pity, since the international nature of the reviewers would very properly reflect the realities of modern science.
