Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History
180 RUDER I ER: T IIK I' A I.I ACY O l PEER REVIEW necessary funds. However, the isolation of the internal workings of the science establishment from public view has thus far obfuscated the fact that return on any investment in speeding the dissemination process can be matched by few, if any, other investments of our resources at present. Society spends large sums for research and development of specific projects initiated by scientists but negligible amounts on the publication bottleneck, the major path for dissemination of the knowledge so obtained which, in essence, determines the return to society on its initial investment. It is the anachronism of our times that, because of cost, many of the technological offsprings of the science process, e.g. modern communications, computers, fast publication techniques, psycho-social advances, opinion surveys, advanced management methods and priority mail, among others, are not being maximally employed to further enhance the process that gave birth to them. The present state of research publication may be generally compared to the proverbial shoemaker without shoes. 8. TOWARD A SCIENCE OF PEER REVIEW Peer review is such a multi-faceted, strictly human endeavour that the perennial question, "Does peer review work?" invariably results in a dialectic controversy. The most effective known approach to minimize this is an operational one — reasoning based on measurement — which requires the question to be reframed as "precisely how well does peer review work?" This stress on measurement switches the basic emphasis from disagreement to agreement. Once the parameters of peer review are properly measured, ways to improve it become self-evident through tests, further refinement, further tests, etc. — the prototype of the scientific method. This demands that we begin with study of the basic phenomena themselves, review case histories. The one presented here graphically confirms the need for publication of many more to enable a start in this direction. Nonetheless, it is also imperative to determine what we now glean from this one case. An expressly recognised goal of review is to preclude defective work; a heretofore neglected goal stressed herein is the necessity to also preclude erroneous rejection of publishable work. Erroneous acceptance is minimized by the current practice of parallel reviewing and although it is desirable to contain it, acceptance errors are ameliorated by the back-up practice of assigning high priority to correction of published errors. It may be further ameliorated by publishing referee comments, e.g. as by SST when warranted. However, erroneous rejection leads to conflict, which necessitates sequential reviewing, but this has no back-up error-correcting mechanism other than the unsatisfactory, lengthy and uncertain one of submission elsewhere. It is erroneous rejection that results in gross inequities in the review process, causes the most friction and dissatisfaction to authors and journals and produces the most serious consequences for society by slowing technological growth rate. By merely up-grading the precision of sequential reviewing a much improved system is obtainable with minimal disturbance to the present system. The primary parameter of sequential reviewing is the fraction p of contested manuscripts per review cycle (which may include one or more parallel reviews). Let N s be the number of submitted manuscripts and N be the
