Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History
181 RUDER I ER: T IIK I' A I.I ACY O l PEER REVIEW number subjected to review. Then N s - N is the number rejected primarily for nontechnical reasons, as nonconformance to editorial standards. After m sequential review stages the number that remain contested becomes PiP 2. . . p m N. If <p> is a mean value for all stages, the lower limit to n defined above in the introduction is then given by <p> m = 10" n (4) For (p) = 0.01, n = 2m. For m = 2 (3), the occurrence rate of unresolved conflicts is 1 out of 10" (10 6) reviewed manuscripts. Such examples clearly define the goal of sequential reviewing: (i) The value of p must be minimized, (ii) Use m stages to obtain any desired maximum rejection error rate, i.e. desired minimum value of n. (iii) Prevent p from degenerating between stages. The lack of attention to rejection error in present peer review is not conducive to a minimum value of p or its uniformity between stages. The result is a high conflict rate, e.g. as in Physical Review Letters 19 A0 ) • An intrinsic cause is suggested by this case history to lie embedded in the properties of the human mind, as summarised by the Innovation Theorem. Since minds are immensely varied and their modes of operation are still unknown, broad generalisations, theories and panaceas for improving peer review do not yet have an adequate operational basis. The alternative is the empirical one of identifying each specific source of human error and devising appropriate means to minimize it. This approach demands intimate knowledge of review details, as afforded by case histories. Heuristic conclusions deriving from this one case history include the following: 1. The decision to reject was not based on a technical resolution of the conflict but on the prevalent criterion of consensus of the referees. This criterion was not fail-safe because the basic issues were by-passed. No decision should ever be forced when technical disputes remain unresolved. Elimination of this major source of rejection error is simply obtained by formally defining a contested review to be considered complete only when there is agreement between author and reviewers. Decision to accept or reject is then automatic and anticlimactic. 2. Such a definition shifts the traditional emphasis on erroneous acceptance, which has built-in error correction mechanisms, to at least equal emphasis on erroneous rejection, which has substantially none. Preoccupation with erroneous acceptance stems from the birth of modern science when there were no precedents for publication norms except the empirical need to establish order and rigour.' 7 > Today the norm of scientific methodology is well entrenched but the old tradition lingers on. Must we wait for a worldwide crisis to realize that the needs of yesterday are reversed by the needs today for a growth of science independent of its own dogma, traditions and preconceptions and a concomitant need to publicly acknowledge all unfalsifiable innovations and dissident views as rapidly as possible? To expedite the desired resolution of author-reviewer disagreement we need only institute appropriate specific measures, such as : (a) If public recognition and education by the journals for the consequences of erroneous rejection increase general awareness of authors and
