Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History

171 RUDER

I ER: T

IIK I'

A

I.I ACY O

l PEER REVIEW should be deemed feasible. How often it has actually occurred is immaterial; what is significant is that there is no sure recourse to authors to correct a defective rejection or, in the case of deliberate malpractice'

1 4'

5>, any form of redress. The absence of negative feedback is further aggravated by the high positive feedback inherent in the dissemination of scientific ideas. When published research in a new area stimulates further interest, this leads to further publications, still further interest, etc. Although such a snowball effect is useful in rapidly exploiting breakthroughs, the net long-term effect of such a happenstance type of growth has never been fully evaluated, especially in regard to the consolidation of tacit unrigorous assumptions that may unknowingly usurp other, more superior, courses of develoDment. Moreover, it is also well known that the overall effect of positive feedback in any system is rapid uncontrolled growth and long-term instability. The incredible short term gains man has achieved in the relatively brief life of modern science are now beginning to spin off numerous long-term problems, especially from the population explosion it has stimulated. The crucial role of the referee system in delaying the response time of the science establishment to meet the challenges to society induced by past technological successes has already been noted.'

6

> The most damning indictment of the present referee system is that the haphazard growth it has fostered has thus far provided no capacity to reliably ensure a stable final state toward which science, and hence civilization, is headed. In short, there is no positive proof or assurance that modern science cannot selfdlestruct. It is primarily for this reason that it is timely to place the present error-prone open-loop referee system under the microscope. 2. QUANTITATIVE EVALUATION OF PEER REVIEW The application of the scientific method — reasoning based on experience — to peer review has always been hampered by the lack of adequate raw data. Every established science is based on measurement of some observable object or phenomena in nature. In a science of peer review, the relevant phenomena are the details of the review process itself, specifically the total communications involved — submitted reports, review reports and all other interchange of information affecting the final decision. However the secrecy heretofore imposed on reviews has precluded dissemination of such data for quantitative evaluation. As a result, prior discussions, criticisms and studies of peer review and suggestions for improving it'

1 3"

1

3>, although often incisive, are anecdotal, subjective and/or limited. To rectify this a sufficient number of case histories must be fully exposed to view so that the variegated factors contributing to review decisions can first be systematically analysed and measured. It is only then that ways to improve the review process can be realistically instituted with confidence. With this objective in mind, such a case history is presented below. However it is first useful to explore the kinds of information required from such data. Despite the scattered published criticisms of peer review, the prevailing view is that it works. But to what precision? In an outstanding quantitative study from analysis of the records of 14,512 manuscripts submitted to The Physical Review from 1948 to 1956, Zuckerman and Merton'

7

8* concluded,