Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History

170 RUDER

I ER: T

IIK I'

A

I.I ACY O

l PEER REVIEW anonymous referees of ostensible expertise. Yet, although author reports are required to approach the peak of scientific methodology, referee reports which judge them have had no such explicit restraint — maximum secrecy, unprescribed rigour, lack of precise judgement standards, cursory investigation and no guaranteed impartiality. Credibility of referee reports rests mainly in the belief, often just a hope, that the referee is truly a peer for the material being judged and that such peer knowledge has been adequately applied.

(1) No matter what justification is cited for peer review, an ironic truth remains: The scientific method is not being rigorously applied in the process of selecting those examples of the scientific method deemed worthy of preservation. If this seems harsh, consider the effect of a referee error that results in a firm rejection. The only recourse for the author is to submit again elsewhere. The chances of subsequent acceptance are best if his thesis is evolutionary, for the error is then more likely to be localised, i.e. peculiar to the referee. If his thesis is revolutionary, the error is probably more widespread,»!.e. peculiar to established doctrine, and the probability of publication within his lifetime is minimal. Although uncommon, revolutionary ideas have played a prominent role in the development of science. There is no specified error-correction mechanism in peer review for insuring that any valid development, evolutionary or revolutionary, is not disregarded for an excessive period or is forever overlooked. Rejection errors have far reaching adverse consequences for the research community. They result in needless additional time and effort caused by resubmissions, increase the total referee load on the journals, reduce the time spent by authors and referees on primary projects and, especially, have disastrous psychological effects on authors. Mayer and Boltzmann, for example, were so depressed by rejection it was a contributing factor in their attempted suicides.'

2 > Boltzmann succeeded but Mayer only broke his legs, was confined to a mental institution for a while and was finally recognised for his work on energy conservation after a loss to society of about 15 years. At the other extreme is a recent case known to the author of complete abandonment of research because of disillusionment with the review system.'

3 > Between these extremes lies a spectrum of psychological effects which rob society of the full potential of many of its most creative members. Measurement of the total cost of review errors has been ignored for too long. Gross inequities from rejection errors are inherent in the open-loop nature of the present review system. This is contrary to the existence of closed-loop negative feedback systems in almost every facet of society for correcting major errors, viz. separation of executive and judicial functions; law enforcement; elections; elaborate court systems for redress; arbitration; open refereeing, as in sports contests; ombudsmen, forums; etc. The lack of any prescribed errorcorrecting mechanism built into the referee system brands it as one of the most autocratic in society today. An author may spend months, years and even a lifetime in preparing a manuscript, yet under the present system it may be arbitrarily rejected by a referee after a cursory reading, akin to the flick of the wrist used by some Nazis in deciding acceptance or rejection for survival. Arbitrary action is not precluded by the present system; in science what is not expressly impossible