Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History

172 RUDER

I ER: T

IIK I'AI

.I ACY O

l PEER REVIEW "The referee system here apparently does what it is supposed to do: Sift out the good papers from the bad." In other words, for this "low-rejection journal" the system worked somewhat better than chance. Reassuring as this may be, it is not unexpected. The success of modern science implies that its review system must have thus far been statistically effective to some significant degree. But if the actual rejection error rate on publication decisions in this sample is expressed as 1 in 10", what is the precise value of n? The upper limit to n is log 14,512 = 4.2, but its actual value is not possible to ascertain from such a contemporary study. Some revolutionary ideas have been widely rejected after publication for 25 and 50 years, as for Maxwell's electromagnetic theory and Wegener's continental drift theory, respectively, so the possible rejection of just one revolutionary unpublished advance in the sample cited cannot be reliably determined for at least a comparable period. The Zuckerman and Merton analysis provides a lower limit to n of the order of 1 in accord with a 20 (5) percent rejection rate for single (multiple) authors.'

78 > This and the upper limit of 4.2 may be far too low to insure that key ideas are not passed by for excessive periods. Premature rejections belie the vaunted exhaustiveness of science. How many other key innovations essential to our long-term survival, as those of Carnot, Gibbs, Goddard, Mendel and others'

6', have long been ignored for the wrong reasons? What is the distribution of number of such rejections versus delay in acceptance? The history of innovation indicates that it must be quite skewed with a tail that may asymptotically approach zero. This is in contradiction with the prevalent view stated, e.g. by Cole and Cole'

1

4 > that if a "scientist who makes a discovery had not made it, it would have been only a matter of time — probably a relatively short period — before the discovery would be made by another scientist". This popular notion is actually unverifiable and therefore untenable as a working hypothesis for a study of the review process. There is no evidence that n is ideally infinite over the still uncertain life span of modern science; rather, the life span must be generally assumed to be interrelated with n. Its measure is a vital one for exploring means to insure long-term survival. The current pressure on technology to solve the problems wrought by the population explosion in this century also demands increasing attention to time factors in the review process. Population growth has heretofore been governed primarily by immutable biophysiological factors; growth of knowledge is governed by unrelated and little understood psycho-social factors. The usual answer to the Malthusian doomsday predictions is the expectation of new advances by the "technological optimists".'

1

6' But if the disorganising effects to society of the population explosion reach the stage wherein they physically impair the ability to solve technical problems, a worldwide catastrophic situation develops. The current state of civilization suggests that such a process may have already begun and that the possibility of eventual self-destruction cannot be positively eliminated. It is consequently fail-safe to insure that the technological growth rate exceeds the growth rate of the disorganising effects of the population explosion. To guarantee the required technological solutions we need to minimise unnecessary delays in the growth of knowledge. What are the causes