Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science and a Case History
182 RUDER I ER: T IIK I' A I.I ACY O l PEER REVIEW reviewers for the necessity to attain agreement, a spontaneous improvement in review precision and speed should ensue. (b) More circumspect and rigorous argument is encouraged if all reviewer comments are subject to the possibility of publication. The role of anticipation of rejection in enhancing author performance, commonly advanced to support peer review, is thereby extended to reviewers. Occasional publication of selected examples for tutorial purposes may serve to additionally educate the science community. A special journal, as suggested by Commoner' 1 2 >, may be warranted. (c) Investigate the possibility of classifying the significance of all manuscripts from evolutionary to revolutionary on a scale of, say, 0 to 10, and assign a suitable weighted mean R derived from author, reviewer and editor estimates. Besides the value of R as a caution flag for properly resolving the more innovative approaches, determination of the potentially useful p-R distribution function may be facilitated. (d) The inherent role of the editor as adjudicator is too often degraded to a clerical role, as in this case history, so that reviewers are tacitly assigned the dual role of prosecutor and judge. In a dispute the reviewer is no longer a "referee" but a contestant and should be so regarded. If the editor, or a designated impartial arbitrator, does not exercise a supervisory role, e.g. akin to a referee in a sports contest, the result is an increase in the incidence of arbitrariness, as in M, and rejection for the wrong reasons, as in C, D and N. (e) Because of the express need to resolve a dispute, it is expedient to require reviewers to indicate whether and/or how author errors may be corrected to allow acceptance where possible. Suggestions from reviewers are not the rule, but are often very useful to authors, even for language usage as by N. Besides quickly resolving a dispute they induce respect for peer review' 3 3 > and should be made mandatory by editorial dictum. (f) For a more serious dispute the simple expedient of explicitly delineating and narrowing the boundary of a disagreement provides a useful resolution vector. (Of course, all prior information must be meticulously forwarded to reviewers and authors.) In the case history five reviews were not made available to the author and there was no attempt in the final review stage to relate to the prior reviews. One reviewer (M) reneged and the other (N) went off in a different direction. Delineation of the boundaries of a dispute may be forced by editorial edict, as by demanding authors and reviewers to indicate agreement or disagreement on all segments of reports on suitable forms that must be returned with comments. (g) In the event of an impasse a "closed-loop" review may be instituted by requiring author and reviewer to communicate directly, with copies sent to or through the editor. Speedier resolution is facilitated by the higher information transfer rate, but close supervision by editor or arbitrator is essential to prevent degeneration, e.g. as for a boxing match vis-a-vis a bar-room brawl. An example of a successful application is mentioned in 0. Useful authorreviewer dialogues with maintenance of anonymity have also been instituted by Chest' 1 3'. The still higher information transfer rate allowed by telephone suggests a further extension worth investigating.
