Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

53 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW As we shall also see, the problem in the physical sciences is a quite different one. If the author is told that the subject is inappropriate for the journal to which it has been submitted, a search should be made for a more appropriate alternate journal. The last two reasons for manuscript rejection (i.e, lack of originality or lack of journal space) seem somewhat subjective. A letter to the editor would accordingly seem pointless, and again, alternate journals should be considered. 7.9. Developing a peer-review appeals systems for grsnt submissions. Both the National Institutes of Health (NIH) and the Alcohol, Drug Abuse, and Mental Health Administration (ADAM HA) granting agencies have recently established formal peer-review appeal systems. According to Holt (1985), classes of problems that might justify an appeal include: the granting agency's refusal to accept an application; an applicant's disagreement over the assignment of his grant (whether to a specific study section or Institute); an author's doubts about the level of knowledge of specific study section members; or evidence of bias associated with the peer review. In a survey reported by Mitroff and Chubin (1979 p. 222), more than 70% of NSF applicants favor what is referred to as a "formal" appeal system as a remedy for mistakes. A similar process is in operation at the NSF, whereby a principal investigator (P.I.) whose grant has been disapproved can call the designated program officer, who will provide a detailed account of the specific reasons for disapproval. If the P.I. is still dissatisfied, other options are available, including a request that the Assistant Director of NSF reconsider the grant disapproval on the basis of arguments set forth by the P.I. These developments portend a healthier climate for handling legitimate claims about perceived unfairness or incompetence; they become especially important as less and less funding is made available to support worthy research endeavors.

1

0 8. Concluding comments Given that rejections of manuscripts and disapprovals of grant proposals can seriously affect one's research career (e.g., jeopardize or delay a potential promotion), what conclusions can be drawn from this report? As Hamad (1986, p. 24) notes, this depends to a large extent on what one views as "closer to the truth": (a) that most published research is either "significant and essential to the progress of science"; or (b) that most of it is "neither significant nor essential." Harnad adds that there is some evidence to support both propositions. If one believes (a), then one becomes very concerned that at least some meritorious research will be rejected, disapproved, or delayed unnecessarily, thereby hampering scientific progress. On the other hand, if one believes proposition (b), then the potential problem of false negatives (rejecting meritorious scientific documents) becomes less significant. For if most research Is unimportant and leads nowhere, then it matters less if some of it is rejected or delayed; indeed, this would even come as a welcome relief to many who, like Lock (1985), are deeply concerned about the literature glut. Our own view tends toward proposition (a), especially with respect to the peer review of grants. It has recently been noted that although more and better scientists are being produced each year, the funds available to support research are not keeping pace (e.g., Ison 1985; Koshland 1985). In a debate reported in a Science editorial by Culliton (1984), NIH director James B. Wyngaarden addressed the problem he refers to as distinguishing '"shades of excellence' among competing grants that are all at the top." He went on to state that: in many institutes, there is money enough to fund those grants with top priority scores of 160 to 170, while those rated only slightly lower at 171 to 180 end up in the reject pile. Nearly everyone agrees that there is no objective way the peer-review system can make such fine-tuned distinctions about quality, (p. 1401) Consistent with this statement, the current editor of Science notes that "with quality as high as it is today, and funding low, a committee of Solomons would have difficulty distinguishing between grants that should and should not be awarded" (Koshland 1985, p. 1387).

11 Given the decreasing availability of research funds, the current situation has become markedly more competitive, with NIH grants usually requiring priority scores of 125 or considerably less, in order to assure funding. The inability of referees to make "fine-tuned" distinctions also affects the peer review of manuscripts submitted to prestigious scientific journals (e.g., the New England Journal of Medicine (NEJM), the Journal of Clinical Investigation (JCI) and Science). Wilson (1978) reported that of all manuscripts rejected by JCI in 1970, 85% of them were later published elsewhere: "The journals in which these papers were published constitute a distinguished list of publications, with 14 journals accounting for one-half the papers." Similarly, the editor of the NEJM noted that 85% of his journal's rejections in 1975 were subsequently published or accepted for publication elsewhere (Relman 1978). Approximately 70% of these initially rejected manuscripts appeared in very distinguished medical specialty journals, general medical research journals (i.e., Journal of Clinical Investigation (JCI), Journal of Laboratory and Clinical Medicine, Journal of Applied Physiology), or general medical journals (i.e., Journal of the American Medical Association (JAMA), Lancet, British MedicalJournal(BMJ), Canadian Medical Association Journal, Medical Journal of Australia). The remaining 2Í4 articles were published in local, state, and other types of journals. It should also be noted further that of the 85% initially rejected articles, both for NEJM and JCI, the majority were either not changed or changed in only minor ways in their ultimately accepted versions. Similar results were published by Lock (1985, p. 67): 79% (or 1223) of the manuscripts submitted to the BMJ in 1979 were rejected; 68% of these "were published elsewhere, 15% in high-impact-factor journals, and 10% in high-impact-factor general journals" (based on the average number of citations of the published articles). The results of these three studies are consistent with those of a MEDLINE (computer) search reported by the former editor of Science. Abelson (1980) noted that although Science rejects about 80% of submissions to the journal, "almost all of our rejected material has appeared in other journals" (p. 62). Finally, Garvey et al. (1979) have reported that the bulk of manuscripts rejected by social science journals are also subsequently published,