Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation

102 CICHETTI: THE RELIABII

.ITY OF PEER REVIEW Cicchetti, D. V., Aivano, S. L. At Vitale, J. (1976) A computer program for assessing the reliability and systematic bias of individual measurements. Educational and Psychological Measurement 36:761-64. [aDVC] (1977) Computer programs for assessing rater agreement and rater bias for qualitative data. Educational and Psychological Measurement 37:195201. [aDVC] Cicchetti. D. V.. Lee, C., Fontana, A. F. Ac Dowds, B. N. (1978) A computer program for assessing specific category-rater agreement for qualitative data. Educational and Psychological Measurement 38:805-13. (aDVC) Cicchetti, D. V., Sharma, Y. At Cotlier, E. (1982) Assessment of observer variability in the classification of human cataracts. Yale Journal of Biology and Medicine 55:81-88. (rDVC) Cicchetti, D. V., Showalter, D. Ac Tyrer, P. (1985) The effect of number of rating-scale categories upon levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement 9:31-36. |aDVC) Cleary, F. R. Ac Edwards, D. J. (1960) The origins of the contributors to the A.E.R. during the fifties. American Economic Review 50:101114. (aDVC) Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46. (aDVC, RR) (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70:21320. [aDVC] (1988) Statistical power analysis for the behavioral sciences. 2nd ed. Lawrence Erlbaum. (rDVC, RR) Cole, J. Ac Cole, S. (1973) Social stratification in science. University of Chicago Press. (aDVC) (1981) Peer review in the National Science Foundation: Phase II of a study. National Academy of Science;. [aDVC] (1985) Experts' "consensus" and decision-making at the National Science Foundation. In: Selectivity in information systems: Sunyival of the fittest, ed. K. S. Warren. Praeger. (aDVC) Cole, S. (1978) Scientific reward systems: A comparative analysis. In: Research in sociology of knowledge, sciences, and art, ed. R. A. Jones. JAI Press. (aDVC) (1983) The hierarchy of the sciences. American Journal of Sociology 89:111— 39. |aDVC], SC Cole. S., Cole, J. Ac Simon, C. A. (1981) Chance and consensus in peer review. Science 214:881-86. (aDVC, SC, LLH) Cole, S.. Cole, J. Ac Dietrich, L. (1978) Measuring the cognitive state of scientific disciplines. In: Toward a metric of science: The advent of science indicators, ed. Y. Elkana, J. Lederberg, R. K. Merton, A. Thackray, Ac J. Zuckerman. Wiley. [SC] Cole, S., Rubin, L. Ac Cole. J. (1978) Peer review in the National Science Foundation. National Academy of Sciences. (aDVC) Cole, S., Simon, G. Ac Cole. J. (1988) Do journal rejection rates index scientific consensus? American Sociological Review 53:152-56. (SC) Colman, A. M. (1982a) Came theory and experimental games: The study of strategic interaction. Pergamon Press. (AMC) Colman, A. M. (1982b) Manuscript evaluation by journal referees and editors: Randomness or bias? Behavioral and Brain Sciences 5:205-06. [AMC] Conger, A. J. (1980) Integration and generalization of Kappa for multiple raters. Psychological Bulletin 88:322-28. (rDVC) (1985) Kappa reliabilities for continuous behaviors and events. Educational and Psychological Measurement 45:861-68. (rDVC) Conn, H. O. (1974) An experiment in blind program selection. Clinical Research 22:128-34. (aDVC) Cotlier. E., Fagadau, W. Ac Cicchetti, D. V. (1982) Methods for evaluation of medical therapy of senile and diabetic cataracts. Transactions of the Opthalmologic Societies of the United Kingdom 102:416-22. IrDVC) Cox, R. (1967) Examinations and higher education: A survey of the literature. Universities Quarterly 21:292-340. [DL] C

randall, R. (1986) Peer review: Improving editorial procedures. BioScience 36:607-09. [RC] (1987a) Cauntlet thrown: Publication procedures are challenged. Dialogue fy\PA Division 8) 1:5. (RC) '198~b) We need research on what constitutes good journal papers - and good editing - not guesswork on how to improve manuscripts! American Psychologist 42:407-08. (RC) (1990) Improving editorial procedures. American Psychologist 45:66566. [RC] Crane, D. (1967) The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist 32:195201. [aDVC] (1972) Invisible colleges. University of Chicago Press. [DLE] Cronbach, L. J. (1981) Comment on "Chance and consensus in peer review." Science 214:1293. [LLH] Culliton, B. J. (1984) Fine-tuning peer review. Science 226:140102. [aDVC, RC] Darley, J. M. At Latane, B. (1968) Bystander intervention in emergencies: Diffusion of responsibility. Journal of Personality and Social Psychology 8:337-83. [AMC] Darlington, R. (1980) Another peek in the file drawers (unpublished manuscript). [PHS] Davies, M. At Fleiss, J. L. (1982) Measuring agreement for multinomial data. Biometrics 38:1047-51. [rDVC] DeBakey, L. At DeBakey, S. (1976) Impartial, signed reviews. New England Journal of Medicine 294:564. [aDVC] Delucchi, K. L. (1983) The use and misuse of chi-square: Lewis and Burke revisited. Psychological Bulletin 94:166-76. [rDVC] Diamond, J. (1985) Variations on a theme. Nature 314:222-23. [aDVC] Dickersin, K. (1990) The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263:138589. [SPL] Doherty, M. E. At Tweney, R. D. (1988) The role of data and feedback error in inference and prediction. Final report for ARI Contract MDA903-85K-0193. [MEC] Eckberg, D. (1982) Theoretical implications of failure to detect prepublished submissions. Behavioral and Brain Sciences 5:25-26. [DLE] Eells, W. C. (1930) Reliability of reported grading of essay type examinations. Journal of Educational Psychology 21:48-52. [DL] Eichorn, D. H. At VandenBos, C. R. (1985) Dissemination of scientific and professional knowledge. American Psychologist 40:1301-16. [RFB] Eight APA journals initiate controversial blind reviewing (1972) APA Monitor, pp. 1, 5. [aDVC] Epstein, W. M. (1990) Confirmatory response bias among social work journals. Science, Technology and Human Values 15:9-38. [rDVC, MJM] Estes, W. K. (1975)-Some targets for mathematical psychology. Journal of Mathematical Psychology 12:263-82. [PHS] Evans. J. T., Nadjari, H. I. Ar Burchell, S. A. (1990) Quotational and reference accuracy in surgical journals: A continuing peer-review problem. Journal of the American Medical Association 263:135354. [JSA] Feinstein, A. R. (1987) Clinimetrics. Yale University Press. [rDVC] Feinstein, A. R. At Cicchetti, D. V. (1990) High agreement but low kappa: 1. The problems of two paradoxes. Journal of Clinical Epidemiology 43:54349. [arDVC] Feynman, R. P. (1985) Surely you are joking, Mr. Feynrrum. Bantam. [PHS] Finn, R. H. (1970) A note on estimating the reliability of categorical data. Educational and Psychological Measurement 30:71-76. [aDVC] Fisher, A. (1989) Seeing atoms. Popular Science: 102-07. [JSA] Fiske, D. W. At Fogg, L. (1990) But the reviewers are making different criticisms of my paper!: Diversity and uniqueness in reviewer comments. American Psychologist 45:591-98. [rDVC, JSA] Fleiss, J. L. (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76:378-82. [rDVC] (1975) Measuring agreement between two judges on the presence or absence of a trait. Biometrics 31:651-59. [aDVC] (1981) Statistical methods for rates and proportions, 2nd ed. Wiley. [aDVC, RR] Fleiss, J. L. At Cicchetti, D. V. (1978) Inference about weighted kappa in the non-null case. Applied Psychological Measurement 2:113-17. [rDVC] Fleiss, J. L. Ar Cohen, J. (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 33:613-19. [aDVC] Fleiss, J. L. Ar Cuzick, J. (1979) The reliability of dichotomous judgments: Unequal numbers of judges per subject. Applied Psychological Measurement 3:537-52. [rDVC] Fleiss, J. L., Cohen, J. At Everitt, B. S. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72:323-37. [rDVC] Fleiss. J. L.. Nee, J. C. M. At Landis, J. R. (1979) Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin 86:97477. [rDVC] Freeman, C. At Tyrer, P., eds. (1989) Research methodology in psychiatry: A beginner's guide. Royal College of Psychiatrists/Gaskell Books. [PT] Fuller. S. (1989). Philosophy of science and its discontents. Westview Press. [MEC] Furchtgott. E. (1984) Replicate, again and again. American Psychologist 39:1315-16. [aDVC] Garber, H. L. (1984) On Sommer At Sommer. American Psychologist 31:1315. [aDVC]