Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
ANGELO S. DENISI, W. ALAN RANDOLPH and ALLYN G. BLENCOE: Potential Problems with Peer Ratings
166 D E NISI & AL .: POTENTIAL PROBLEMS WI TH PEER RATINGS Table 1 Means and Standard Deviations for All Variables at Time 1 and Time 2 for Positive and Negative Feedback Groups and the Results of Several Comparisons Negative Feedbac k Pasitive Feedbac k Time 1 Time 2 Time 1 -Time 2* Timet Time 2 Time 1-Time 2» Observer measures Socioemotional behavior 2.84 2.56 <1 2.75 2.75 <1 (1.08) b (.56) (86) (.50) Task behavior 4.76 4.52 <1 5.20 5.28 <1 (.72) (.50) (.75) (.68) Perceptual measures Perceived performance 6.21 5.64 5.28** 6.24 6.74 <1 (.89) (.81) (1.46) (.82) Cohesiveness 5.39 4.95 4.18* 5.70 5.95 1.02 (.92) (.42) (.79) (.77) Satisfaction 5.63 5.09 3.85 5.77 5.81 <1 (1.02) (94) (1.04) (.88) A verage peer rating 5.33 4.42 6.08** 5.64 6.20 7.42' given (1.40) (1.09) (1.13) (.81) Actual performance 39.00 37.50 2.81 40.56 40.94 <1 (3.62) (4.94) (5.24) (3.97) •These are the results of simple effects analyses comparing ratings at time 1 and time 2. ^Standard deviations are shown in parentheses, •p < .05 ••pc.01 to 2 = .09). Thus, knowledge of peer ratings did affect most of the dependent variables of interest, although rated socioemotional behavior and objective performance did not seem to be affected. To test the exact nature of these effects over time, for subjects learning of positive and negative peer ratings, a series of simple effects analyses, comparing time 1 and time 2 means within feedback groups, was conducted. Winer (1975) suggests that this type of analysis is more appropriate than simple r-tests, given the evidence of sign of feedback X time interactions, and the tests were conducted using formulae provided in his book. As predicted, subjects learning of positive peer ratings raised scores on all perceptual measures from task 1 to task 2, collected more points on the second task, and were rated higher on task behavior for task 2. However, only one change was significant (average peer rating given: F=7.42, pc.Ol). Also as predicted, subjects learning of negative peer ratings lowered scores on all perceptual measures from task 1 to task 2, collected fewer points on the second task, and were rated lower on task and socioemotional behavior for task 2. These changes were significant for perceived performance (F=5.28,p< .05,) cohesiveness (F=4.18,/?< .05), and average peer rating given (F=6.08,p<.01). Thus, these results support the predicted stronger effect of learning about negative peer ratings. Discussion The results from this study seem to raise some doubts about the use of peer ratings for feedback, but the limitations of the present study are
