Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)

ANGELO S. DENISI, W. ALAN RANDOLPH and ALLYN G. BLENCOE: Potential Problems with Peer Ratings

171 D

E

NISI & AL.: POTENTIAL PROBLEMS WI TH PEER RATINGS this was to insure that each subject recorded only his or her impressions in the questionnaires. Observers were asked to watch and listen for an> such discussions and to remind subjects of the instructions (using micro phones from behind the one-way glass) if anyone did begin to discuss the questionnaires or ratings. No one discussed either with fellow group members. The experimenter returned to the room and gave each subject the false average peer rating (for overall performance) in a sealed envelope. Subjects were instructed to read but not discuss the feedback and were quickly given instructions for the second task and told to begin work. The sealed envelopes did not contain the subject's true average peer ratings. Instead, all persons in positive feedback groups were informed that they had received an average rating of about 6.0 (7.0 was the highest rating). All those in negative feedback groups were informed that they had received an average rating of about 2.5 (1.0 was the lowest rating). No feedback was given following the second task, although subjects did complete the peer ratings and the other questionnaire measures. In addition, subjects were asked to recall the average peer rating that they had received following the first task, as a form of manipulation check. All were able to recall their average rating within .1 of a point. Discussions with subjects during debriefing indicated that they did believe the feedback received. Results It was hypothesized that knowledge of peer ratings would affect group member interactions, perceptions, performance, and subsequent peer ratings and that the nature of these effects would depend on the sign of the peer ratings. A series of analyses therefore was conducted considering the sign of the peer rating feedback (either positive or negative) and time (after task 1, before feedback; or after task 2, following feedback) as independent variables, and the following dependent variables: task and socioemotional behavior exhibited (rated by observers), cohesiveness, satisfaction, perceived performance, actual performance (total points collected), and average peer rating given. Table 1 presents the means (and standard deviations) for all dependent variables, broken down by time and sign of peer rating feedback. Because the various dependent variables were correlated, a multivariate analysis of covariance (MANCOVA) was conducted. It revealed a multivariate effect for the sign of peer feedback (F=4.53, p< .01). A series of univariate analyses of covariance (ANCOVA) was then conducted on time 2 ratings with time 1 ratings as covariates, following Huck and McLean (1975), after an initial test indicated that common slope could be assumed (F< 1). The ANCOVA results indicated significant (p< .05, or better) peer feedback effects on perceived performance (F=7.32, w

2 = .ll), cohesiveness (F=6.24, w

2=.10), satisfaction (F= 10.65, w

2=.16), average peer rating given (F= 16.50, w

2 = .20), and rated task behavior (F= 5.62,