Braun Tibor, Schubert András (szerk.): Szakértői bírálat (peer review) a tudományos kutatásban : Válogatott tanulmányok a téma szakirodalmából (A MTAK Informatikai És Tudományelemzési Sorozata 7., 1993)
ANGELO S. DENISI, W. ALAN RANDOLPH and ALLYN G. BLENCOE: Potential Problems with Peer Ratings
163 D E NISI & AL.: POTENTIAL PROBLEMS WI TH PEER RATINGS worked in one of 34 small groups of 3 to 5, depending on the number reporting for a particular session. Preliminary analyses indicated no effects for subject sex, the sex composition of the groups, or any sex x condition interactions. In addition, there were no effects for group size. Thus all results are presented for the total sample. On reporting, subjects were given a general description of the procedures to be followed and were told that they would be working as a group on two tasks. Following each task they would be asked to complete a questionnaire that included an evaluation of every other member of their group by name. They also were informed that each person would be shown the evaluations, but only the mean rating received, so that individuals could not tell how any one person rated them. (Of course, in smaller groups, a group member receiving negative feedback would know that no peer could have rated him or her as an outstanding performer.) Subjects also were informed that two persons would be observing their group from behind a one-way glass, but the observers' only role was to record group interactions. After questions were answered, subjects were asked to sign informed consent forms. There was no penalty for refusal to participate. Alternatives were available for students to satisfy this portion of their course requirement. The two group problems were variations of a truck routing task developed by DeNisi and Pritchard (1978), which required subjects to map a crosscountry route to maximize, within certain constraints, the value of the cargo trucked. The two variations were pretested to insure equivalence. This particular task was appropriate for the present study because several distinct pieces of information had to be considered, and up to five people each had to assume a real role within the group. The task also required coordination among group members so that members had continuous interaction. It therefore was impossible for any one person to perform the task alone, and it was difficult for any one member to be disassociated from the outcome, which clearly is a group product. Finally, based on extensive pretesting of the original task, subjects had no feel for the number of points (measuring the value of the cargo) that were needed to represent good or poor scores. Thus, there was little potential for task-generated feedback that could interfere with the manipulated peer feedback (discussed below). Peer rating feedback was given after subjects completed all measures following the first task, and till peer feedback was false. No feedback was given following the second task. Instead, after completing the questionnaires and the peer ratings forms a second time, subjects were debriefed and dismissed. Measures 1. Task and Socioemotional Behavior. Bales' (1950) interpersonal process analysis (IPA) form was used for assessing group interaction during each task. Two independent observers, blind to feedback conditions, rated each group on all 12 of Bales' categories for each task, recording evaluations
