“Now that I have you thoroughly confused, let me pause to hear your own dismayed cry.” Â -Ray Bradbury
360-Degree Feedback Research
Current research suggests that rater agreement will indeed vary in 360-degree feedback assessments largely because feedback research shows that different rater sources provide unique, performance-relevant information (Nowack, 2009 ((Nowack, K. (2009). Leveraging Multirater Feedback to Facilitate Successful Behavioral Change. Consulting Psychology Journal: Practice and Research, 61, 280-297.)); Lance, Hoffman, Gentry & Barankik, 2008 ((Lance, C.E., Hoffman, B.J., Gentry, W. & Baranik, L.E. (2008). Rater source factors represent important sub-components of the criterion construct space, not rater bias. Human Resources Management Review, 18, 223-232))).
Given these findings, vendors who do not provide a way for participants to evaluate within-rater agreement in feedback may increase the probability that average scores used in reports can be easily misinterpreted, particularly if they are used by coaches to help participants using 360-degree feedback assessment focus on specific competencies and behaviors for developmental planning purposes.
Participants should reflect on why specific behaviors might be perceived and experienced positively by some raters and negatively by others. Â A large discrepancy by raters often suggests a polarized perspective and one that might require additional information gathering to truly understand the meaning of within-rater differences on the 360-degree feedback behavior in question.
Coach’s Critique:
Imagine the confusion that can be caused as a result of a participant’s bosses, peers, direct reports all rating them differently… Imagine the challenge that participant faces when trying to interpret such discrepant views… Imagine the assumptions and misinterpretations that occur as a result of these differences…
When interpreting 360-degree feedback results, it seems necessary to have a way to gauge at the different ratings not only between groups but within groups. For one thing, the 360 tool should be able to demonstrate these differences so that a participant can determine the weights of each set of scores. Since the interpretation process may be somewhat subjective, it seems important to utilize a tool that can demonstrate results in the most objective way.
Another recommended strategy to minimize the challenges of rater differences is to work with a coach, consultant help participants properly interpret the results, by focusing on patterns of ratings rather than individual scores, and facilitate a dialouge that determines reasons for variability in opinions of raters.
What are your suggestions about how to handle rater disagreements in 360-degree feedback reports?
Sandra Mashihi
Dr. Sandra Mashihi is a senior consultant with Envisia Learning, Inc. She has extensive experience in sales training, behavioral assessments and executive coaching. Prior to working at Envisia Learning, Inc., She was an internal Organizational Development Consultant at Marcus & Millichap where she was responsible for initiatives within training & development and recruiting.. Sandra received her Bachelor’s of Science in Psychology from University of California, Los Angeles and received her Master of Science and Doctorate in Organizational Psychology from the California School of Professional Psychology.
I think this is a matter of sharing enough information so a participant knows when there is a problem, yet not so much information that raters perceive that they will be hunted down for sharing their views. But I wonder if we aren’t giving participants enough credit here. When I think about my working relationship on a specific behavior with each of my co-workers, it’s pretty easy to see that they might perceive my behavior differently. Coaching would certainly help bring this to light (and help ensure that light is more objective) for someone less experienced in report interpretation, so I am a big fan of that suggestion.