“Once you label me, you negate me.” -Soren Kierkegaard
An important issue for response scales is considering how to best label the response scale anchors. There are several effects that can arise as a result of rating scale labels.
1. Clearly labeled scales results in less leniency effects. Negative skew or low variability in scale ratings are fairly common in 360-degree feedback. Few raters use the entire rating scale range, and most scores tend to be inflated, independent of the scale being used.
2. Positively worded scales reduces response bias and increases full use of response scale. Recent research suggests that the use of positively worded scales can result in lower mean scores and increased variability, relative to typical, anchored scale, English, Rose, & McClellan (2009) ((English, A., Rose, D., McClellan, J. (2009).Rating Scale Label Effects on Leniency Bias in 360-degree Feedback. Paper presented at the 24th Annual Meeting of the Society for Industrial Organizational Psychologist. New Orleans, LA))Positive-worded scales are comprised of anchors with a larger number of positive verbal qualifiers and should be considered for coaches, consultants, and organizations creating their own customized 360-degree feedback assessments.
Example of a Traditional Frequency Scale
1: To a very small extent
2: To a small extent
3: To a moderate extent
4: To a large extent
5: To a very large extent
Example of a Positively-Worded Response Scale
1: Almost never
2: Sometimes
3: Frequently
4: Almost always
5: Always
3. Clearly labeled response scales results in reduced ambiguity. Weng (2004) ((Weng, L. (2004). Impact of the number of response categories and anchor lables on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.))analyzed the reliability of Likert-type rating scales and found that scales with all response options clearly labeled yielded higher test-retest reliability than those with only the end-points labeled. This suggests that having all of the points on a scale clearly labeled helps reduce ambiguity.
4. Appropriate number of response options reduces chance of context effects. Cools, Hofmans, and Theuns (2006) ((Cools, W., Hofmans, J., & Theuns, P. (2006). Context in category scales: Is “fully agree†equal to twice agree? European Review of Applied Psychology, 56, 223-229))found that a scale with five response options was the least prone to context effects and that the use of extreme answer categories on the left and right ends of the scale did not improve the psychometric properties of the scale.
Viswanathan, Bergen, Dutta, and Childres (1996) ((Viswanathan, M., Bergen, M., Dutta, S., & Childres, T. (1996). Does a single response category in a scale completely capture a response? Psychology & Marketing, 13, 457-479))in studying the optimal number of response categories and appropriate category descriptors found that the right number of categories was important, so that there was not a mismatch between participants’ natural responses and the response categories.
Coach’s Critique:
The problem with providing feedback through 360 is that raters are set up in a position to answer questions based on question items and rating labels that are provided to them. For this reason, it is particularly important to phrase not only items, but rating scales as clearly as possible to increase probability of accurate feedback. In addition, since raters are generally reluctant to answer accurately regardless of the scales, consultants and implementers of 360s need to consider what encourages accurate feedback. For instance, it is typical that respondents in non-positively worded scales are more reluctant to use the full range scale, which negatively skews results because with these type of scales, raters are generally rating towards the “high†end of the scale. With that said, it is important to ensure the optimum number of rating points on a scale, most appropriate response scale (visit blog on “Best” Response Scale for 360 Feedback), provide clearly labeled scale definitions, and provide a positive scale to avoid ambiguity and negatively skewed results.
Sandra Mashihi
Dr. Sandra Mashihi is a senior consultant with Envisia Learning, Inc. She has extensive experience in sales training, behavioral assessments and executive coaching. Prior to working at Envisia Learning, Inc., She was an internal Organizational Development Consultant at Marcus & Millichap where she was responsible for initiatives within training & development and recruiting.. Sandra received her Bachelor’s of Science in Psychology from University of California, Los Angeles and received her Master of Science and Doctorate in Organizational Psychology from the California School of Professional Psychology.
