“The problem today is one of too many choices, few of which compel but all of which distract.” -Joel Bryant
The number of items on a scale is often a point of confusion for individuals looking for appropriate 360-degree tools. With all the research that has been done on this subject, there is no definitive agreement about the optimal number of response categories that should be used in order to get the most reliable data. However, there is a general range provided in the 360-degree feedback literature that suggests the number of response categories to use, as follows:
Bandalos & Enders (1996) ((Bandalos, D.L., & Enders, C.K. (1996). The effects of non-normality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.))found that reliability was highest for scales having a 5 to 7 points.
Preston & Colman (2000) ((Preston, C.C. & Colman, A.M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15.))examined response categories ranging from 2 to 11, and found test-retest validity was lowest for 2 to 4 point scales, was highest for 7 to 10 point scales, and decreased for scales with more than 10 response categories.
Later, research research by Lozano, Garcia-Cueto, and Muniz (2008) ((Lozano, L.M., Garcia-Cueto, E., Muniz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. European Journal of Research Methods for the Behavioral and Social Sciences, 4, 73-79))also looked at the reliability and validity of scales ranging from 2 to 9 response options with four different sample sizes, and found that the optimum number of statistically favored response scales were between 4 and 7 options.
Current findings from the 2009 Benchmark study of the 3D Group suggests that the most popular number of items is a 5-point scale (76 percent), followed by a 7-point scale (16 percent).
Coach’s Critique:
When it comes to choosing an appropriate 360 rating scale, it seems like different numbers of choices have different benefits and disadvantages. What’s better?  3-point scale? 4-point scale? 5-point scale? 7-point scale? 10-point scale? As a designer of surveys and as a rater of surveys, here is my experience with different rating scales.
- With a 3-point scale, it might be visually easier to focus on. It is seemingly less time-consuming, which may perhaps result in an increased chance of completion of the test or a smaller chance of rater fatigue.
- With a 4-point scale, you are setting raters up to have to rate either “good or badâ€, “effective or ineffectiveâ€, “high frequency or low frequencyâ€. Can the ratee simply be moderate at an item?
- With a 5 or 7-point scale, you are likely increase reliability based on various researchers findings. In addition, you are less likely to have middle range ratings. There are enough options to ensure reliability, but not too many to make it time consuming or cause confusion.
- With a 10-point scale, while it seems like more options would produce more accuracy, I believe that too many options will decrease the value and weight of each number. Possibly, a 10-point scale may result in the same effect as a 3-point scale.
What are your preferences as far as the number of rating points on a 360-degree rating scale?
Sandra Mashihi
Dr. Sandra Mashihi is a senior consultant with Envisia Learning, Inc. She has extensive experience in sales training, behavioral assessments and executive coaching. Prior to working at Envisia Learning, Inc., She was an internal Organizational Development Consultant at Marcus & Millichap where she was responsible for initiatives within training & development and recruiting.. Sandra received her Bachelor’s of Science in Psychology from University of California, Los Angeles and received her Master of Science and Doctorate in Organizational Psychology from the California School of Professional Psychology.
