More Choices OR Less? How to Determine Optimal Number of Ratings on 360-Degree Rating Scale

January 20, 2016 by Sandra Mashihi
“The problem today is one of too many choices, few of which compel but all of which distract.” -Joel Bryant

The number of items on a scale is often a point of confusion for individuals looking for appropriate 360-degree tools. With all the research that has been done on this subject, there is no definitive agreement about the optimal number of response categories that should be used in order to get the most reliable data. However, there is a general range provided in the 360-degree feedback literature that suggests the number of response categories to use, as follows:

Bandalos & Enders (1996)1found that reliability was highest for scales having a 5 to 7 points.

Preston & Colman (2000)2examined response categories ranging from 2 to 11, and found test-retest validity was lowest for 2 to 4 point scales, was highest for 7 to 10 point scales, and decreased for scales with more than 10 response categories.

Later, research research by Lozano, Garcia-Cueto, and Muniz (2008)3also looked at the reliability and validity of scales ranging from 2 to 9 response options with four different sample sizes, and found that the optimum number of statistically favored response scales were between 4 and 7 options.

Current findings from the 2009 Benchmark study of the 3D Group suggests that the most popular number of items is a 5-point scale (76 percent), followed by a 7-point scale (16 percent).

Coach’s Critique:

When it comes to choosing an appropriate 360 rating scale, it seems like different numbers of choices have different benefits and disadvantages. What’s better?  3-point scale? 4-point scale? 5-point scale? 7-point scale? 10-point scale? As a designer of surveys and as a rater of surveys, here is my experience with different rating scales.

  • With a 3-point scale, it might be visually easier to focus on. It is seemingly less time-consuming, which may perhaps result in an increased chance of completion of the test or a smaller chance of rater fatigue.
  • With a 4-point scale, you are setting raters up to have to rate either “good or bad”, “effective or ineffective”, “high frequency or low frequency”. Can the ratee simply be moderate at an item?
  • With a 5 or 7-point scale, you are likely increase reliability based on various researchers findings. In addition, you are less likely to have middle range ratings. There are enough options to ensure reliability, but not too many to make it time consuming or cause confusion.
  • With a 10-point scale, while it seems like more options would produce more accuracy, I believe that too many options will decrease the value and weight of each number. Possibly, a 10-point scale may result in the same effect as a 3-point scale.

What are your preferences as far as the number of rating points on a 360-degree rating scale?

  1. Bandalos, D.L., & Enders, C.K. (1996). The effects of non-normality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160. []
  2. Preston, C.C. & Colman, A.M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15. []
  3. Lozano, L.M., Garcia-Cueto, E., Muniz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. European Journal of Research Methods for the Behavioral and Social Sciences, 4, 73-79 []

Dr. Sandra Mashihi is a senior consultant with Envisia Learning, Inc. She has extensive experience in sales training, behavioral assessments and executive coaching. Prior to working at Envisia Learning, Inc., She was an internal Organizational Development Consultant at Marcus & Millichap where she was responsible for initiatives within training & development and recruiting.. Sandra received her Bachelor’s of Science in Psychology from University of California, Los Angeles and received her Master of Science and Doctorate in Organizational Psychology from the California School of Professional Psychology.

Posted in 360 Degree Feedback

If You Enjoyed This Post...

You'll love getting updates when we post new articles on leadership development, 360 degree feedback and behavior change. Enter your email below to get a free copy of our book and get notified of new posts:

Follow Envisia Learning:

RSS Twitter linkedin Facebook

Are You Implementing a Leadership Development Program?

Call us to discuss how we can help you get more out of your leadership development program:

(800) 335-0779, x1