“Success and failure are both greatly overrated. But failure gives you a whole lot more to talk about.â€
Hildegard Knef
Looking for an easy, simple, reliable, and valid way to measure whether a coaching, training or consulting intervention has impact?
The “post-then-pre” method of self-report evaluation offers one solution for documenting behavior change. The data collection instruments are relatively easy to develop, use, and analyze ((Nowack, K. (1986). Pre-post-then evaluation of a behavioral modeling approach to supervisory skills training. Performance & Instruction, 25, 14-16)).
Problems with Typical Approach to Evaluating Leadership Programs
A typical approach to evaluation has been to use a pretest-posttest research design to document behavior change. However, in certain types of self-report program evaluation, pretest-posttest comparison results may be an inaccurate assessment of impact because participants may have limited knowledge preventing them from accurately assessing baseline behaviors ((Metzoff, B. (1981). How to get accurate self-reports of training outcomes. Training & Development Journal, 35, 56-61)).
By the end of the program, their new understanding of the program content may have an impact on the responses on their self-assessment (this is referred to as a response shift bias). If a pretest was used at the beginning of the intervention, participants have no way to correct an answer at the end of the program if they made an inaccurate assessment at baseline.
The evaluation problem, then, is that a pre-assessment taken at the beginning of an intervention may be invalid because participants have limited knowledge in responding accurately to the questions being asked to respond to after the intervention has been completed.
Consider the following pre-assessment question for an executive coaching intervention: “Have you utilized a participative approach to involving your team in problem solving, decision making or planning processes?” To answer this question accurately, the respondent must have some idea what is meant by utilizing a “participative approachâ€.
An executive who doesn’t know exactly what is meant by “participative leadership†may tend to overestimate his/her behavior on some type of pre-assessment. After actually participating in the executive coaching program and learning about different strategies and approaches for including employees in problem solving, the participant can more validly answer the question.
Now suppose the executive has demonstrated an increase in applying more participative approaches with his/her team as a result of the coaching intervention. On any post assessment aimed at measuring this change in behavior, the executive reports the same level of utilizing participative approaches with his/her staff. The post-assessment level is accurate, but because the pretest was an overestimate (due to the executive’s lack of knowledge), it will appear that no change in behavior has occurred between pre-assessment and post-assessment.
Such an evaluation result makes it appear that the coaching had no effect on this type of behavior when, in fact, it significantly increased this leadership behavior.
Using the Post-Then Approach to Evaluation
The “post-then” evaluation design corrects this problem. The problem is handled by not providing any pre-assessment at the beginning of the intervention. Then, at the end of the intervention, the participant answers two questions. The first question asks about behavior as a result of the program (“Post†evaluation).
Then the participant is asked to report what the behavior had been before the program (“Then†evaluation). This second question is really the pretest question, but it’s asked after the program when the participant has sufficient knowledge and/or experience to answer the question validly. Research has confirmed that this “post-then†approach is a more valid way of evaluating interventions. These change scores also lend them selves to statistical analyses—you can utilize a paired sample t-test to determine whether changes over time are meaningful with most statistical packages.
The “post-then” design accounts for changes in learners’ knowledge and/or behavior by allowing participants to first report present behaviors (post); and then rate how they perceived these same behaviors just before taking the course (then pre). The retrospective pretest at the end of the program is more accurate because it’s answered in the same frame of reference as the post-assessment. Thus, the problem of what’s called “response-shift bias” in self-report, pre-post designs is minimized ((Howard, G. & Daily, P. (1979). Response shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 141-150)).
Using a “post-then” evaluation design for self-reported behavioral changes can provide substantial evidence for intervention impact. Although a leadership example was provided here, the methodology can be adapted and easily applied to other behaviors, knowledge and attitudes. Using a “post-then” evaluation design greatly helps practitioners provide greater evidence about the impact of their interventions using a fairly simple, but powerful, self-report approach.
So, how much would you say you knew about this quick and powerful way to improve evaluation of leadership development programs before you read this blog? Be well…..
[tags]surveys, Envisia, Envisia learing, retention, talent management, engagement surveys, job satisfaction, worklife balance, job stress, kenneth nowack, ken nowack, nowack[/tags]
Ken Nowack
Kenneth Nowack, Ph.D. is a licensed psychologist (PSY13758) and President & Chief Research Officer/Co-Founder of Envisia Learning, is a member of the Consortium for Research on Emotional Intelligence in Organizations. Ken also serves as the Associate Editor of Consulting Psychology Journal: Practice and Research. His recent book Clueless: Coaching People Who Just Don’t Get It is available for free for a limited time by signing up for free blog updates (Learn more at our website)
