Summary of "Measuring Leadership Development"

Measuring Leadership Development
Quantify Your Program’s Impact and ROI on Organizational Performance
by Jack Phillips, Patricia Pulliam Phillips and Rebecca L. Ray
McGraw-Hill © 2012from getAbstract © 2013

Take-Aways

• Your organization’s success depends on the depth and quality of its leadership.

• Track, measure and evaluate your investment in training and developing leaders.
• Ensure that leadership development programs directly affect your organization’s highestpriority needs. Articulate the connections clearly before launching any program.
• Determine the appropriate level of measurement for each program based on costs,
importance, scope, executive interest and the future of the program.
• Level 1 evaluation measures students’ “reactions”: Did they like the program?
• Level 2 measures “learning”: What and how much did they learn?
• Level 3 measures “application”: Are they applying what they learned on-the-job?
• Level 4 measures “business impact”:What was the learning effect on the organization?
• Level 5 measures “return on investment” (ROI): Did the program make or save the
company more money than it cost?
• Conduct Level 1 evaluation on all programs; Level 2 on about 80%; Level 3 on 30%;
Level 4 on 10%, and Level 5 on 5%.

Rating (10 is best)

Overall: 8
Applicability: 9
Innovation: 8
Style: 7

Relevance

What You Will Learn

In this summary, you will learn:r1) Why leaders and leadership development are becoming

more critical, 2) How to evaluate a leader development program using five levels

of measurement, and 3) How to calculate the return on investment of a leadership

development program.

Recommendation

Human resources professionals and leaders know it’s important to measure, track and

evaluate their investments in human capital programs, yet HR is notoriously behind other

areas of business in making budget decisions based on evidence and data. HR gurus Jack

Phillips and Patricia Pulliam Phillips teach constantly about measurement; they publish

at least one book each year, often for HR professionals. In this volume, writing with

learning expert Rachel L. Ray, they focus on leadership development, offering a clear,

credible process for collecting, measuring and reporting on training outcomes and for

improving, expanding or ending those programs based on hard evidence. getAbstract

recommends their packed-tight manual to readers seeking the skill and knowledge to

evaluate leadership development programs and to help their firms make better decisions.

Summary

Who Really Cares About Leadership Development?

Executives all claim to care about leadership, but few consistently demonstrate their

companies’ commitment to that philosophy. Corporations promote workers to managers

and managers to executives based on performance, often with an incomplete consideration

of the new executive’s leadership potential. Does your organization make people into

supervisors, managers and executives based largely on their technical skills, their

contribution or their leadership skills?

Chief executives consistently rate talent as a pressing concern. Command and control

techniques alienate creative employees and stifle innovation, but so does neglect. Skilled,

white-collar workers need leaders with increasingly sophisticated competencies and

skills, both hard and soft, and with a keen sense of when to offer help and direction and

when to hold back. Leaders are most effective when they drive team performance – that

means engaging, inspiring and coaching, doing fewer tasks themselves and spending more

time helping others achieve better results.

Moving from individual contributor to neophyte leader and from manager to executive are

transformational leaps. Great leaders gain much of their knowledge by accumulating years

of experience, but good organizations accelerate their leaders’ development by investing

in programs that develop the skills leaders need. Measurement and analysis are the only

ways to learn what investments in leadership development return the greatest results.

Setting the Stage for Leadership Development Program Measurement

Measuring a program’s impact can be costly, so first choose the appropriate level of data

collection and evaluation:

1. “Reaction” – Measure 100% of your leadership training against participant feedback.

Ask every trainee to complete a short survey after the course.

2. “Learning evaluation” – Assess about 80% of your leadership development courses

to measure how much knowledge students gained. Use pre- and post-training tests.

3. “Application and implementation evaluation” – Assess about 30% of your courses

to check how well people apply the information they learned on-the-job.

4. “Business impact” – Expensive, ongoing programs – about 10% of your leadership

development efforts – warrant a determination of their effect on your organization.

5. “Return on investment” – Reserve a full ROI analysis for only about 5% of your

leadership development programs.

Suppose you’re managing two important long-term and high-profile leadership development programs. One is new and the other is a mandatory program that has been in place for years. You should measure the mandatory program through Level 3 or 4, but not Level 5. Why? Since it is mandatory, ROI is irrelevant, except for data that helps improve the course. On the other hand, if you’re getting ready to spend a significant sum of money on the new program, assume that you’ll eventually need to justify the investment even if no one has asked for an evaluation yet. Since the program is important, costly, visible and ongoing, it’s a strong candidate for comprehensive evaluation, including the determination of business impact and ROI. Though you cannot skip any levels and you always must consider the time and cost of a Level 4 or 5 evaluation, you can shorten the initial stages to focus on the advanced stages.

Set the stage for evaluation at all levels by documenting the course’s intended objectives. Link each course’s goals directly to a business priority. With a clear, predetermined understanding of every leadership training session’s objectives, you’ll know the benchmark you’re trying to hit.

Getting Your Data

Now that you know the level of evaluation you’re going to conduct and its goals, make a plan for collecting data. For example, use reaction surveys to collect the data for Level 1 measurement. At Level 2, compare pre- and post-training test scores to obtain your data. To compare pre- and post-performance records (such as the quality of answers that call center workers give to customers), you need to know where those records are stored and obtain any necessary permission to use them. At Level 3, conduct focus groups or watch learners at work. If you’ve put supervisors through an employee recognition program, you might conduct a Level 3 assessment by observing their post-training interactions with their teams. This could be hard, costly and time consuming, especially if you want to watch them as a “mystery shopper” so they don’t know you’re there. Here, surveying learners and their staff might be slightly less reliable but more practical.

At Level 4, you may need financial, sales or HR data, so investigate the access required. Choose the best, most credible information sources, depending on the difficulty and cost of obtaining data. To conduct a Level 5 ROI analysis, gather the relevant financial and personnel figures.

For example, to know if a leadership program is worth its cost, you may need attrition rates, absenteeism and productivity data, and performance-related costs, all of which may come from different departments and databases. Usually, you’ll need data from the participants and possibly their managers or peers. Prepare learners before and during the course to improve their future participation. If you can’t obtain evaluation data, you must assume no improvement took place.

Measuring Reactions, Learning, Business Impact and Application

Any course worth teaching is probably worth evaluating at Level 1, at least. Do participants think the course is valuable? Is the material relevant? Are the instructors effective? Do students think they will use what they learned? Would they recommend the course? How do they rate it? Such data are very valuable. To capture them, use a short questionnaire after the course. Then – in a step many organizations fail to take – analyze the information, share it with the right people and use it to improve programming. Often leadership development courses teach difficult-to-measure soft skills such as empathy, listening, judgment, recognition and team building. Nevertheless, have learners complete knowledge tests before and after the course to obtain insights into their retention and learning gains and, at Level 2, their use of the new information on the job.

Leadership development programs are worthwhile only if they lead to targeted behavioral changes. Nonetheless, you probably can’t afford to evaluate every program through Level 3. For the roughly 30% of programs where you want to measure behavior change at this level, rely on participant and manager surveys, interviews, focus groups, and, to a lesser degree, “action plans,” observation and performance monitoring. Write your survey, focus group or interview feedback questions thoughtfully. Train interviewers to build trust and to deliver consistent interviews. Consider focus groups to save time and money and to benefit from small-group conversational dynamics. Ask learners to commit to using what they learned. You can refer to this “performance contract” and action plan for up to a year to assess the transfer of learning to the job.

Even when people apply new knowledge, you don’t know for sure that their improvement helps your business. With Level 4 evaluation, you can set and measure such goals as better employee engagement, lower absenteeism, less turnover or higher performance against a measurable output – all reasonable expectations from leadership development programs. Be precise about your conclusions, because your credibility is at stake. Exclude any anomalies from your analysis. If the team of a sales manager who took the course sells 500% more the month after the course, but the mean increase across all teams whose bosses took the course is 15%, exclude the outlier entirely.

To be sure your conclusions are accurate, factor in external events that might affect your results. For example, say you wait eight months after a leadership program to let the knowledge spread and then you observe apparent results: Absenteeism is down 5%, turnover is down 4%, and both are lower than expected. You’re thrilled, but can you assume your program is 100% responsible for the gains? Consider what else happened in those eight months that might have affected these results. Personnel changes, new competitors, other programs or a changing economy are just a few potential influencers. Estimate the impact of external events by asking your participants and their associates for feedback. Alternatively, turn to comparison groups or a “trend line analysis” to isolate the impact of your program from external factors. Use at least one of these processes to derive an “attribution estimate.” For leadership programs, after this length of a lapse since the training, such attribution estimates are likely to be in the 60% to 80% range.

If you arrived at your attribution estimate by consulting the people who know best, take another conservative step before you announce the training’s outcome. Ask your experts how confident they are in their attributions. Virtually no one will claim to be 100% certain because too much time has passed and too much has happened since the course. Instead, your respondents might say they are, say, 75% confident in their estimates. Average their individual responses to determine a “confidence level.” Use very conservative estimates and calculations to gain credibility. Taking your attribution estimate at 70% and your confidence level at 75%, apply simple math to arrive at your impact estimate. Use the 5% reduction in absenteeism in this formula: 5 × 0.75 × 0.70 = 2.6%. Use the 4% turnover reduction the same way: 4 × 0.75 × 0.70 = 2.1%. This means your program is responsible for a 2.6% reduction in absenteeism and a 2.1% reduction in turnover.

Calculating ROI

Ultimately, you must evaluate some programs against monetary returns. To calculate ROI, you want to convert your measured gains to money. This is hard with some gains. For instance, employee engagement scores might improve six months after a leadership program. Such an improvement may have been a major goal, but attributing the right portion of it to your course and expressing it in dollars is time consuming and difficult. You might assert that higher engagement led to improved customer retention, which generated more profit. While that is entirely possible, your credibility and the defensibility of your claims may suffer because the line from engagement to money is indirect. Instead, attribute the improvement to “intangibles” and report them, but don’t use them in your ROI calculation. Intangibles are gains you report but don’t convert to monetary value.

However, you have credible data that attributes 2.6% of the reduction in absenteeism and 2.1% of the reduction in turnover to your leadership program, so in those areas, you have gains you can use to calculate hard dollar ROI. To convert those benefits to financial terms, gather data from HR and finance to determine the unit costs of absenteeism and turnover, like the expense of hiring and training. Apply your percentages to convert the absenteeism and retention gains to money. For example, when a $20-an-hour nursing assistant is absent, how much does the stand-in worker cost? Replacing a mid-level sales associate might take weeks of advertising and screening, plus training and a drop in sales while the new hire learns the business. Use such data to derive the total monetary benefit of your program’s impact on absenteeism and turnover.

To calculate your ROI, add all your leadership development program costs: licenses, trainers’ fees, meeting rooms, catering, materials, travel and administration, plus the cost of the evaluation and of trainees being away from work. Subtract these costs from the total monetary benefits and divide the result by the total costs. To derive your ROI percentage, multiply the resulting number by 100. The formula is: ROI = [(total benefits – total costs) × 100] / total costs. For short or one-time programs, this should be your final estimate, even though the program’s benefits might last years. You may calculate future ROIs for long-term programs.

Reporting Your Results

Share your results with your stakeholders in an appropriate venue and format. One size does not fit all. For some, a detailed report might be best. For others, an in-person slide presentation is better. Some leaders might require only a one-page summary; others a sitdown meeting. Do not take reporting for granted or treat it as an afterthought. How you frame and position your findings will affect the trust people place in your conclusions and recommendations.

About the Authors

Former bank president and HR executive, Jack Phillips has written or edited more than 50 books. Patricia Pulliam Phillips, writer or editor of more than 30 books on evaluation, heads the ROI Institute. Rebecca L. Ray leads human capital at the Conference Board.