The problem with measuring return of investment in a training program starts with the term itself.
“refers to how much profit a company makes relative to the money spent to manufacture and sell a product. How can you apply that to the intricacies inherent in training people?” says Albert Siu, chief learning officer at AT&T's headquarters in Basking Ridge, N.J., who likens the pursuit of ROI from employee training to the quest for the Holy Grail.
Still, the need to measure training exists, even if it is only testing what specific learning has occurred. Models such as ROK, or return on knowledge, and ROO, the return on objectives, acknowledge the idea that the “product” Siu refers to is people: what they learn and how they apply that knowledge toward what the company wants to accomplish.
“Many companies talk the rules of the new game, where people are seen as assets, but they [these companies] still see people as costs,” says Laurie Bassi, director of research at the Society for the Advancement of Behavioral Analysis and CEO of Human Capital Capability, in Chevy Chase, Md. “To maximize the value of your human capital, you really have to measure it and manage it as a strategic asset. But the crux of the matter is that people aren't being managed as a strategic asset. In my view, only a small percentage of firms are serious about viewing their people as assets rather than costs.”
In that context, she adds, the term ROI “needs to be gotten rid of altogether — there's just too much baggage associated with it.”
Pfizer: Plan Early, Measure Selectively
For Terry Shoemaker, director of measurement, evaluation, and strategic analysis for New York-based Pfizer, ROI measurements need to be integrated into a training program at the start to be valid, and done only if and when they are necessary.
“It's absolutely critical, right at the beginning, for trainers to know what the company's business plan is and how they want to achieve it,” he says. In fact, when Shoemaker was an independent consultant, companies would bring him in at the eleventh hour to develop an ROI analysis for a training program that was about to be implemented. “I'd often counsel them not to do it because it was at the last minute,” he says. Perhaps indicative of how important ROI has become to corporate training, Shoemaker was hired by Pfizer a year ago to fill this newly created position.
Pfizer asks some very crucial questions at the start of any training program: Do we want to teach employees how to use new technology? Do we want to teach managers how to manage better? Do we want to reduce turnover? For some of these purposes, they choose to measure ROI; for others, they don't.
“I don't know of any company that does ROI on every training program,” Shoemaker says. “You have to remember that developing an ROI analysis can be very expensive. Put it this way: Sometimes you have to do an ROI on your potential ROI analysis.”
AT&T: Teach ‘The Right Stuff’
At AT&T, Siu uses this term to define appropriate content and delivery. “First we ask ourselves if we're offering the right stuff, meaning the training content. If it's training about the corporate culture, the CEO answers that question. If it's sales, then the question is answered by the sales executive. The right stuff content is relative to the training, so the ROI becomes relative too.”
The next issue has to do with who the trainees should be.
“We ask if we're using our training dollars wisely — are the right people getting the content we want them to get?” he says. “Problems arise when companies try to have all their people learn everything. We advocate that our people receive the content they need — no more, no less.”
AT&T also carefully analyzes different methods of content delivery. “If you have content that a broad-based volume of employees need to know, e-based learning is likely the best way to go,” Siu explains. “For small groups, traditional methods are usually best.”
Fusion Productions: Identify Stakeholders
Ed Simeone, executive producer for Fusion Productions, a Webster, N.Y.-based firm that specializes in integrating learning, meetings, and technology, says the place to start an ROI analysis is with the stakeholders. “It may be the CEO, the vice president of sales, the regional executives, or the salespeople.
“You need that interaction with as many stakeholders as possible and as early as possible,” he adds. “Involve them in determining their program expectations. Brainstorm these expectations until they come to agreement on which are most crucial.” Only when those expectations are clear can valid measurements be formulated.
With more varied input comes a broader perspective. “There was a time when companies left employee training to individuals or single departments. You don't want to do that,” he says. “If, for instance, you leave it to a sales executive or human resource director, the focus … will be too narrow, because they'll have only their own perspective to work from.”
Terry Shoemaker on THE KIRKPATRICK MODEL
You can't talk about training ROI without going to its beginning: Kirkpatrick's Four Levels of Evaluation, developed by Donald Kirkpatrick, PhD, in 1959. Each level represents a more precise measure of training program effectiveness but requires a more focused, time-consuming, and, hopefully, more precise analysis.
We asked Terry Shoemaker, director of measurement, evaluation, and strategic analysis for New York-based Pfizer, to describe Kirkpatrick's Levels. “Most training programs are doing Level 1 evaluation 100 percent of the time,” he explains. “Then, focus starts depleting by half. In other words, Level 2 is used maybe 50 percent of the time, Level 3 [about] 25 percent of the time, and Level 4 about 12 percent or 13 percent.”
Level 1) Reaction — Simply put, the first level measures how participants responded to the training.
Level 2) Learning — This level attempts to assess whether the participant advanced his or her knowledge of a skill, knowledge, or attitude.
Level 3) Transfer — Here, you consider whether the participant is applying the learned material successfully. “If the answer to the question posed in Level 2 is no, you figure out why they didn't. If the answer is yes, then … you determine whether they are using that knowledge.”
Level 4) Results — This level — the most difficult — attempts to measure what effects the skills are having on company performance. “Here, you ask whether a company's sales are up as a result of a sales training program, or whether turnover is down after … a program where that's been the objective. You have to be aware that the outcomes you're trying to reach could have happened anyway, regardless of training. So the way you measure this has to be very well thought out.” Pfizer is trying to develop another level —
Level 5) that would “dollarize” training results. “We're looking at ways to show how training programs that deal with something other than dollars — such as increasing employee satisfaction or reducing turnover — can be translated monetarily. Of course, to do that effectively, you have to know exactly the costs of the training program,” he says. “You also have to come to accept that ROI won't tell to the penny the value of your training.”