1. VOLUME 4
BUSINESS INSIGHTS FROM UNC EXECUTIVE DEVELOPMENT
WHITE PAPERS
FEATURED:
Beyond Smiley Sheets:
Measuring the ROI of
Learning and Development
Leadership Agility:
Using Improv to Build
Critical Skills
Talent Builders:
Lead the Way in Developing
Your People
Wired to Learn:
How New Technologies Are
Changing L&D Delivery
Ready to Serve:
How and Why You Should
Recruit Veterans
3. Inside this issue
Beyond Smiley Sheets: Measuring the ROI of Learning and Development
page 4
Leadership Agility: Using Improv to Build Critical Skills
page 14
Talent Builders: Lead the Way in Developing Your People
page 24
Wired to Learn: How New Technologies Are Changing L&D Delivery
page 34
Ready to Serve: How and Why You Should Recruit Veterans
page 44
(Note: The information or conclusions expressed in the following white papers are the authors’ review of findings expressed by the
organizations. All brand representations are registered trademarks owned by the respective companies or organizations.)
3
5. MEASURING THE ROI OF LD PROGRAMS
• ffers suggestions on how to ensure that LD
O performance or behavior and how those changes have
evaluations reflect what executive leadership expects. benefited the entire organization (i.e., bottom-line
• rovides steps to consider when evaluating the ROI of
P results).
development programs. Kirkpatrick argued that to obtain a meaningful evalua-
• hares examples of companies that have effectively
S tion, it was necessary to evaluate the LD experience
demonstrated the value of their LD programs. at each level. This has proven easier said than done.
Most learning professionals find it a challenge to assess
beyond the first level to demonstrate long-term learning
Kirkpatrick’s Four Levels and organizational benefits, and it is this longer-term,
bottom-line impact that many senior leaders want to see.
of Evaluation
Donald Kirkpatrick is perhaps best known
for creating the classic four-level model
of evaluation. His simple paradigm—
which still serves as an excellent
framework when evaluating
development programs—identified
four distinct evaluation tiers:
1. Reaction
2. Learning
3. Behavior
4. Results
The first level, reaction, captures
participants’ satisfaction with the
experience immediately following the
event. These smiley sheets gather
participants’ thoughts and feelings about
the program, the moderator, the content,
the venue, etc.
Level two, learning, captures the increase in
knowledge, skills or capabilities as a result of the experience.
Level three, behavior, assesses the transfer of learning—whether
participants successfully applied what they learned to their work.
This level can also involve assessing changes in behavior and
attitudes that result from the experience.
The final level, results, assesses
participants’ changes in
5
7. MEASURING THE ROI OF LD PROGRAMS
Tips to Measuring ROI
Don’t go overboard. ROI need only demonstrate value beyond a reasonable doubt.
1.
Find out what the executive sponsor identifies as success and stick to that.
Shift from a quality to a results mindset. When designing LD programs, it’s all
2.
about delivering a quality experience that encourages learning. When evaluating, it’s
about results. When calculating ROI, focus less on the quality of the experience and
more on the effect of learning.
Calculate ROI continuously. Always know how the LD program is performing so
3.
adjustments can be made. This not only helps improve the program, but can justify how
dollars are being spent at any time.
Build a step-by-step case for ROI. Analyze organizational needs and develop strategic
4.
learning plans, prioritize them and present them with sound justification—based on
anticipated ROI – about why senior leaders should support them.
Gather data beyond the program delivery and don’t forget data that is already
5.
available. Evaluation and feedback should come from as many sources as feasibly
possible—from participants, their supervisors, peers and senior leaders.
ROI isn’t just about money. When analyzing results, consider such learning
6.
measurements as quality, effectiveness, job impact and business results.
7. conservative in ROI calculations. To compensate for bias, self-reported ROI should
Be
be factored down and follow-up evaluations should be weighed more than evaluations
reported immediately after the program.
Represent the money outlay as a per participant ratio. Personalize it. Show the per
8.
participant cost (versus a total cost) to make the investment more palatable
(e.g., for this target population we are looking at a $7,000 investment in learning for an
employee responsible for, on average, $1million worth of business).
Communicate the story behind the numbers. This is where using anecdotal
9.
information can be helpful in confirming the numbers. It never hurts to highlight data
with meaningful examples.
10. the ROI numbers are low, don’t be discouraged. ROI is intended to assess what is
If
working and what should be shelved or revamped.
Source: Garvey, 2012.
7
9. MEASURING THE ROI OF LD PROGRAMS
Ways to Evaluate LD These metrics can be used effectively for standard
training and development programs. The challenge,
Initiatives however, is to capture the less tangible, more higher-
order benefits of learning, such as revenue generation
The metrics that could be used to evaluate LD or cost-savings realized through application exercises
initiatives are extensive. Here is a laundry list of built into the program. Because leadership programs
measures that organizations often use to evaluate often focus on the development of skills such as
their learning programs: adaptability, collaboration, communication, decision
making, innovativeness and leading change, the
verage change in performance appraisal ratings
A
•
intangible benefits are more challenging to measure
over time using conventional metrics (Kalman, 2012).
ustomer satisfaction ratings
C
•
Linda O’Connell, principal of Learnologies, LLC,
recommends blending ROI metrics with participant
mployee engagement survey scores
E
•
and stakeholder feedback and anecdotes to better
urnover rates
T convey the total value of leadership development
•
programs. Jack Philips, chair of the ROI Institute agrees.
ercentage of promotions
P
•
He recommends integrating anecdotal evidence at the
same time traditional data is collected. It can then be
roductivity rates over time
P
•
used to augment the data when it is reported to senior
etention rates
R leaders (Kalman, 2012).
•
What Fortune 500 Companies Are
Using to Measure LD Effectiveness
• mployee replacement costs*
E
• urnover costs
T
• conomic value of employee behaviors
E
• conomic benefits of increased job satisfaction,
E
organizational commitment or similar job attributes
*According to the Society for Human Resource
Management, the average replacement cost of an
employee is between 100 and 125 percent of the
employee’s annual salary.
Source: Green Brainard, 2005
9
11. MEASURING THE ROI OF LD PROGRAMS
Duke Energy has used this approach to good effect. Its In cases where hard data is difficult to identify, think
two-week Strategic Leadership Program, focuses on creatively. At IBM Europe, for example, a key aspect of
developing mid-level managers’ leadership skills, teaches a leadership development program for high-potential
participants how to evaluate business decisions and how women is a robust mentorship program. To ensure
to execute business strategies. The program culminates the mentorship program is working as intended, the
with participants working through an actual strategic company expects participants to be promoted within
challenge identified by senior executives (Palmer, 2010). a year of the start of the mentorship. Failure to obtain
a promotion is seen as the sponsor’s failure, not the
Step 4: Integrate Learning Programs candidate’s (Cater Silva, in Kelly, 2012). While this may
into the Organization’s Performance be too radical a metric for some organizations, the lesson
Management System and Hold All here is not to be afraid to think differently when creating
evaluation criteria.
Stakeholders Accountable
To increase the probability that the targeted outcomes of Also, do not overlook data that is often readily available
a program remain a focus area for participants, ensure when planning evaluation. Retention rates of program
performance appraisal goals reflect those targeted participants versus non-participants and promotion
outcomes for participants and their supervisors. This step and engagement scores which translate to increased
may require gathering information about an employee’s employee participation and productivity are areas where
productivity before and after the program. In cases where employers realize real value. These should be tracked
it is too late to gather pre-program data, consider using a and measured, not only for participants, but for their
control group of equivalent employees, business units or subordinates and supervisors too.
organizations to compare against.
Example: J.C. Penny
In 2008, J.C. Penney found itself among the bottom of all retailers in customer
satisfaction. The organization needed to engage and educate its 155,000 associates in
1,100 stores, logistics centers and headquarters to raise its customer satisfaction rating
from 43 percent. J.C. Penney therefore had a straightforward measure for their program’s
success: improved customer satisfaction ratings. The retailer educated its associates
through a multi-faceted approach that had distinct messages for each targeted population
(store leaders, store associates and home office associates). A key to the initiative’s success
was to obtain support from all store leaders. Store managers then received the learning
so they could “champion” the message. All store associates then participated in learning
sessions—delivered by store managers and the training supervisor team--within three
months of the store manager’s learning program.
Customer satisfaction ratings increased from 48 to 63 percent by the end of 2010,
ranking J.C. Penney as top in customer service.
Source: Chief Learning Officer staff, 2010, Business Impact Division.
11
13. Lead your
HR organization
into the future.
B U S I N E S S A N D H U M A N R E S O U R C E S
Now more than ever, senior HR leaders need the
knowledge, skills and experience to respond to
emerging trends that are shaping the future of global
business. Offered in partnership with the Society for
Human Resource Management (SHRM), UNC’s
UNC EXECUTIVE DEVELOPMENT
Business and Human Resources program is designed
The Power of Experience.
to equip senior HR leaders with the most up-to-date
business knowledge and skills needed to succeed in
the rapidly changing business environment today -
and tomorrow.
To learn more, visit www.bhr.uncexec.com.
13
15. LEADERSHIP AGILITY: USING IMPROV TO BUILD SKILLS
methods. In the following pages, we will explore one
of these non-traditional methods – using improv to
develop more agile business leaders. About Improv
“ ome people misunderstand improv….It seems
S
Promise that improv is all about being funny. But it is not.
Improv is about being spontaneous. It is about
This white paper:
being imaginative. It is about taking the unexpected
• efines what improv is (and isn’t).
D and then doing something unexpected with it….
• utlines the rules of improv and discusses how these
O The key is to be open to crazy ideas and building
rules apply to your organization. on them. And funnily enough, this is exactly what
is needed if we are going to make our enterprises
• xamines how improv can be used to develop specific
E more creative and agile.”
skills and behaviors and build agile business leaders.
– aul Sloane
P
• ffers examples of how organizations are using
O
The Leaders Guide to Lateral Thinking Skills
improv to effectively develop talent.
(in Gotts and Cremer, n.d.).
• rovides practical ways you can introduce improv in
P
your organization.
What Is Improv?
In essence, improv, short for improvisation, is There is no doubt that improv can be funny—think
performing without a script; it is spontaneous invention Stephen Colbert, Steve Carell, Bill Murray, Wayne
(in the form of acting, singing, playing musical Brady, Tina Fey or Amy Poehler. But being funny is not
instruments, etc.) that is often needed to create necessarily the goal, according to Greg Hohn, a member
something entirely new and unique. Improvisation is of Transactors Improv group since 1989, its executive and
often thought of as “off the cuff” activity, with little artistic director since 1996, and the teacher for UNC’s
or no preparation or forethought – but this can be Applied Improvisation for Communication course at
misleading. Giving an impromptu, extemporaneous UNC Kenan-Flagler. Improv is performing without a script,
speech or presentation requires skill and confidence that notes Hohn. It’s about working off the top of your head,
one can acquire through improv – but it’s not improv. being mindful and reacting to what’s around you and
Real improv requires preparation, and often practice, being entirely in the moment—not the past or the future,
to develop the ability to act and react in the moment. but the now. “Improv is about realizing that everything
Some of the basic skills improvisation requires are the you need is in the moment. If you are aware of it, you
ability to listen and be aware of the others, to have can act on it.” explains Hohn.
clarity in communication, and to possess the confidence
Bob Kulhan, CEO of Business Improvisations in Chicago/
to find choices instinctively and spontaneously.
New York/Los Angeles, adds honesty to what defines
Improvisation can take place as a solo performance or in
improv. “Improv teaches people how to react, adapt and
collaboration with other performers. It can be dramatic
communicate honestly with each other. Honesty is a key
or comedic – and the popularity of improv comedy
to improv,” notes Kulhan, who has worked with the top
continues to grow. Many people had their first exposure
business schools and companies in the world. “The focus,
to improv comedy through the British (and subsequent
concentration and honesty required in improv readily
American) television show, Whose Line Is It Anyway?, a
apply to the communication skills required in business,”
popular improvisational comedy show that featured the
says Kulhan.
short-form style of improvisation.
15
17. LEADERSHIP AGILITY: USING IMPROV TO BUILD SKILLS
can be a useful framework for brainstorming, ideation Rule #4: There are no mistakes, only
and innovation, problem solving, and conflict resolutions.
opportunities
“Yes, and” can help to foster cooperation among
employees because it requires active listening, acceptance If you follow the other rules of improv, then the
of different points of view, and contribution; it serves to possibilities are limitless – and there are no mistakes.
build rather than tear down, which can allow for more You listen, react and create something new, and this
honest and effective interactions. simple act can lead to something novel and unexpected.
While there are definitely mistakes to be made in
business, this fourth and final rule of improv is about
Rule #3: Make statements accepting and moving on. Improv is about moving
As Fey writes in Bossypants, whatever the problem, be forward and exploring new, unchartered territory – not
part of the solution. People who constantly ask questions looking backward or placing blame. Organizations and
put pressure on others to find solutions or to make business leaders can waste a lot of time pouring over
suggestions. We have all worked with those people. They past mistakes and casting blame, which can lead to an
have the power to slow progress, sometimes causing overly cautious culture where everyone is afraid to make
the organization to grind to a halt. Organizations need mistakes and no one takes chances. Good leaders take
problem solvers, or people who make statements; they responsibility, learn from mistakes and move on. To be
need solutions. There’s no shortage of hard questions and successful now and in the future, companies need to
challenges to overcome, and while it’s important that you be free to try new things, take risks, experiment and
ask tough questions, organizations need leaders who can innovate. The real failure is not learning from past
bring solutions to the table. Making statements in improv mistakes.
sets a direction that everyone can follow. It provides a
Popsicles and penicillin, Slinkies and stainless steel,
roadmap, not a final destination but a direction. Making
Play-doh® and Post-it® notes, they were all “happy
statements in improv is leadership – and everyone takes
accidents.” The inventors of all these innovations
the lead - helping to create a path and setting down that
regarded their “mistakes” as opportunities and
road together.
enriched (and in some cases, actually saved) lives.
yes AND...
17
19. LEADERSHIP AGILITY: USING IMPROV TO BUILD SKILLS
Tina Fey’s Rules for Improv
“The first rule of improvisation is AGREE. Always agree and SAY YES. When you’re
improvising, this means you are required to agree with whatever your partner has
created. So if we’re improvising and I say, “Freeze, I have a gun,” and you say, “That’s
not a gun. It’s your finger. You’re pointing your finger at me,” our improvised scene has
ground to a halt. But if I say, “Freeze, I have a gun!” and you say, “The gun I gave you
for Christmas! You bastard!” then we have started a scene because we have AGREED
that my finger is in fact a Christmas gun.
Now, obviously in real life you’re not always going to agree with everything everyone
says. But the Rule of Agreement reminds you to “respect what your partner has created”
and to at least start from an open-minded place. Start with a YES and see where that
takes you.
As an improviser, I always find it jarring when I meet someone in real life whose first
answer is no. “No, we can’t do that.” “No, that’s not in the budget.” “No, I will not hold
your hand for a dollar.” What kind of way is that to live?
The second rule of improvisation is not only to say yes, but YES, AND. You are supposed
to agree and then add something of your own. If I start a scene with “I can’t believe it’s
so hot in here,” and you just say, “Yeah…” we’re kind of at a standstill. But if I say, “I
can’t believe it’s so hot in here,” and you say “What did you expect? We’re in hell.” Or
if I say, “I can’t believe it’s so hot in here” and you say, “Yes, this can’t be good for the
wax figures.” Or if I say, “I can’t believe it’s so hot in here,” and you say, “I told you we
shouldn’t have crawled into this dog’s mouth,” now we’re getting somewhere.
To me YES, AND means don’t be afraid to contribute. It’s your responsibility to contribute.
Always make sure you’re adding something to the discussion. Your initiations are
worthwhile.
The next rule is MAKE STATEMENTS. This is a positive way of saying, “Don’t ask
questions all the time.” If we’re in a scene and I say, “Who are you? Where are we?
What are we doing here? What’s in that box?” I’m putting pressure on you to come
up with all the answers.
In other words: Whatever the problem, be part of the solution. Don’t just sit around
raising questions and pointing out obstacles. We’ve all worked with that person. That
person is a drag. It’s usually the same person around the office who says things like
“There’s no calories in it if you eat it standing up!” and “I felt menaced when Terry
raised her voice.”
(Tina Fey’s Rules for Improv continue on page 20.)
19
21. LEADERSHIP AGILITY: USING IMPROV TO BUILD SKILLS
Example: Performance of a Lifetime
Performance of a Lifetime (www.performanceofalifetime.com) is a global
organizational change company headquartered in New York that develops
people and organizations through the art of performance. It designs and delivers
customized programs to develop leadership, enhance collaboration and communication,
and to drive culture change. Performance of a Lifetime clients include American Express,
Credit Suisse, TripAdvisor, Marathon Petroleum, Johns Hopkins Hospital, and PwC.
Next Steps: Applying Improv in Your Organization
Improv can be a very effective method to develop the As you consider whether improv would be effective
critical skills required for today’s business leaders in a in your organization, you should:
hands-on, energizing way. Improv is increasingly finding
• Reflect on the qualifications and competencies
its way into leadership development programs. Pepsi,
that your leaders will require.
McDonald’s and the United Way have all used improv
– ow well do they align with the skills you can
H
in their corporate training programs (Glazer, 2008). U.S.
develop with improv?
Cellular has used improv to improve diversity awareness
in their organization, and Nike used improv to help • onsider the individual business leaders and whether
C
managers design new shoes (Hastings, 2009; Thilmany, they would benefit from this type of experience.
2007). Alain Rostain, founder of the New York-based – re there specific leaders, teams, divisions or
A
consulting firm Creative Advantage, has used improv with business units that would be appropriate or others
such clients as Kraft, Starbucks, GE and R.J. Reynolds to that might resist this non-traditional approach?
improve creativity and brainstorming (Golden, 2011).
• valuate other leadership development activities
E
Improv can develop a wide range of skills and behaviors, that may already be in place to develop these skills
but it does have limitations and may not be the best and whether they would be more or less effective
solution for every talent development challenge you face. than improv.
COMING SOON!
IMPROV
TO A WORKPLACE NEAR YOU
21
23. If you’re ready totake the
next step in your career,
make a quick trip back to
the classroom first.
E X E C U T I V E D E V E L O P M E N T I N S T I T U T E
At UNC’s Executive Development Institute,
you’ll gain the core knowledge of an MBA program
without the long-term time commitment. You’ll
also learn how to view the business world from a
senior executive’s perspective. And you’ll develop the
UNC EXECUTIVE DEVELOPMENT
key leadership characteristics that lead to effective
The Power of Experience.
strategic performance. The result? In two weeks,
you’ll be fully prepared for that next step.
To learn more, visit www.edi.uncexec.com.
23
25. TA L E N T B U I L D E R S : D E V E L O P I N G Y O U R P E O P L E
Promise Become Students of How
This white paper will benefit all leaders who are serious
about managing talent effectively in their organization.
to Build Better Leaders,
It also provides helpful perspective for HR Professionals Faster
and Talent Management Specialists in the field. The Talent builders stay current on talent management best
authors, Jim Shanley, Corey Seitz and Marc Effron, are practices. They read at least one book or article a year on
three of the world’s most experienced and accomplished building talent. They invite experts to audit their talent
Talent Management Practitioners and have led the Talent building practices and are open to implementing new
Management Function at some of the world’s most innovations that fit their business and talent development
successful talent machines, including Bank of America, needs. They reach out to great talent developers inside
Avon, Novartis and Johnson Johnson. and outside their organization to discuss and learn new
tips and tactics to develop their team. Great developers
leverage “teaching moments”. They recognize and act
Win Today and in the on memorable teaching moments, especially with high
Future potentials. They continually ask their people two critical
questions:
Talent builders identify the organizational capabilities
• What did you learn from that?
and talent that they need to have in their organization
to perform at a high level in today’s environment. • What would you do differently next time?
In addition, they also look ahead and identify the
organizational capabilities and talent that they will
need to win in the marketplace in 36-48 months.
Ensure that Talent
This requires that they assess their talent against both
today’s requirements and tomorrow’s projected Is on the Agenda
requirements. It means that they must constantly ask Great talent builders communicate to their organization
themselves, “Am I recruiting and developing against that leadership and talent matters…they are passionate
tomorrow’s standards?” They ensure that no matter what that the organizations with better leaders will drive higher
form their organization’s talent review process takes, they results. They demand that they and their leaders are
are looking to the future and predicting what type of producing leaders better than themselves. They ask their
talent and how much talent they will need for the future. direct reports several key questions:
• Whom do you have in your organization/team today
Drive and Expect who is as good or better than you and could replace
World-Class Performance you immediately?
• Whom do you have in your organization/team who
at All Levels has the potential to be as good or better than you
They demand performance that is at a “higher standard”. and could replace you in 1-2 years?
Talent builders drive the mindset in their organization that
• Whom do you know within the company but outside
performance must continually improve. They are never
your organization who could replace you (and is as
satisfied with status quo performance. They realize that
good or better than you?) today and in 1-2 years?
without top performance, nothing else much matters.
• Whom do you know externally who could replace
you today and in 1-2 years?
25