Measuring What Ma.ers
Moving beyond basic LMS metrics to an outcomes-­‐based
measurement model
June
20,
2017

Housekeeping
#LearningImpact
@Axonify
Event
is
being
recorded
Please
leverage
the
Q&A
window

Dani Johnson
Vice President, Learning and Development
Research
Bersin by Deloi.e
Deloi.e ConsulDng LLP
Carol Leaman
Chief ExecuDve Officer
Axonify

Who we are
Global provider of leading practices, trends, and benchmarking research in talent management,
learning, and strategic HR.
6 research practices
• Human Resources
• Leadership Development
• Learning & Development
• Talent Acquisition
• Talent Management
• Tools & Technology
Offerings
• WhatWorks® Membership: Research, Tools, Education, Consulting
• IMPACT: The industry’s leading conference on the Business of Talent
• Advisory Services & Consulting
Human
Resources
Talent
Acquisition
Leadership
Development
Learning &
Development
Talent
Management
Tools &
Technology

Measuring What MattersCopyright © 2017 Deloitte Development LLC. All rights reserved. 5
• Background on the New High-Impact Learning
Organization
• Four major findings from the High Impact
Learning Organization Study
• Case study – measurement in practice
Agenda

Measuring What MattersCopyright © 2017 Deloitte Development LLC. All rights reserved. 6
The new HILO model

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 7
Learning Organization Maturity Model
BersinbyDeloitte
Episodic / Programmatic
Effects Change through Courses / Programs • Relies on a Top-Down Approach
to Development • Drives Development of Selected Knowledge & Skills •
Focuses on Enabling Performance in Current Role
Level 1
Responsive / Contextualized
Effects Change through Creating Employee-Centric Learning • Design Thinking
Applied to Employee-Development Initiatives • Focuses on Implementing
Transparent Systems & Processes That Personalize Development
Level 2
Continuous / Empowering
Effects Change through Empowering Individual-Driven Development •
Leverages Experiences & Consistent Feedback for Development • Continuously Evolves Technology
Infrastructure • Focuses on Creating Conditions That Enable Development
Level 3
Anticipatory / Flow
Effects Change by Anticipating Needs and Then Augmenting & Leveraging the Developmental
Opportunities Intrinsic in Work • Concentrates on Long-Term Career Success & Creating Enduring
Commitment to Development • Fosters a Culture of Courage & Innovation • Integrates Feedback
Loops & Data into Work Design • Focuses on Supporting Employees’ Ability to Evolve & Adapt
Level 4
6%
20%
39%
35%
Source: Bersin by Deloitte, 2017

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 8
Learning Organization Maturity ModelBersinbyDeloitte
Episodic / Programmatic
Level 1
Responsive / Contextualized
Level 2
Continuous / Empowering
Level 3
Anticipatory / Flow
Level 4
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017
• Focus on data and reports
easily obtained from the LMS
or other system of record.
• Heavy focus on completion
data, assessments, “doing”
information.
LEVEL
1

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 9
Learning Organization Maturity ModelBersinbyDeloitte
Episodic / Programmatic
Level 1
Responsive / Contextualized
Level 2
Continuous / Empowering
Level 3
Anticipatory / Flow
Level 4
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017
• Data gathering from several
learning sources
• More holistic reports that
provide picture of
development and needs.
• Algorithms and analytics
provide a more personalized
experience for the employee.
LEVEL
2

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 10
Learning Organization Maturity ModelBersinbyDeloitte
Episodic / Programmatic
Level 1
Responsive / Contextualized
Level 2
Continuous / Empowering
Level 3
Anticipatory / Flow
Level 4
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017
• Multiple types of data –
learning and otherwise
• Data usage (performance &
learning) to provide the best
developmental experiences to
individuals
LEVEL
3

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 11
Learning Organization Maturity ModelBersinbyDeloitte
Episodic / Programmatic
Level 1
Responsive / Contextualized
Level 2
Continuous / Empowering
Level 3
Anticipatory / Flow
Level 4
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017
• All available data used to
inform employee
development and work
improvement.
• Focus on using data to
determine appropriate risks
and utilizes mistakes as data
to direct strategy.
LEVEL
4

L&D: Radical Change or Get Out of the GameCopyright © 2017 Deloitte Development LLC. All rights reserved. 12
• Main focus on data and
reports easily obtained
from the LMS or other
system of record.
• Heavy focus on
completion data,
assessments, “doing”
information.
• Data gathering from
several learning sources
• More holistic reports that
provide picture of
development and needs.
• Algorithms and analytics
provide a more
personalized experience
for the employee.
• Multiple types of data –
learning and otherwise
• Data usage (performance
& learning) to provide the
best developmental
experiences to individuals
• All available data used to
inform employee
development and work
improvement.
• Focus on using data to
determine appropriate
risks and utilizes mistakes
as data to direct strategy.
Measurement & Analytics at each level in the HILO model
LEVEL
1
LEVEL
2
LEVEL
3
LEVEL
4
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Measuring What MattersCopyright © 2017 Deloitte Development LLC. All rights reserved. 13
Four Major Insights from HILO

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 14
Use of data for design, by level
Finding 1: Mature organizations use data for better design.
64%
82%
32%
55%
21%
35%
11%
22%
Measuring how our development
experiences are being improved
Incorporating feedback into design
of future development activities
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 15
Use of data for design, by level
Finding 1: Mature organizations use data for better design.
64%
82%
32%
55%
21%
35%
11%
22%
Measuring how our development
experiences are being improved
Incorporating feedback into design
of future development activities
Source: The High Impact Learning Organization Model, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 16
How data may affect design
Finding 1: Mature organizations use data for better design.
82%
66%
64%
59%
35%
35%
32%
30%
27%
21%
17%
14%
13%
3%
2%
1%
Elearning courses (self-paced)
Learning Management System
Webinars
Videos
Business performance dashboards
Social Learning / networking platforms
Virtual classroom platform
Collaboration software / platform
Messagaing
Mobil apps
Podcasts
Search software
MOOC platform / environment
AI
VR
Wearables
Usage of Tech Types
1
2
2
2
3
4
4
4
5
12
26
27
32
34
43
49
Elearning courses (self-paced)
Webinars
Learning Management System
Videos
Podcasts
Virtual classroom platform
Collaboration software / platform
MOOC platform / environment
Social Learning / networking platforms
Wearables
Business performance dashboards
Messagaing
Mobil apps
AI
VR
Search software
Level 4 vs. Level 1 usage
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 17
How data may affect design
Finding 1: Mature organizations use data for better design.
82%
66%
64%
59%
35%
35%
32%
30%
27%
21%
17%
14%
13%
3%
2%
1%
Elearning courses (self-paced)
Learning Management System
Webinars
Videos
Business performance dashboards
Social Learning / networking platforms
Virtual classroom platform
Collaboration software / platform
Messagaing
Mobil apps
Podcasts
Search software
MOOC platform / environment
AI
VR
Wearables
Usage of Tech Types
1
2
2
2
3
4
4
4
5
12
26
27
32
34
43
49
Elearning courses (self-paced)
Webinars
Learning Management System
Videos
Podcasts
Virtual classroom platform
Collaboration software / platform
MOOC platform / environment
Social Learning / networking platforms
Wearables
Business performance dashboards
Messagaing
Mobil apps
AI
VR
Search software
Level 4 vs. Level 1 usage
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 18
Use of performance data by level
Finding 2: Performance data is playing a larger role in employee development.
90%
93% 94%
71%
93%
50%
91%
71%
91%
64%
84%
53%
81%
60%
82%
39%
69%
28%
79%
47%
65%
29%
61%
15%
Formal performance
appraisals
Feedback from customers/
clients
Feedback from managers /
mentors
Peer feedback / ratings Self-evaluations Talent or work management
systems
Level 4 Level 3 Level 2 Level 1
On average, mature organizations collected
performance data from more sources and more
often than less mature organizations.
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 19
Use of data sources by L&D function
Finding 3: Mature organizations use a wider variety of data.
42%
36%
34%
30%
29%
28%
27%
23%
22%
21%
15%
13%
12%
12%
11%
3%
Worker Feedback
Line Manager Feedback
Feedbaack from business line leaders
Assessments based on competency / capability frameworks
Follow-up assessments
Performance metrics
Organizational KPIs
Benchmarking (internal & external)
Observable behavior changes
Talent metrics
Steering committee discussions
Activities of HR specialists such as OD
Data from other human capital functions
Other human capital systems
Usage of worker development methods / initiatives
Automated data collection & analysis
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 20
Data sources used by L&D, Level 1 vs Level 4
Finding 3: Mature organizations use a wider variety of data.
10
5
5
4
4
3
3
3
3
3
3
3
3
2
2
2
Automated data collection and analysis
Other HC systems
Usage of worker development methods
Talent Metrics
Data from other HC functions
Benchmarking
KPIs
Steering committee discussions
Activities of HR specialists
Follow-up assessments
Assessments based on competency/capability frameworks
Observable behavior change
Performance metrics
Feedback from Business line leaders
Line Manager Feedback
Worker Feedback
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 21
Metrics that matter to business leaders
Finding 3: Mature organizations use a wider variety of data.
0%
25%
50%
75%
100%
0% 25% 50% 75% 100%
Valuable
Tracked / Measured
Impact on behavior / job
performance
Sponsor / client
satisfaction
Business results
Impact on talent / organizational
metrics
Line manager satisfaction / feedback
Content access methods
Learning scores /
transfer
Learner
demographics
Hours delivered /
time spent learning
Costs / expenses
Learner
satisfaction /
feedback
Enrollments, starts,
usage, and completions
Source: State of Learning Measurement, Bersin by Deloitte, 2015

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 22
Continuous sensing data collection by maturity level
Finding 4: Data collection isn’t just something that happens at the end of a course.
55%
22%
14%
10%
Level 4 Level 3 Level 2 Level 1
Source: The High Impact Learning Organization Study, Bersin by Deloitte / Dani Johnson, 2017

Learning MeasurementCopyright © 2017 Deloitte Development LLC. All rights reserved. 23
Holistic learning measurement / analysis / data collection
Finding 4: Data collection isn’t just something that happens at the end of a course.
Outcome
(the business)
Efficiency
(L&D)
Alignment
(managers /
leaders)
Effectiveness
(learners)
• Time to develop
• Cost to develop / deliver
• Cost of development per minute / hour
• Cost per participant
• Resource hours to develop / deliver
• Total enrollments
• Learning offerings correspond to competency model
• Learning has strong executive sponsors
• Managers support participation among team members
• Program objectives align with business needs
• New behaviors reflect what was introduced in a
learning offering
• New knowledge and skills
• Learner satisfaction
• Adoption
• Key performance indicators (KPIs)
• Revenue
• Profitability and growth
• Market share
• Customer / satisfaction
• Employee engagement / retention
• Productivity
Source: State of Learning Measurement, Bersin by Deloitte, 2015

Measurement in PracDce

Case Study: Large Global OrganizaDon
• Over 75,000 employees
• Currently focused on logisDcs and distribuDon network
• Over 150 faciliDes across the U.S.
• Looking to dramaDcally reduce safety incidents
• Aiming to build a world-­‐class safety culture

Assess Hope

Large
Global
OrganizaJon

Did the number of
safety incidents
meet expectations?
Large
Global
OrganizaJon

Was the ladder
climbed in the
proper way?
Did the number of
safety incidents
meet expectations?
Large
Global
OrganizaJon

How do we measure the behaviors?

How do we measure the behaviors?

Did they gain
knowledge on the
proper technique
for climbing a
ladder?
Did the number of
safety incidents
meet expectations?
Large
Global
OrganizaJon
Was the ladder
climbed in the
proper way?

How do we measure knowledge?

How do we measure knowledge?

How do we measure knowledge?

Ladder Handling
Learning Modules
and Reinforcement
Did the number of
safety incidents
meet expectations?
Large
Global
OrganizaJon
Did they gain
knowledge on the
proper technique
for climbing a
ladder?
Was the ladder
climbed in the
proper way?

How do know if it’s the right content?
High
level
of
knowledge
Correct
behavior
demonstrated
Good
content
High
level
of
knowledge
Incorrect
behavior
demonstrated
Content
may
be
an
issue

Did the number of
safety incidents
meet expectations?
Large
Global
OrganizaJon
Ladder Handling
Learning Modules
and Reinforcement
Did they gain
knowledge on the
proper technique
for climbing a
ladder?
Was the ladder
climbed in the
proper way?

Measurement: The Future

Global LogisDcs Company
• Focus on decreasing
shipping errors, producing
higher revenue
• Content focused on
procedures
• Significant improvement
overall revenue
• Content directly Ded to
business impacts

“Outcomes First” Method

“Outcomes First” Method

Dani Johnson
Vice President, Learning and Development
Research
Bersin by Deloi.e
Deloi.e ConsulDng LLP
Carol Leaman
Chief ExecuDve Officer
Axonify
Q&A

Thank you!