Watershed co-hosted a workshop with ATD Nashville and Rustici Software to help L&D pros get started with learning analytics. Facilitators Mike Rustici and John Mattox, Ph.D., explored the five steps of learning analytics and helped attendees choose the right approach for their learning analytics programs.
2. “Scalable learning is the
new reason for large
organizations to exist”
John Hagel
Founder
Deloitte Center for the Edge Innovation
3. Discover
Discover new knowledge through
study, experimentation and
practice.
Digest
Assess the validity of knowledge
and ensure it has a positive
impact.
Distribute
Rapidly disseminate new
knowledge to the entire
organization.
Discover
Digest
Distribute
Learning Embedded Everywhere
5. isn’t it
because…
Small moves, smartly made
“In an exponential world, small moves,
smartly made can set big changes in motion”
– John Hagel
6. getting
there
What is the smartest move organizations can make right now to lay a foundation and
set themselves on a path to becoming a scalable learning organization?
What is the biggest enabler of scalable learning?
7. “Without data you’re just
another person with an
opinion”
W Edward Deming
Management Systems Legend
13. A brief history of
Learning Technologies
Interoperability
SCORM 1.2
2001
Executive
Order 13111
1999
SCORM 2004
4th Edition
2009
Project
Tin Can
20102004 2006
2013
Experience API
1.0.0
2012 2014 20172000
1.0 1.1 200
4
2004
2nd
ed.
2004
3rd ed.
DoDI 1322.26 Enterprise-
Scale
Deployments
1.0.1 1.0.2 1.0.30.9 0.95
2016
14. xAPI &
learning analytics platform
all data
Learning is so much bigger than the
LMS. Track informal and experiential
activities.
xAPI and the LRS present a transformative
technology for learning analytics. Seamlessly
bring all your data together in one platform and
perform real-time analytics tailor made for L&D.
business
impact
Include data about behaviors and
performance in your learning
analytics real time
Create, view and distribute reports
seamlessly in real-time.
15.
16. Understanding learning analytics: Complexity
measurement
The simple act of
tracking things and
recording values. Can
be passively or
actively collected.
evaluation
The process of trying to
make meaning from the
data measured.
Descriptive analytics.
Does the data mean
something “good” or
“bad”?
advanced
evaluation
When data sets get large
enough we can use
advanced evaluation
techniques to discover
powerful insights. Data
mining, AI, machine
learning, etc.
predictive and
prescriptive
Make predictions and
decisions based on data
and advanced
algorithms.
Recommendations
engines are the best
example in learning.
17. Learning
Experience
Learner
Learning
Program
Understanding learning analytics: Categories
Understand an overall
learning program. Is this
initiative helping to meet
business objectives.
Understand a learner or group
of learners. Ensure
organizational readiness and
compliance.
Understand more about a
specific learning activity.
Maximize effectiveness and
spot problems.
19. next generation
learning ecosystem
03connected to
outcomes
Learning outcomes on their own are
only marginally interesting. Incorporate
learning into larger business objectives
and bring the data together to show its
effectiveness.
02
mobile, social,
informal
Learning happens everywhere and now
we can track it. The constraints of old
are gone. Embrace the types of
learning that your learners want to use.
Provide resources in the moment of
need where the learners live.
01away from the LMS
How much of what you’ve learning in
your lifetime has come from an e-
learning course in an LMS? LMSs
aren’t going away, but it’s time for us to
recognize that learning happened
everywhere.
20. Visa University:
A new
ecosystem
A next generation learning
ecosystem brings this all
together.
This example from Visa
shows that the LMS is just
one small part of a modern
learning ecosystem.
24. Code Blue
When a patient's heart stops
and needs to be resuscitated.
25. Learning Ecosystem
Mock Code Blue/ In person
Observation
Mobile Simulation/
Defibrillator App
LMS Training Lab
Learning Analytics Platform
(Watershed)
26. Does the amount of training activity over time have an impact on Code Blue KPIs?
Time to
defibrillation
Time to
chest
Time to first
drug
Arrival
times
Time hands off
chest
Training
Activities
Jan 2015 Feb 2015 Mar 2015
Apr
2015
May
2015
Jun
2015
Training
Activities
CodeBlue
KPIs
Training Effectiveness
28. Getting Started
yes you can!
01
Gather your data
Just start collecting data in a common
format in a central location. Make it low
friction and ensure you have access.
04
Explore your data
Starting to understand not just “what” is
happening but “why”. Form questions and
see if your data can answer them. Often the
best answer another question. Can you see
impacts on the larger business?
02
Get to know your data
Understand what you have. What is out
there, what is reliable and what is missing?
Do some simple evaluations and create
baselines.
05
Experiment with your data
Start a new learning program with analytics
in mind. Set up well controlled experiments
and A/B tests to validate a hypothesis.
Create a culture of continuous
improvement.
03
Operationalize your data
Automate your way out of “Excel hell”.
Define some interesting metrics and KPIs,
start monitoring them routinely. What trends
do you notice?
06
Show off your data
We’ve long known that learning is vital, not
we can probe it. Go forth and show the
world!
29. our
contacts
watershed
210 Gothic Ct
Franklin, TN 37067
+1-844-220-0822
http://watershedlrs.com
@watershedlrs
me
Mike Rustici
Founder and CEO
mike.rustici@watershedlrs.com
@mike_rustici
Notes de l'éditeur
Hi, I’m Mike
I want to talk about big shifts and big opportunities in L&D that are happening
Lots of new things are suddenly possible. Some big and lofty but also some big wins for small steps forward.
So “what is scalable learning”?
Scalable learning is not just learning new things that others know, but rather
Creating and discovering new knowledge – this is more than just finding a new book to read
Assessing its validity
Rapidly disseminating it
Ensuring it has a positive performance impact
Discover -> Digest -> Distribute
Is your LMS going to get you there?
Fortunately there is a new generation of learning tools, models and ecosystems emerging
Option – “is your learning org going to get you there”
But the good news is, In an exponential world, small moves, smartly made can set big changes in motion
Is this a better opening concept?
Learning analytics is the big thing that’s now possible
It’s not big and scary, many types of learning analytics that range from simple to complex
It’s about attitude and strategy
It’s learning’s turn to take a seat at the table
To do so, we need to think strategically, we need to think in the same terms the business thinks in and align our strategy and outcomes to the strategy and outcomes of the business - often performance
Let’s take a look at what happened to marketing over the past 20 years. You’ll see a lot of parallels to learning and a path forward for us.
Learning depts refer to their audience as customers. And that makes so much sense.
Marketing analogy
Marketing went through this same transition over the past 20 years and we’re seeing the same thing happen in learning (summary of this article)
More tools
More data
Missing piece is analytics
xAPI is enabling this transition in learning, just like marketing
It’s all about removing friction
Removing just a little bit of friction has a big impact
Apple design methodology
Kirkpatrick model has been around since 1954, but how much of it are we using.
Remember, when we asked people why they aren’t doing more learning analytics, one of the chief responses was “it’s too hard”
That is changing, the friction is disappearing and that will have a profound impact.
What does this look like in the real world
There is a shift in the learning eco system
New learning ecosystem (summary of this webinar)
Can move away from the monolithic LMS
LRS centric, mobile, social, informal,
Removed constraints
Connect to performance
One of the first to deploy and publicize at enterprise scale is Visa
Notice:
-LMS is just one small part
-Inclusion of non-learning tools Sharepoint and SurveyMonkey
-Learning Experience platform as front end
Applied is the world’s leading industrial distributor, operating at an annual revenue of nearly $3billion.
Comprised of hundreds of distribution centers, every dot represents a small business with a manager and team
To meet business targets, management needs to be competent on financial and sales acumen, which is why AIT invested $81 million to convert to a new ERP system to streamline reports and sales/operational accountabilities that were previously tracked manually by disparate systems.
After successful SAP training and system implementation, many business KPIs dipped. SAP put a spotlight on operational metrics that were distilled into stoplight reports. But SCMs responsible for impacting results weren’t able to make improvements simply by seeing numbers on stoplight charts.
Applied’s Organizational Learning (OL) department identified that SCMs could benefit from a better understanding of business insights, finance, and P&L concepts to see how their actions can directly impact financial statements
Here’s why. Symptoms, does not point to the solution
Once they got SAP, we focused on operations efficiency
Revenue
Gross Profit
Operating Expense
Billing
Inventory
Freight
Processing
Rebates
What do you notice? Notice a lot of talented people at Applied
If you don’t get your learning data aligned to business purpose, all your decision making is based on Intuition and hope.
And, hope is not a methodology.
Red = Failure ?
Typical LMS style reporting – but only reflects LMS output
Fluidity of Competency
What does this do for leadership?
Legitimized learning data
Getting this done took a lot of excel work – V-Lookups, etc.
What does it really tell us? Johnny did not complete some courses
Background info (do not deliver this portion)
A Group
Should be running profitable locations.
A+ group = best in class
B Group
Grasps financial concepts, but training and coaching needed on application of SOW.
C Group
Good practices, but may struggle with financial impacts of COGS and Ops expenses. Review P&L statement concepts with manager.
D Group
Needs the most help. Evaluate. Review webinars, coach on concepts and technique. Watch for improvement trends.
VIDEO
Not just big ROI, but little stories, with each visualization helping you to dig deeper. That’s the value of having everything in one, automated place.
It’s important to develop skills during training before performing tasks in the real world.
Real-world errors can have significant consequences.
The organization needs to be confident of learner’s competency.
Skills to be learnt are practical and physical.
MedStar’s training program is made up of three main elements:
The Learning Management System manages learner and organization data, and tracks a record of classroom training attended.
A mobile app, called Zoll, allows clinicians to practice the steps involved in defibrillation on their own as often as they want to.
In situ mock Code Blue simulations enable clinicians to practice resuscitating a medical dummy as a team.