With the growing popularity of online and advanced learning technologies, more data on learner performance and training effectiveness are becoming available at a rapid pace. Even with these resources, learning and development professionals struggle with how to collect, gain access to, and use data to make informed decisions about training programs. In this interactive session, you’ll first hear from internal and external evaluation consultants who will give a brief overview of measurement strategies for evaluating training programs. This will be followed by a group activity designed to promote the application of the measurement techniques discussed in the session within a case example.

1. TICE 2018 - June 27, 2018
Demonstrating Business Value:
APPLYING DATA SCIENCE TO TRAINING
Evaluation
One-Sheeter
Takeaway
LEVEL AND DESCRIPTION METHODS OR MEASURES WHY TO USE THIS LEVEL
Level 1
Reaction
Immediately after training
to measure experience,
quality of facilitation,
recommend to others, face
validity of program -
whether learners think the
program did what was
intended - and learning
objectives were met
❖ Surveys
❖ Word of mouth
❖ Focus groups
❖ Facilitator feedback
❖ Demonstrate chain of impact for
higher level evaluation
❖ Identify perceived strengths and
weaknesses of the program
❖ Evaluate facilitator effectiveness
❖ Market programs
❖ Determine how learners plan to use
newly acquired skills
❖ Forecast business impact of
program
Level 2
Learning
Before, during, and/or
immediately after training to
provide feedback to
learners, improve program,
and have learner
demonstrate learned
knowledge, skill, or attitude
targeted in program
❖ Knowledge checks
❖ Assessments
❖ Projects
❖ Behavioral Scoring
❖ Facilitator Feedback
❖ Demonstrate chain of impact for
higher level evaluation
❖ If the program represents critical
skills or knowledge that must be
transferred to the job because of
regulations, profitability, or
customer service
❖ The program is offered as part of a
certificate program

2. TICE 2018 - June 27, 2018
Demonstrating Business Value:
APPLYING DATA SCIENCE TO TRAINING
Evaluation
One-Sheeter Takeaway
LEVEL AND DESCRIPTION METHODS OR MEASURES WHY TO USE THIS LEVEL
Level 3
Application
Collected typically 30 to 90
days post-training to
measure learning transfer
on the job, identify or
confirm barriers and
enablers to success,
frequency of use
❖ Survey (for learner or
manager)
❖ Direct observation
❖ Feedback from
manager
❖ Interviews
❖ Focus groups
❖ Demonstrate chain of impact for
higher level evaluation
❖ The program targets specific
business objectives
❖ Behavior change is critical for
success or business continuity
❖ The program is expensive
❖ The program is highly visible in the
business unit
Level 4
Business
Impact
Collected typically 60 to 180
days post-training to
provide evidence of the
contribution to the business
unit or evidence of the
effectiveness and efficiency
of the learning program
❖ Sales metrics
❖ Customer service
surveys
❖ Other business
metrics
❖ The program is aimed at meeting
specific business objectives
❖ The program targets a business unit
strategic initiative
❖ Large team member populations
will participate
❖ The program in expensive or high
visibility