Training program effectiveness a measuring instrument (1)

Training Program Evaluation
A Measuring Instrument
A Manual
#2447, Sector 79, Mohali.
www.thegrowthfactor.in
Mb: 9781925257. Email: thegrowthfact@gmail.com

1
1
Introduction:
What gets measured ….. gets done!
This manual describes key aspects regarding measuring the effectiveness of a
training programme.
Training Evaluation: Need, Significance, and Purpose
❏ To determine the effectiveness of a training programme and find out
where the desired expectations have and have not been met.
❏ To know the strength and weakness of the training program and
suitably revise and redefine.
❏ To determine whether a program should be continued or dropped.
❏ To test the clarity, validity of the tools of the training programme.
❏ To provide information to develop strategy for the future programme.
❏ To identify how it benefits the participants.
❏ To identify the barriers and constraints and how to reduce it.
❏ To analyze in terms of cost and benefits of a training programme.
❏ To give feedback to instructors and course developers.
❏ To justify and increase in the budget.
Training Evaluation: Principles
❖ Summative Vs Formative Training evaluation:
❏ Formative techniques of training evaluation aim to catch problems (and
fix them) early on, before they negatively impact learning.
❏ Summative techniques are also known as post-training evaluation
techniques, because they happen after training is completed.
❖ Determine What to Measure:
The popular training effectiveness measures are:
➔ New skills and knowledge
➔ Learning experience
➔ Culture impact
➔ Customer happiness
➔ Efficiency impact
➔ Financial impact
❖ Choose the Right Training Evaluation Tools

2
2
➔ Observations
➔ Tests
➔ Surveys
➔ Online Interviews
➔ Focus groups
➔ Performance records
❖ Select the Appropriate Training Evaluation Method:
1. Krikpatrick’s Four -level Training Evaluation Model
2. The phillips ROI Model
3. Kaufman’s Five level of Evaluation
4. Anderson’s Model of Learning Evaluation
Kirkpatrick’s Four-level Training Evaluation Model:
What is this method of Evaluation:
❏ Dr. Donald Kirkpatrick developed this model in the 1950s.
❏ The Kirkpatrick model is probably the best known model for analyzing
and evaluating the results of training and educational programs.
❏ This method breaks the evaluation process down into four simple levels.
How to Implement this method of Evaluation?
Evaluation
level &
type
Evaluation
description and
characteristics
Examples of evaluation
tools and methods
Relevance and
practicability
Reaction Reaction evaluation is
how the delegates felt,
and their personal
reactions to the
training or learning
experience, for
example:
Did the trainees like and
Typically 'happy sheets'.
Feedback forms based
on subjective personal
reaction to the training
experience.
Verbal reaction which
can be noted and
analyzed.
Can be done
immediately when
the training ends.
Very easy to obtain
reaction feedback.
Feedback is not
expensive to gather
or to analyze for

3
3
enjoy the training?
Did they consider the
training relevant?
Was it a good use of their
time?
Did they like the venue,
the style, timing,
domestics, etc.?
Level of participation.
Ease and comfort of
experience.
Level of effort required
for making the most of
the learning.
Perceived practicability
and potential for
applying the learning.
Post-training surveys or
questionnaires.
Online evaluation or
grading by delegates.
Subsequent verbal or
written reports given by
delegates to managers
back at their jobs.
groups.
Important to know
that people were not
upset or
disappointed.
Important that
people give a positive
impression when
relating their
experience to others
who might be
deciding whether to
experience the same.
Learning Learning evaluation is
the measurement of the
increase in knowledge
or intellectual
capability from before
to after the learning
experience:
Did the trainees learn
what was intended to
be taught?
Did the trainee
experience what was
intended for them to
experience?
What is the extent of
advancement or change
in the trainees after the
training, in the direction
or area that was
intended?
Typically assessments
or tests before and after
the training.
Interview or
observation can be used
before and after.
Methods of assessment
to be closely related to
learning aims.
Measurement and
analysis is possible and
easy.
Reliable, clear scoring
and measurements
need to be established, so
as to limit the risk of
inconsistent assessment.
Hard copy, electronic,
online or interview
assessments are all
possible.
Relatively simple to
set up, but more
investment and
thought required
than reaction
evaluation.
Highly relevant and
clear-cut for certain
training such as
quantifiable or
technical skills.
Less easy for more
complex learning
such as attitudinal
development.
Cost escalates if
systems are poorly
designed, which
increases work
required to measure
and analyze.
Behavior Behavior evaluation is
the extent to which the
Observation and
interview over time are
Measurement of
behavior change is

4
4
trainees applied the
learning and changed
their behavior, and
this can be immediately
and several months
after the training,
depending on the
situation:
Did the trainees put
their learning into effect
when back on the job?
Were the relevant skills
and knowledge used?
Was there noticeable
and measurable change
in the activity and
performance of the
trainees when back in
their roles?
Was the change in
behavior and new level
of knowledge
sustained?
Would the trainee be
able to transfer their
learning to another
person?
Is the trainee aware of
their change in
behavior, knowledge,
skill level?
required to assess
change, relevance of
change, and
sustainability of change.
Arbitrary snapshot
assessments are not
reliable because people
change in different ways
at different times.
Assessments need to be
subtle and ongoing, and
then transferred to a
suitable analysis tool.
Assessments need to be
designed to reduce
subjective judgment of
the observer or
interviewer, which is a
variable factor that can
affect reliability and
consistency of
measurements.
The opinion of the
trainee, which is a
relevant indicator, is also
subjective and
unreliable, and so needs
to be measured in a
consistent defined way.
360-degree feedback is
useful method and need
not be used before
training, because
respondents can make a
judgment as to change
after training, and this
can be analyzed for
groups of respondents
and trainees
Assessments can be
designed around relevant
performance scenarios,
less easy to quantify
and interpret than
reaction and learning
evaluation.
Simple quick
response systems
unlikely to be
adequate.
Cooperation and skill
of observers, typically
line-managers, are
important factors,
and difficult to
control.
Management and
analysis of ongoing
subtle assessments
are difficult, and
virtually impossible
without a well-
designed system from
the beginning.
Evaluation of
implementation and
application is an
extremely important
assessment - there is
little point in a good
reaction and good
increase in capability
if nothing changes
back in the job,
therefore evaluation
in this area is vital,
albeit challenging.
Behavior change
evaluation is possible
given good support
and involvement
from line managers
or trainees, so it is
helpful to involve

5
5
and specific key
performance indicators
or criteria.
Online and electronic
assessments are more
difficult to incorporate -
assessments tend to be
more successful when
integrated within
existing management
and coaching protocols.
Self-assessment can be
useful, using carefully
designed criteria and
measurements.
them from the start,
and to identify
benefits for them,
which links to the
level 4 evaluation
below.

6
6
Results Results evaluation is
the effect on the
business or
environment resulting
from the improved
performance of the
trainee - it is the acid
test.
Measures would
typically be business or
organizational key
performance indicators,
such as:
Volumes, values,
percentages, timescales,
return on investment,
and other quantifiable
aspects of
organizational
performance, for
instance; numbers of
complaints, staff
turnover, attrition,
failures, wastage, non-
compliance, quality
ratings, achievement of
standards and
accreditations, growth,
retention, etc.
It is possible that many
of these measures are
already in place via
normal management
systems and reporting.
The challenge is to
identify which and how
relate to the trainee's
input and influence.
Therefore it is important
to identify and agree
accountability and
relevance with the
trainee at the start of the
training, so they
understand what is to be
measured.
This process overlays
normal good
management practice - it
simply needs linking to
the training input.
Failure to link to training
input type and timing
would greatly reduce the
ease by which results can
be attributed to the
training.
For senior people
particularly, annual
appraisals and ongoing
agreement of key
business objectives are
integral to measuring
business results derived
from training.
Individually, results
evaluation is not
particularly difficult;
across an entire
organization it
becomes very much
more challenging, not
least because of the
reliance on line-
management, and the
frequency and scale
of changing
structures,
responsibilities and
roles, which
complicates the
process of attributing
clear accountability.
Also, external factors
greatly affect
organizational and
business
performance, which
cloud the true cause
of good or poor
results.
Advantages and Disadvantages of this Method of Evaluation:
Advantage:

7
7
★ The model can be used to evaluate classroom training as well as
eLearning.
★ The model provides a logical structure and process to measure learning.
★ When used in its entirety, it can give organizations an overall
perspective of their training program and of the changes that must be
made.
★ It has been used to gain a deep understanding of how eLearning affects
learning, and if there is a significant difference in the way learners learn.
Disadvantage:
★ Reaction level: Results should not be used as a solid basis for changing
the educational content or strategy.
★ Learning level: Requires more money and time than level 1. Also
requires greater insight to evaluate processes to develop valid measures
of learning.
★ Behaviour level: Requires significantly more investment of time and
money. Requires in-depth insight into performance interventions and
root causes of performance deficiencies.
★ Result Level: Substantial level of investment and expertise are required
to develop level 4. Often hard to decide whether or not this level is
required.

8
8
The Phillips ROI Model:
Why this method of Evaluation?
❏ Jack Phillips felt that Kirkpatrick's Learning Evaluation Model was
incomplete. One of the biggest changes he made was adding a fifth level
of evaluation, return on investment (ROI).
❏ This level, commonly referred to as level 5 evaluation, compares the
program's overall cost to its monetary benefits.
❏ This is presented as a cost/benefit ratio.
❏ A key component to evaluating at this level is isolating the training
benefits and eliminating any non-training factors that may have
contributed to the organizational impact.
What is this method of Evaluation?
❏ The Phillips ROI Methodology is a step-by-step credible process that
meets the need to show the value used by organizations all over the
world.
❏ The methodology is a simple to use and proven process that will
generate a balanced set of data that is believable, realistic and accurate –
particularly from the perspective of sponsors and key stakeholders.
❏ The ROI Methodology is a balanced approach to measurement that
captures six types of data:
1. Reaction and Planned Action (Level 1)
2. Learning (Level 2)
3. Application and Implementation (Level 3)
4. Business Impact (Level 4)
5. Return on Investment (Level 5)
6. Intangibles
ROI (%) = [(Total Revenue - Total Cost)/Total Cost] * 100
The Phillips ROI model works by focusing on the following areas:
● Data collection
● Isolating the impact of training versus the impact of other factors
● Accounting for intangible benefits
● Calculating the return on investment (ROI)

9
9
To meet the end goal of calculating ROI, the Phillips model calls for business
data to be gathered before, during and after the training. This data is analyzed
for quantifiable factors such as:
● Process improvements
● Productivity improvements
● Increased profits
Using this data, you can then compare the cost of the training with the
monetary benefit that is provided. This gives you an indication of the value of
the training, and the impact it had on the organization's bottom line.
Advantages:
★ Balanced set of measures.
★ Bridges evaluation disciplines
★ Balanced research and reality
★ Flexible and Credible
Disadvantages:
★ The ROI process can fail or succeed based on these implementation
issues.

10
10
Kaufman’s Five Level of Evaluation:
❏ Kaufman divides Kirkpatrick’s Level 1 (Reaction) into two sections:
“Input” and “Process.”
❏ He grouped Kirkpatrick’s Levels 2 and 3 as ‘Micro’ levels.
❏ Kaufman’s fifth level evaluates results for both the customer and society
in general.
❏ This model is positioned as “more practical” than Kirkpatrick.
❏ Kaufman’s model is one of a number of learning evaluation models that
build on the Kirkpatrick Model, one of the most popular and widely-
used training evaluation models of all time.
❏ As Kaufman’s Five Levels of Evaluation is closely based on Kirkpatrick’s
four levels, it’s worth recapping the Kirkpatrick levels first.
Level 1a: Input
➔ This covers the training materials such as digital resources that are used
to support the training or coaching.
Level 1b: Process
➔ The second part of the first level measures process acceptability and
efficiency. In other words, the actual delivery of the learning experience.
Level 2: Acquisition

11
11
➔ Kaufman’s second level studies the payoffs for both individuals and
small groups. The ‘micro-level client’ would usually be the learner, so
you would be studying whether they acquired the learning and whether
they used it on the job.
Level 3: Application
➔ The third level evaluates how well participants utilize what they learned
in their on-the-job performance.
Level 4: Organisational payoffs
➔ Kaufman’s fourth level measures payoffs for the organization as a whole.
The ‘macro-level client’ would typically be the business or organization
undertaking the evaluation. This level includes performance
improvement evaluations and a cost-benefit and/or cost-consequence
analysis.
Level 5: Societal Outcomes
➔ Kaufman’s fifth level focused on what he termed ‘mega-level clients’.
This could refer to a business’ clientele and/or to society as a whole.
Advantages:
★ The value of separating quality standards for your materials from
standards for your delivery method.
★ The value of data besides the input you receive from learners.
Disadvantage:
★ The addition of societal consequences.
★ The overly complex terminology.

12
12
Anderson’s Model of Learning Evaluation
❏ This Anderson’s Model for Learning Evaluation is a high levelled
evaluation model.
❏ Its main use is for organization’s level rather than specific learning
interventions. Thus, it aligns the learning function with the organization’s
strategic priorities.
❏ The Anderson model of learning evaluation is a unique three-stage
learning evaluation cycle that is designed to be applied at an
organizational level.
❏ The Anderson model distinguishes itself by focusing on the two areas that
organizations frequently struggle with. These are:
➔ Conducting evaluations
➔ Gathering evidence
❏ Anderson’s Value of Learning model helps to align an organization’s
training programs with its strategic priorities.
❏ It does this by focusing on the evaluation of learning strategy, rather than
the outcome of individual programs.
❏ Anderson’s model is a three-stage cycle that helps an organization
determine the best training strategy for their needs.
The three stages are:
➢ Stage 1: Determine the current alignment of training against strategic
priorities for the organization.
➢ Stage 2: Use a range of methods to assess and evaluate the contribution
of learning.
➢ Stage 3: Establish the most relevant approaches for your organization. 
❏ The most relevant approach for any given organization will depend on
its stakeholders’ goals and values. Anderson’s model suggests four
categories of measure, as follows:
● Emphasis on the short-term benefits
● Emphasis on long-term benefits
● Senior management trust in learning contribution
● The organization requires learning value metrics

13
13
Advantages:
★ No ‘One-size-fits-all’.
★ Alignment of learning function with strategic priorities.
★ Addresses evaluation and value challenges.
Disadvantages:
★ Limited depth - it only gives an organization a high-level evaluation of
the effectiveness of their training programs
★ Limited direction on direct learning evaluation-It doesn’t give
specific direction about how to evaluate each learning initiative or
learning journey.
Conclusion:
The evaluation of training shows that whether the output, outcome and
impact of a training programme on measures regarding training effectiveness
indicators found it to leading to achievement of objectives of the organizations
and training need analysis. It also informs whether the process indicators
such as training methods, size of participants, and periodical training were
appropriate and adequate or not. Moreover, it must be kept in mind that the
effectiveness is also dependent on individual characteristics like listening
capability, nature and self-interest of trainees. Evaluation can add value to the
training itself. So the organizations should make attempts to rectify the snags
and pitfalls that have crept into needs assessment and evaluation of training,
so that training can become a powerful intervention to improve manager’s
performance and subsequently impact organizational performance.

14
14
The Growth Factor
The Growth Factor has been established for strengthening the competence of
individuals and organizations through the provision of world-class
competency building interventions and coaching from one of the best in that
particular field. We have brought together an ensemble of eminent experts
who have proven experience in corporate education and have guided national
& MNCs.
We are committed to delivering a significantly high rate of return on
investment (RoI) on interventions implemented with our collaboration. Our
competent team of trainers have been carefully chosen to add significant
value to corporate results. We work towards developing reflective, scholarly
practitioners who are equipped with cutting-edge knowledge and skills. Our
pedagogy is deeply rooted in action and actionable learning that is scholarly in
letter and spirit. We are a one-stop solution for all your training & consultancy
needs as we bring to the table the following value propositions for the
organizations:
• Quality assurance with respect to training & trainers
• A wide variety of trainers from different fields to choose
from
• Multiple trainers to choose from the same field
for Training, Consultancy, & Business Coaching:
Mb: 70093-56858, 97819-25257. Email: thegrowthfact@gmail.com
www.thegrowthfactor.in
Human Potential Maximization….. that is what we stand for

Training program effectiveness a measuring instrument (1)

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Training program effectiveness a measuring instrument (1)

Similaire à Training program effectiveness a measuring instrument (1) (20)

Dernier

Dernier (20)

Training program effectiveness a measuring instrument (1)