9. Cognitive learning
How much did trainees learn from the training?
Post-training learning
How much learning does the trainee demonstrate immediately
after training?
Retention
How much learning does the trainee demonstrate back on the
job?
The second evaluation level is used to determine learning
results. Did students actually learn the knowledge, skills and
attitudes the program was supposed to teach? It asks the
questions:
What knowledge was acquired?
What skills were developed or enhanced?
What attitudes were changed?
The results are usually determined by pre-and post-test scores
and on-the-job assessments or reports from supervisors. The
second evaluation level is not as widely used as the first level,
but it is still very common.
“Why Measure Training Effectiveness?” (2008) Retrieved
09/03/08 from
http://www.businessperform.com/html/evaluating_training_effe
ctiven.html
Clark, D. R. (2008), “Instructional System Design;” Retrieved
09/03/08 from
http://www.nwlink.com/~donclark/hrd/sat6.html
12. Results
What tangible outcomes or results occurred as a result of
training?
Kirkpatrick’s level four evaluates the final results of the
training. It asks the question – What effect has the training
achieved? Effects can include such things as morale, teamwork,
and most certainly, the monetary effect on the organization’s
bottom line. Management wants to know if they received value
for the training dollars spent and what their return on
investment was.
Collecting and analyzing evaluation at this level can be difficult
and time-consuming. Part of the difficulty comes from the
challenge of isolating the training variable from other factors in
the organization that may also affect learners’ behaviors. When
employee behavior changes, it is difficult to know if the change
is the result of training or the result of some other
environmental factor.
Level four evaluations are done through financial reports,
quality inspections and interviews with management personnel.
Source: Clark, D. R. (2008).
Kruse, K. Evaluating e-Learning: Introduction to the
Kirkpatrick Model. Retrieved 09/02/08 from: http://www.e-
learningguru.com/articles/art_2_8.htm.
____. (2008). Why Measure Training Effectiveness? Retrieved
09/03/08 from
http://www.businessperform.com/html/evaluating_training_effe
ctiven.html.
26. or service or otherwise on a password-protected website for
classroom use.
Increased competition for investment dollars requires
organizations to decide whether to invest in training or to invest
in something else. A well-designed benefit-cost ratio analysis
can aid in the decision-making process by allowing several
different investment options to be compared with each other.
The problem is that some benefits derived from training can be
intangible and difficult to quantify. How do you measure and
put a dollar value on increased morale or better teamwork?
Consequently, gathering and compiling the information needed
for an accurate benefit-cost analysis can be a complicated task.
Source: U.S. Dept. of Labor Retrieved 09/02/08, from
http://www.careertools.org/pdf/AdvancedROI.pdf.
Making HRD programs investments leading to measurable
payoffs in the future
Two practical methods
Evaluation of training costs (return on investment (ROI))
Utility analysis
38
Cost-Benefit Analysis
Companies may desire to quantify whether the benefits of
training outweigh the costs
Cost-benefit analysis
31. ΔU = (N)(T)(dt)(SDv) – C
Where
N = number of trainees
T = Time benefit expected to last
dt = Effect size
True size of difference in Std Dev terms
SDy = Dollar value of job performance in Std Dev terms
C = costs of training
Compute minimum annual benefits needed to break even
Use break even analysis to determine minimum effect size (dy)
that will yield required minimum benefit
Use the results from meta-analytic analysis to determine
expected cost and expected payoff
NOTE: Use the company statisticians and financial staff to help
(and they take partial ownership of results)
43
How Technology Impacts Evaluation
Reaction
Easy to gather continuous feedback online
Could use a discussion thread or “chat room” to allow trainees
to discuss their experiences with online learning
Learning
Very easy to test trainees electronically
Can also link to a learning management system
Behavior
Very hard to capture electronically
Some relevant data may be available in other information
systems, e.g., appraisals, promotions, turnover, and discipline
data
Results—Even harder to do online than traditionally, without
36. Interval
Ordinal
Ratio
3.5 points
QUESTION 2
1. The “standard normal distribution” table (i.e., the z-table)
shows the relationship between two variables: the z-score and a
probability (i.e., an area under the normal distribution).
True
False
3.5 points
QUESTION 3
1. Given a normally distributed variable (x), the formula: z =
(x - µ) ÷ σ can be used to determine the value of x at a specific
point on the x-axis, if you know the z-score.
True
False
3.5 points
QUESTION 4
1. A probability distribution is a theoretical but a frequency
distribution is based on actual results/data.
True
False
3.5 points
QUESTION 5
1. Which of the following symbols represents the variance of
the population?
37. σ2
σ
μ
s2
3.5 points
QUESTION 6
1. When calculating a binomial probability, the answer must be
in the range of 0.00 to 1.00.
True
False
3.5 points
QUESTION 7
1. Which of the following is NOT an assumption for (property
of) the binomial distribution?
The trials are independent
Two outcomes are possible on each trial in the experiment
The experiment consists of n identical trials
The probabilities of success and failure at each trial must be
equal
3.5 points
38. QUESTION 8
1. A negative value of Z indicates:
The number of standard deviations a measurement is to the right
of the mean
The number of standard deviations a measurement is to the left
of the mean
That a mistake has been made in computations, since Z cannot
be negative
The area corresponding to the z-score is negative
3.5 points
QUESTION 9
1. A variable with equal distances between points on a scale,
which also has a true meaningful zero is:
Ordinal
Cardinal
Ratio
Nominal
3.5 points
39. QUESTION 10
1. A probability distribution showing the probability of x
successes in n trials, where the probability of success does not
change from trial to trial, is termed a:
uniform probability distribution
binomial probability distribution
t distribution
normal probability distribution
3.5 points
QUESTION 11
1. For which level(s) of measurement are arithmetic operations
inappropriate?
The ratio scale
The interval scale
Both the ratio and interval scales
The nominal scale
40. 3.5 points
QUESTION 12
1. In a binomial experiment:
The probability does not change from trial to trial
The probability does change from trial to trial
The probability could change from trial to trial, depending on
the situation under consideration
None of the above
3.5 points
QUESTION 13
1. In constructing a frequency distribution, as the number of
classes are decreased, the class width:
Decreases
Remains unchanged
Increases
Can increase or decrease depending on the data values
41. 3.5 points
QUESTION 14
1. The measure of central tendency that is the point at which
half the scores fall above and half the scores fall below is the:
Mode
Range
Median
Mean
3.5 points
QUESTION 15
1. In a sample of 800 students in a university, 240, or 30%, are
Business majors. The 30% is an example of:
A sample
A population
Statistical inference
Descriptive statistics
3.5 points
QUESTION 16
42. 1. Given a normally distributed variable (x), if you know that
for a value of x = -27 the z-score = 0.67, what is the probability
of any x value being below -27?
0.9965
0.7486
-0.2514
0.2514
3.5 points
QUESTION 17
1. Data which uses numerical values, has an equal distance
between points on the scale, and does not have an absolute zero
is:
Nominal
Ordinal
Normal
Interval
3.5 points
QUESTION 18
1. The primary method for controlling/minimizing sampling
43. error is to:
Increase sampling bias
Take multiple samples
Select samples from a heterogeneous population
Increase sample size
3.5 points
QUESTION 19
1. A numerical value used as a summary measure for a sample,
such as sample mean, is known as a:
Population parameter
Sample parameter
Sample statistic
Population mean
3.5 points
QUESTION 20
1. The summaries of data, which may be tabular, graphical, or
numerical, are referred to as:
44. Inferential statistics
Descriptive statistics
Statistical inference
Report generation
3.5 points
QUESTION 21
1. The Quickie Sales Corporation has just been given a sales
estimate for the upcoming quarter. The estimate indicates that
sales (in millions of dollars) will be normally distributed with µ
= 325 and σ = 60. Assuming this estimate is accurate, what is
the probability that Quickie will have quarterly sales in excess
of $350 million?
QUESTION 22
The U.S. Postal Service handles seven basic types of letters and
cards: third class, second class, first class, air mail, special
delivery, registered and certified. The mail volume during 1977
is given in the table below. What was the average revenue per
ounce for these services during the year?
Type of Mailing
Ounces Delivered (in millions)
Price per Ounce
Third class
16,400
$0.05
45. Second class
24,100
0.08
First class
77,600
0.13
Air Mail
1,900
0.17
Special delivery
1,300
0.35
Registered
750
0.40
Certified
800
0.45
QUESTION 23
1. Here are the ages of 30 people who bought CD players at
Symphony Music Shop last week: 26, 37, 40, 18, 15, 45, 32,
68, 31, 37, 20, 32, 15, 27, 46, 44, 62, 58, 30, 42, 22, 26, 44, 41,
34, 55, 50, 63, 29, 22. Construct a frequency distribution
(table) with 6 classes to summarize these data.
QUESTION 1
1. Data that are categorical but can be rank-ordered in a
meaningful way are called:
Nominal
46. Interval
Ordinal
Ratio
3.5 points
QUESTION 2
1. The “standard normal distribution” table (i.e., the z-table)
shows the relationship between two variables: the z-score and a
probability (i.e., an area under the normal distribution).
True
False
3.5 points
QUESTION 3
1. Given a normally distributed variable (x), the formula: z =
(x - µ) ÷ σ can be used to determine the value of x at a specific
point on the x-axis, if you know the z-score.
True
False
3.5 points
QUESTION 4
1. A probability distribution is a theoretical but a frequency
distribution is based on actual results/data.
True
False
3.5 points
QUESTION 5
1. Which of the following symbols represents the variance of
the population?
47. σ2
σ
μ
s2
3.5 points
QUESTION 6
1. When calculating a binomial probability, the answer must be
in the range of 0.00 to 1.00.
True
False
3.5 points
QUESTION 7
1. Which of the following is NOT an assumption for (property
of) the binomial distribution?
The trials are independent
Two outcomes are possible on each trial in the experiment
The experiment consists of n identical trials
The probabilities of success and failure at each trial must be
equal
3.5 points
QUESTION 8
48. 1. A negative value of Z indicates:
The number of standard deviations a measurement is to the right
of the mean
The number of standard deviations a measurement is to the left
of the mean
That a mistake has been made in computations, since Z cannot
be negative
The area corresponding to the z-score is negative
3.5 points
QUESTION 9
1. A variable with equal distances between points on a scale,
which also has a true meaningful zero is:
Ordinal
Cardinal
Ratio
Nominal
3.5 points
QUESTION 10
49. 1. A probability distribution showing the probability of x
successes in n trials, where the probability of success does not
change from trial to trial, is termed a:
uniform probability distribution
binomial probability distribution
t distribution
normal probability distribution
3.5 points
QUESTION 11
1. For which level(s) of measurement are arithmetic operations
inappropriate?
The ratio scale
The interval scale
Both the ratio and interval scales
The nominal scale
3.5 points
50. QUESTION 12
1. In a binomial experiment:
The probability does not change from trial to trial
The probability does change from trial to trial
The probability could change from trial to trial, depending on
the situation under consideration
None of the above
3.5 points
QUESTION 13
1. In constructing a frequency distribution, as the number of
classes are decreased, the class width:
Decreases
Remains unchanged
Increases
Can increase or decrease depending on the data values
3.5 points
51. QUESTION 14
1. The measure of central tendency that is the point at which
half the scores fall above and half the scores fall below is the:
Mode
Range
Median
Mean
3.5 points
QUESTION 15
1. In a sample of 800 students in a university, 240, or 30%, are
Business majors. The 30% is an example of:
A sample
A population
Statistical inference
Descriptive statistics
3.5 points
QUESTION 16
1. Given a normally distributed variable (x), if you know that
52. for a value of x = -27 the z-score = 0.67, what is the probability
of any x value being below -27?
0.9965
0.7486
-0.2514
0.2514
3.5 points
QUESTION 17
1. Data which uses numerical values, has an equal distance
between points on the scale, and does not have an absolute zero
is:
Nominal
Ordinal
Normal
Interval
3.5 points
QUESTION 18
1. The primary method for controlling/minimizing sampling
error is to:
53. Increase sampling bias
Take multiple samples
Select samples from a heterogeneous population
Increase sample size
3.5 points
QUESTION 19
1. A numerical value used as a summary measure for a sample,
such as sample mean, is known as a:
Population parameter
Sample parameter
Sample statistic
Population mean
3.5 points
QUESTION 20
1. The summaries of data, which may be tabular, graphical, or
numerical, are referred to as:
54. Inferential statistics
Descriptive statistics
Statistical inference
Report generation
3.5 points
QUESTION 21
1. The Quickie Sales Corporation has just been given a sales
estimate for the upcoming quarter. The estimate indicates that
sales (in millions of dollars) will be normally distributed with µ
= 325 and σ = 60. Assuming this estimate is accurate, what is
the probability that Quickie will have quarterly sales in excess
of $350 million?
QUESTION 22
The U.S. Postal Service handles seven basic types of letters and
cards: third class, second class, first class, air mail, special
delivery, registered and certified. The mail volume during 1977
is given in the table below. What was the average revenue per
ounce for these services during the year?
Type of Mailing
Ounces Delivered (in millions)
Price per Ounce
Third class
16,400
$0.05
Second class
55. 24,100
0.08
First class
77,600
0.13
Air Mail
1,900
0.17
Special delivery
1,300
0.35
Registered
750
0.40
Certified
800
0.45
QUESTION 23
1. Here are the ages of 30 people who bought CD players at
Symphony Music Shop last week: 26, 37, 40, 18, 15, 45, 32,
68, 31, 37, 20, 32, 15, 27, 46, 44, 62, 58, 30, 42, 22, 26, 44, 41,
34, 55, 50, 63, 29, 22. Construct a frequency distribution
(table) with 6 classes to summarize these data.
QUESTION 1
1. Data that are categorical but can be rank-ordered in a
meaningful way are called:
Nominal
Interval
56. Ordinal
Ratio
3.5 points
QUESTION 2
1. The “standard normal distribution” table (i.e., the z-table)
shows the relationship between two variables: the z-score and a
probability (i.e., an area under the normal distribution).
True
False
3.5 points
QUESTION 3
1. Given a normally distributed variable (x), the formula: z =
(x - µ) ÷ σ can be used to determine the value of x at a specific
point on the x-axis, if you know the z-score.
True
False
3.5 points
QUESTION 4
1. A probability distribution is a theoretical but a frequency
distribution is based on actual results/data.
True
False
3.5 points
QUESTION 5
1. Which of the following symbols represents the variance of
the population?
σ2
57. σ
μ
s2
3.5 points
QUESTION 6
1. When calculating a binomial probability, the answer must be
in the range of 0.00 to 1.00.
True
False
3.5 points
QUESTION 7
1. Which of the following is NOT an assumption for (property
of) the binomial distribution?
The trials are independent
Two outcomes are possible on each trial in the experiment
The experiment consists of n identical trials
The probabilities of success and failure at each trial must be
equal
3.5 points
QUESTION 8
1. A negative value of Z indicates:
58. The number of standard deviations a measurement is to the right
of the mean
The number of standard deviations a measurement is to the left
of the mean
That a mistake has been made in computations, since Z cannot
be negative
The area corresponding to the z-score is negative
3.5 points
QUESTION 9
1. A variable with equal distances between points on a scale,
which also has a true meaningful zero is:
Ordinal
Cardinal
Ratio
Nominal
3.5 points
QUESTION 10
1. A probability distribution showing the probability of x
59. successes in n trials, where the probability of success does not
change from trial to trial, is termed a:
uniform probability distribution
binomial probability distribution
t distribution
normal probability distribution
3.5 points
QUESTION 11
1. For which level(s) of measurement are arithmetic operations
inappropriate?
The ratio scale
The interval scale
Both the ratio and interval scales
The nominal scale
3.5 points
QUESTION 12
60. 1. In a binomial experiment:
The probability does not change from trial to trial
The probability does change from trial to trial
The probability could change from trial to trial, depending on
the situation under consideration
None of the above
3.5 points
QUESTION 13
1. In constructing a frequency distribution, as the number of
classes are decreased, the class width:
Decreases
Remains unchanged
Increases
Can increase or decrease depending on the data values
3.5 points
QUESTION 14
61. 1. The measure of central tendency that is the point at which
half the scores fall above and half the scores fall below is the:
Mode
Range
Median
Mean
3.5 points
QUESTION 15
1. In a sample of 800 students in a university, 240, or 30%, are
Business majors. The 30% is an example of:
A sample
A population
Statistical inference
Descriptive statistics
3.5 points
QUESTION 16
1. Given a normally distributed variable (x), if you know that
for a value of x = -27 the z-score = 0.67, what is the probability
62. of any x value being below -27?
0.9965
0.7486
-0.2514
0.2514
3.5 points
QUESTION 17
1. Data which uses numerical values, has an equal distance
between points on the scale, and does not have an absolute zero
is:
Nominal
Ordinal
Normal
Interval
3.5 points
QUESTION 18
1. The primary method for controlling/minimizing sampling
error is to:
63. Increase sampling bias
Take multiple samples
Select samples from a heterogeneous population
Increase sample size
3.5 points
QUESTION 19
1. A numerical value used as a summary measure for a sample,
such as sample mean, is known as a:
Population parameter
Sample parameter
Sample statistic
Population mean
3.5 points
QUESTION 20
1. The summaries of data, which may be tabular, graphical, or
numerical, are referred to as:
64. Inferential statistics
Descriptive statistics
Statistical inference
Report generation
3.5 points
QUESTION 21
1. The Quickie Sales Corporation has just been given a sales
estimate for the upcoming quarter. The estimate indicates that
sales (in millions of dollars) will be normally distributed with µ
= 325 and σ = 60. Assuming this estimate is accurate, what is
the probability that Quickie will have quarterly sales in excess
of $350 million?
QUESTION 22
The U.S. Postal Service handles seven basic types of letters and
cards: third class, second class, first class, air mail, special
delivery, registered and certified. The mail volume during 1977
is given in the table below. What was the average revenue per
ounce for these services during the year?
Type of Mailing
Ounces Delivered (in millions)
Price per Ounce
Third class
16,400
$0.05
Second class
24,100
74. Audience: The group of learners the objective is written f nn or
(“the learner” or “the student”).
Behavior: The action or observable verb which describes what
the learner will be able to do after completing the instruction
(“Describe”, “Compare/contrast”, “Demonstrate”).
Condition: The tools and resources the students will be able to
use and where to find them (in a lab, in the actual work setting).
Degree: The standard or degree of accuracy to be considered
proficient.
13
Example Verbs
Verbs to Use:
Discuss
Explain
Demonstrate
Identify
List
Perform
Compare/Contrast
Score (on an assessment)
Verbs to Avoid
Understand
Comprehend
Know
Do
77. copied, scanned, or duplicated, in whole or in part, except for
use as permitted in a license distributed with a certain product
or service or otherwise on a password-protected website for
classroom use.
The firm does not have the expertise to design the program in-
house
Management would not likely have the time to design the
program
Firm doesn’t have an HRD department or full-time HRD
professional
Expertise: When an organization lacks specialized KSAOs
Timeliness: When it is timelier
Number of Trainees: For just a few trainees use an outside
training agency
Subject Matter: If the subject matter is sensitive or proprietary
the HRD conduct the program in-house
Cost: Always considers cost
Size of HRD: The size of the HRD department
SOURCE: From Carnevale, P., Gainer, L. J., Villet, J., &
Holland, S. L. (1990). Training Partnerships: Linking
Employers and Providers (p. 6). Alexandria, VA: American
Society for Training and Development .
17
How to Choose a Vendor?
80. copied, scanned, or duplicated, in whole or in part, except for
use as permitted in a license distributed with a certain product
or service or otherwise on a password-protected website for
classroom use.
Training Competency
Knowledge and varied skills needed to design and implement a
training program
Subject matter expertise
Mastery of the subject matter
Training is most effective when trainers possess an advanced
level of expertise as instructors and facilitators
Helping Less Qualified Trainers
Teaming skilled trainers with in-house subject matter experts to
form an instructional team
Using a training technique that does not require a human
trainer, such as computer-aided or online instruction programs
Train-the-trainer programs, which involve identifying in-house
content experts who lack training skills and training them to
become effective trainers
20
Preparing a Lesson Plan
Lesson plan:
Content to be covered
Sequencing of activities
Timing and planning of each activity
84. Too early
Tardy or tired participants
Over lunch hour
When do they eat
Mid-afternoon
Sluggish circadian rhythms
Late in day
Distractions and need to leave early
Considerations
Peak work hours
Staff meeting times
Travel requirements
Training shift workers
Consult with managers and supervisors before scheduling
Do it well in advance of scheduling
Avoids organizational complaints
Other problems
Family obligations
Moonlighters
Tired trainees
Reluctance to give up free time
Inducements
Overtime pay
Comp time
Promotion
Holding at resort with leisure activities
24
The Training Site
Comfortable and accessible
Quiet, private, and free from interruptions
93. development-by-us-companies
The Association for Talent Development (ATD) is a
professional membership organization supporting those who
develop the knowledge and skills of employees in organizations
around the world. The ATD Staff, along with a worldwide
network of volunteers work to empower professionals to
develop talent in the workplace.
37
DesignImplementationEvaluation
Define
objectives
Select
evaluation
criteria
Develop
lesson plan
Develop/acquire
materials
Select
trainer/leader
Select methods
and techniques
Schedule the
program/intervention
Determine
evaluation
design
Conduct
evaluation
of program
or intervention
Deliver the
HRD program
or intervention
Interpret
115. Citizenship
Behaviors
Behavior
Personal
Organizational
Outcomes
ProcessExample
Need is activated
Tension is created
Yes
No
Layoff announced;
Need for security
is activated
Tension expressed in
fear and worry
Improve performance?
Politics?
Job search?
Search for ways
to reduce tension
Performance improvement
leads management to
remove employee's
name from layoff list
Perform behaviors
to reduce tension
Fear and worry
significantly reduced
Tension eliminated
or significantly
reduced?
Need for security
satisfied
Need satisfied;
Need no longer