Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"

Examining the Impact of Training on Business Results

Through Post-Training ROI

Dissertation

Submitted to Northcentral University

Graduate Faculty of the Department of Business and Management
in Partial Fulfillment of the
Requirements for the Degree of

DOCTOR OF PHILOSOPHY

by

JACK L. KULES

Prescott Valley, Arizona
May 2008

APPROVAL


by

Jack L. Kules

Approved by:

__________________________________________ ________________
Chair: Thomas Driver, Ph.D.

__________________________________________ ________________
Member: David Moody, Ph.D.

__________________________________________ ________________
Member: William Shriner, Ph.D.

Certified by:

__________________________________________ ________________
School Chair: Freda Turner, Ph.D.

ABSTRACT



by

Jack L. Kules

Northcentral University, May 2008

Training expense represents a substantial investment in training resource.

This dissertation details research on the business impact of a leadership-training

program using the return on investment (ROI) methodology. The primary

objective of this study was the determination if there were positive financial

impacts of a leadership-training program on a business and to present a

verifiable and valid, substantial ROI with meaningfulness. Training application

and effectiveness were measured through four research tests, and ROI results

and relevance were measured using two additional research tests. Questionnaire

responses and action plan information was examined from 48 employees (from a

target population of about 65) who went through a 15-hour strategic leadership

training program. The training was found to be both effective and successful in

teaching and developing strategic leadership concepts. Perhaps the most

important and tangible indication of successes came directly from the company.

Based on the study’s results, there was no meaningful difference of learning by

the four organizational groups and that all participants have a net positive impact

on business.

iii

ACKNOWLEDGEMENTS

When you begin a learning journey such as the one represented by this

dissertation, as the researcher, you think you know where you are headed but

you cannot be sure of the final destination. The experience of completing this

dissertation has been challenging and rewarding. Not only did it foster a sense

of accomplishment and contribution to my field of study but it also allowed me

to meet and become close to some brilliant people.

I thank the participants in this research who gave their generous input

and support to this project. Likewise, I thank ―SP‖ for allowing me to use many

of their resources to make this dream possible.

A special thank you is extended to my dissertation committee. Dr.

Thomas Driver (Committee Chair), Dr. David Moody, and Dr. William Shriner

supported me all the way through the dissertation process and their direction,

suggestions, and concerns during this project made this journey an exceptional

one.

Finally, I want to thank my wife, Bridget, for all the love, patience,

understanding, and support that she gave me over the past four years—without

which I would never have realized this life-long dream becoming a reality. I am

truly and deeply indebted to her.

iv

TABLE OF CONTENTS

APPROVAL ........................................................................................................ ii

ABSTRACT ....................................................................................................... iii

ACKNOWLEDGEMENTS ................................................................................. iiv

TABLE OF CONTENTS ..................................................................................... v

LIST OF TABLES ............................................................................................. vii

LIST OF FIGURES ............................................................................................ ix

CHAPTER I: INTRODUCTION .......................................................................... 1
Statement of the Problem............................................................................. 2
Definition of Key Terms ................................................................................ 4
Brief Review of Related Literature ................................................................ 5
Highlights and Limitations of Methodology ................................................... 6
Limitations of the Study ................................................................................ 7
Research Expectations ................................................................................ 8

CHAPTER II: REVIEW OF RELATED LITERATURE ...................................... 10
How Much Is Performance Improvement Really Worth? ............................ 10
Using ROI Forecasting to Develop a High-Impact, High-Value Training
Curriculum ............................................................................................ 12
Measuring Return on Investment for a Mandatory Training Program ......... 14
Resisting Measurement: Evaluating Soft Skills Training for Senior Police
Officers ................................................................................................. 16
A Preprogram ROI for Machine Operator Training ..................................... 17
Getting Results With Interpersonal Skills Training ..................................... 19
Training’s Contribution to a Major Change Initiative ................................... 21
ROI Case Studies ...................................................................................... 23

CHAPTER III: METHODOLOGY ..................................................................... 26
Overview .................................................................................................... 26
Restatement of the Problem....................................................................... 27
Statement of Hypotheses ........................................................................... 27
Description of Research Design ................................................................. 29
Operational Definition of Constructs and Key Variables ............................. 34

v

Description of Materials and Instruments ................................................... 35
Selection of Subjects .................................................................................. 37
Procedures ................................................................................................. 38
Discussion of Data Processing................................................................... 44
Methodological Assumptions and Limitations............................................. 46
Ethical Assurances ..................................................................................... 47

CHAPTER IV: FINDINGS ................................................................................ 50
Overview .................................................................................................... 50
Findings...................................................................................................... 51
Analysis and Evaluation of Findings ........................................................... 71
Summary .................................................................................................... 76

CHAPTER V: SUMMARY, CONCLUSIONS AND RECOMMENDATIONS ..... 77
Summary .................................................................................................... 77
Conclusions................................................................................................ 84
Recommendations ..................................................................................... 93

REFERENCES ................................................................................................ 96

APPENDICES ................................................................................................101
Appendix A ............................................................................................... 102
Appendix B ............................................................................................... 108
Appendix C............................................................................................... 110
Appendix D............................................................................................... 118
Appendix E ............................................................................................... 122
Appendix F ............................................................................................... 124
Appendix G .............................................................................................. 131
Appendix H............................................................................................... 136
Appendix I ................................................................................................. 140

vi

LIST OF TABLES

Table 1: ROI Case Studies .............................................................................. 23

Table 2: Demographics of the Population ........................................................ 50

Table 3: Mean and Standard Deviation for Questionnaire Responses
Average Across Four Classes .................................................................... 51

Table 4: Mean and Standard Deviation for the 15 Objectives in Question 1
Across Four Classes .................................................................................. 52

Table 5: Mean and Standard Deviation for the Six Elements in Question 2

Table 6: Mean and Standard Deviation for the Five Skill Areas in Question
3 Across Four Classes ............................................................................... 57

Table 7: Mean and Standard Deviation for the 14 Topics in Question 4

Table 8: Metrics for Action Plans ..................................................................... 63

Table 9: Action Plan Topic Selection ............................................................... 63

Table 10: Action Plan Input from Selection ...................................................... 65

Table 11: Individual Costs for Strategic Leadership Program .......................... 68

Table 12: Total Program Costs, By Class ........................................................ 69

Table 13: One-Way ANOVA of Perceived Relevance of the 15 Objectives
by Organizational Group ............................................................................ 71

Table 14: One-Way ANOVA of Perceived Relevance of the Six Elements
by Organizational Group ............................................................................ 72

Table 15: One-Way ANOVA of Perceived Relevance of the Five Skill
Areas by Organizational Group .................................................................. 73

Table 16: One-Way ANOVA of Perceived Relevance of the 13 Topics by
Organizational Group ................................................................................. 74

Table 17: Mean Time to Action Plan Completion Based on Class................... 74

vii

Table 18: One-Way ANOVA of Mean Time to Action Plan Completion
Across the Four Classes ............................................................................ 75

Table F1: Question Data Table From Questionnaires, Question 1 ................ 125




Table G1: ROI Data Table from Action Plans ................................................ 132

Table H1: Values of One-Way ANOVA of Question 1 ................................... 137




Table I1: Mean ROI Accros Four Classes ..................................................... 141

Table I2: Values of One-Way ANOVA of ROI Results ................................... 142

viii

LIST OF FIGURES

Figure 1. Question 1 Breakdown of Questionnaire Responses for Class 1 ..... 54




Figure 5. Question 2 Breakdown of Questionnaire Responses for Classes 1
and 2 .......................................................................................................... 57

and 4 .......................................................................................................... 57

and 2 .......................................................................................................... 59

and 4 .......................................................................................................... 59


Figure 10. Question 4 Breakdown of Questionnaire Responses for Class 2 ... 61



ix

1

Chapter I: Introduction

Training expenses make up a substantial portion of the budget of an

organization, and have come to be seen as an investment in training resources

(Phillips, 2001). Large training expenditures and the need to show value are two

of the primary drivers that have set in motion an increased emphasis on return on

investment (ROI). Attention to ROI is rapidly becoming a central concern of

organizations. Executives are showing an increased interest in ROI, and have

become mindful of how training budgets have grown with limited or no

accountability (Bartram, 1999; Rothwell, 2003). Executives are now demanding a

return on investment for these programs. Further illustrating its prominence,

numerous case studies (Phillips, 1994; Phillips, 1997; Phillips, 2001) have used

return on investment to validate training’s contribution to business results.

Training budgets can be very large and now have the full attention of

executives. The costs can be immense. IBM has a training budget of about $1

billion, and Kinko’s training budget is over $30 million—or 6% of Kinko’s total

payroll (Phillips, 2001). Regardless of the measurement methodology—total

budget, expenditure per employee, percentage of payroll, percentage of

revenue—a large training budget elicits additional evaluation and measurement.

Executives are now demanding increased accountability for the increasing

training expenditures. The use of the ROI methodology offers a view of training

that reflects the bottom line (Phillips, 2002).

The debate as to what should be measured and which results provide the

best evidence of training success will continue, and no measurement has been

2

clearly proven the most reliable. A valid system would employ a balanced set of

measures that take into consideration trainee preferences, learning retention,

learning application, changes in business measures, and actual ROI (Phillips,

1997). This need for balanced measures is the major driver of ROI methodology,

as it measures financial impact along with other important concerns.

Statement of the Problem

The problem addressed in this study was the determination if there were

positive financial impacts of a leadership-training program on a business and to

present a verifiable and valid ROI with meaningfulness. SP had a major need to

determine the value and applicability of its leadership training to the company for

current and future leaders. Therefore, a study of this nature was proposed and

was based on the hypotheses and research questions addressing the differences

in various learning relationships across four organizational groups in a targeted

company. There were many studies (Phillips, 1994; Phillips, 1997; Phillips, 2001)

that indicated a positive ROI, but the lack of a statistically sound approach in

noted studies (see Review of Related Literature for examples) hinders verifying

the reliability of the data in the studies.

Using the ROI methodology to examine the value of training, when

developed, analyzed and reported with meaningful data, will support an

organization’s business success (the ROI methodology will be covered in

Chapter 3). It can verify or validate that the training initiatives are meeting the

needs of the business and having a positive impact on the bottom line of the

business. This methodology has already been used in numerous case studies

3

(Phillips, 1994; Phillips, 1997; Phillips, 2001) to validate the contribution of

training to business results.

Specialty Pharmaceutical (SP)—the fictitious name providing anonymity to

the organization where this study took place—was the organizational body used

for this research. SP opened three new facilities with due to its substantial growth

and trained a group of key current and potential supervisors and managers in

leadership skills. SP needed to change its predominant method of autocratic,

dictatorial style of management to a leadership style that lends itself to a higher

performance work force. Most of the supervisory staff had modeled their

leadership after the former style, as they had been exposed to it as a worker.

The leadership-training program was designed to promote the essential

skills of creativity development, motivation, delegation, communication, and

decision-making. Emphasis was placed on balancing the human-relations side of

management with the drive for results.

The time-spaced format allowed for the real-world application of concepts

between sessions as well as a forum to report on the results achieved. Each

session contained practical application projects that corresponded to the

participants’ responsibilities—so that the projects completed during the program

would result in improved performance

During the program, each participant completed a one-year management

plan (action plan). This project required follow-up actions to initiate actual cost

savings and improvements in the work setting. This positive form of

accountability was to ensure long-lasting and measurable results.

4

Five research questions were formulated to address the problem which

drove the hypotheses and resulting testing:

1. What differences, if any, existed in the perceived relevance of the 15

objectives across the four classes at SP?

2. What differences, if any, in the perceived relevance of the six elements

of the job existed across the four classes at SP?

3. What differences, if any, in the perceived relevance of the five skill

areas existed across the four classes at SP?

4. What differences, if any, in the perceived relevance of the 13 topics in

one’s own work or that of the work unit existed across the four classes at SP?

5. What are the differences, if any, in the ROI across the organizational

groups at SP?

Definition of Key Terms

Action Plan. A specific plan for the actions or steps that will be undertaken

to implement the ROI methodology within the organization. Action items focus on

specific spheres of influence. (Phillips, 2003)

Evaluation Framework. Defines the levels at which programs are

evaluated and how data were captured at different times from different sources.

The framework involves a four-level evaluation process: reaction, learning,

behavior, and results (Kirkpatrick, 1998).

Isolating Program Effects. Used to ensure accuracy in calculating the ROI

and to ensure an accurate picture of the program’s benefits. Excluding this step

in the process will result in an incorrect, invalid, and inappropriate ROI

5

calculation.

Return on Investment (ROI). This is the ratio of earnings (net benefits) to

investments (costs); it is the most common measure for value-added benefits in

operational functions.

Brief Review of Related Literature

A review of literature related to the topic of return on investment in the

training industry revealed that there has been a body of knowledge generating

the standard approach to determining ROI. Several popular texts (Phillips, 1983;

Phillips, 1994; Phillips, 1997; Phillips, 2001) focus on the methodologies

presented by the ROI Institute (Phillips, 2002). Phillips, along with his partners

Patti Phillips and Ron Stone, created the ROI Institute as a means of

communicating the philosophies and methodologies of ROI to training and

human resource professionals around the world.

Other authors have considered the main views of Phillips and have made

some changes to the methodologies that were established. One major dissident

is Dennis Kravetz, with his own approach to measuring human capital. Kravetz’

approach accommodates the financial aspects of the complete human capital

concept, whereas Phillips’ approach focuses more on the business results

produced by an intervention—such as training (Kravetz, 2004). Although Kravetz’

approach is different, the results are generally complimentary to those used by

Phillips.

The American Society for Training and Development (ASTD) and the

International Society for Performance Improvement (ISPI) have published several

6

books on the topic of ROI by different authors as well as three volumes of case

studies based on Jack Phillips’ ROI methodology (Phillips, 1994; Phillips, 1997;

Phillips, 2001). These case studies are the foundation of the methodology that

was used during the research. Because of the infancy of the methodology, there

is more potential for uncovering new insights and approaches to solving the

problem.

Highlights and Limitations of the Methodology

The research methodology used descriptive and inferential statistics to

characterize the data and to predict similarities. The primary analytical test used

was the one-way analysis of variance (ANOVA).

The data collection process involved using questionnaires and action

plans from the new or potential supervisors who participated in the training

program. It further included objectives and methodologies for each level of

evaluation targeted. These covered the following targets, by objective (Phillips,

1983; Kirkpatrick, 1998):

Reaction. Data collection included a participant feedback form at the end

of the training program to judge reactions to the training in regards to the

relevance and effectiveness.

Learning. Pre- and post-self-assessments, observed behaviors during skill

practice, and review activities were used to evaluate how much they learned.

Job Application. On-the-job behavior changes were monitored and

measured during the action plan implementation.

Business Impact. In the action plan, participants estimated the potential

7

cost benefits of their applied behaviors over a 90-day period after training.

Questionnaires (Appendix A) were administered during a 90-minute

follow-up session which was scheduled about three months after the initial

training and co-facilitated by senior management. The business impact was

evaluated by comparing the identified measures on action plans (Appendix B) at

implementation of the high-performing leadership action plans (at the end of the

training cycle) during those 90 days following implementation.

Although several strategies were available to isolate the effects of training,

most of the methods were thought to be ill-suited to this situation. Participants’

direct estimates were found to be the most appropriate technique. Participants’

estimates of the impact of training have shown to be a reliable indicator of results

through practical application of this process by numerous ROI professionals

(Phillips, 1994, 1997, 2001). It is a proven methodology for deriving reliable data

and establishing reliable metrics and the best fit for use in this ROI study. Seven

examples of its use are provided in the Review of Related Literature.

Limitations of the Study

In the data collection, the focus was on impact and not process.

Consequently, very little effort was made to collect input on the actual training

delivery processes and mechanisms, themselves. Most of the emphasis was on

the effect of the program in relation to the investment required. To remain

objective, data were collected only from people who experienced the training.

A standard practice in ROI evaluation of short-term training programs is to

capture the first-year benefits after the program has been conducted (Brinkerhoff,

8

1994; Graber, 1997). This practice, in essence, limits the analysis of benefits to

one year of operation. Although this could slightly overstate the results in some

cases, it represents a conservative. The benefits obtained in subsequent years

are not necessarily useful to the analysis. In this study, data were collected and

analyzed over a three-month period following the training. This data was then

extrapolated over a 12-month period to simulate standard practice.

It is recognized that not all data collected and analyzed is absolute and

that there may be qualifiers that need to be researched at a later date. There

may be variables that are qualitative in nature that are treated quantitatively for

the purpose of measuring results.

Research Expectations

The business impact of the training program was examined in this

quantitative study. Every attempt was made to uncover specific business results

linked to the training program. The impact of the training program was measured

by the extent of application of the skills and knowledge promoted in the program.

The program’s impact was indicated by the extent to which participants saw a

connection between the training program and the application of skills in the work

setting. It was further shown through their reactions’ consistency with their

responses on the action plans.

Intangible results are those benefits that cannot be assigned a dollar value

or those for which the assigned value is questionable. Even though these

benefits were not used in the ROI calculation, they are important to the goals of

SP. Comments from the learners’ action plan feedback indicated various

9

intangibles that would benefit SP. There will be a follow-up study conducted by

SP to evaluate the intangible benefits.

10

Chapter II: Review of Related Literature

After reviewing more than 150 journal articles and case studies, seven

case studies were selected as having the most relevance to the research topic.

Each of these articles deals with return on investment (ROI) in training programs

and is often cited by experts in the field. The following are short summaries of

each article and the different statistical approaches that each took.

How Much Is Performance Improvement Really Worth?

Berthiez (2001) conducted an ROI study on a sales training program for a

major global automobile corporation in Europe. The primary project objective

focused on the following questions: exactly what financial effect did this specific

training have on the overall bottom-line in sales of new cars? What percentage of

new sales, if any, could the training process claim to represent? The results were

substantial and unquestionably beneficial to executives in determining how to

allocate shrinking budgets to gain maximum return on human performance for

dollars invested (Berthiez, 2001).

The Phillip’s ROI methodology was used in this impact study. The steps

used were data collection, training effects isolation, data conversion to monetary

value, intangible benefits identification, program costs tabulation, and ROI

calculation (ROI methodology will be covered in Chapter 3). In addition to the

overall ROI model, it was found to be useful to add an additional component at

the beginning of the process model—the training needs analysis (TNA). The

focus on a TNA helped to identify clearly what needs to be accomplished with a

given training initiative (Berthiez, 2001).

11

To isolate the relationship between training and performance

improvement, the following three approaches were used: training impact—sales

consultants’ perception of the influence of sales training on actual car sales;

confidence factor—sales consultants’ certainty of their estimates about the

influence of training and other factors; and customer validation—final sales data

collected by the customer and used to substantiate sale consultant estimates.

These approaches were selected for ease of use and the realistic credibility of

sources. Control groups, monitoring on-the-job application of principles learned in

training or trend-line analyses, could have been used to further isolate the data

on training effects. Berthiez suggests it would be beneficial to compare and

contrast other methods of isolating data in future ROI initiatives wherever

practical and cost efficient to do so (Berthiez, 2001).

The conversion of data was relatively easy, since units of cars sold can be

multiplied by a given unit price and unit margin to clearly establish the monetary

benefits. The data were calibrated and crosschecked against actual car sales

results reported in Standard-Poor’s annual report, objective industry statistics,

and internal company sales reports. The findings were discounted by a training

impact of 9% from data taken from the questionnaire responses. Training impact

represented sales consultants’ perceptions of the influence of the sales training

on actual car sales. The findings were discounted further by an average

confidence factor of 65%, representing sales consultants’ certainty of their

estimates regarding the influence of training and other factors. To maintain the

integrity of the statistical data, the study excluded any values that were outside of

12

realistic possibility (Berthiez, 2001).

Data in this study was used to compare against known results of the

business and the industry. The results for the retail distributor—with an ROI of

325%–was conclusive evidence that the investors, the manufacturer, and the

retail distributor did realize a significant payback for the capital invested

(Berthiez, 2001).

Using ROI Forecasting to Develop a High-Impact, High-Value Training
Curriculum

With a variety of approaches to forecasting addressed, Graber (1997)

described the process used by a Midwest electrical power provider to allocate

funds for a variety of training initiatives and projects. The process built on the

principles of forecasting financial benefits and provided an important tool for the

training and human resource managers (Graber, 1997).

The purpose of the ROI forecasting was to identify the training that would

provide the highest possible payback and, more generally, to make wise training

and development decisions. The training itself was seen to have no inherent

value; the worth was dependent on the performance gains it catalyzed, the

performance gaps it addressed, and the opportunities it created in a given

environment. ROI forecasting did not affect the cost of training; however, it

maximized the payback from limited training resources and helped to avoid

training dollars going to waste (Graber, 1997).

The ROI forecasting process began by selecting employees and

supervisors who were most familiar with a job—the subject-matter experts

(SME). The SMEs agreed on the key accountabilities of the job, which were

13

given a weight based on their importance and the typical time spent doing them

during a year. An estimation procedure (Casio-Ramos estimation) was used to

make the weights more accurate. Subject experts picked the highest weighted

accountability and gave it 100 points; every other accountability was then

compared to it and given a lesser number of points. Finally, the subject experts

identified from seven to ten critical skills for each key accountability (Graber,

1997).

Using a five-point rating scale (beginner, novice, skilled, advanced, and

expert), skill assessment questionnaires were completed separately by

employees and their managers, and both perspectives were weighed equally.

Employee skill gaps were identified and the cost of the gaps in terms of lost

performance was estimated. Rather than calculate the value of each employee,

the process was simplified by using the median of the employees’ pay range to

establish their value within each of three levels: professional, supervisory and

middle management (Graber, 1997).

To increase its value as a good measure of training need, the skill gap

was calculated differently than is typically done. A percentage skill gap value was

calculated using the traditional gap rating scale in conjunction with the

importance of the skill to the job. The dollar value of the job was then used to

calculate the dollar impact of the skill gap (Graber, 1997).

For example, if salary and benefits equaled $100,000, formal

presentations (the skill) make up a 4.8% weight, and the subject rated a 3 on

formal presentations (which equates to 50%), the calculation would be: (a)

14

$100,000 x 4.8% = $4,800 (the skill value of a fully qualified employee), and (b)

$4,800 x 50% rating = $2,400 (the size of the gap from the optimum) (Graber,

1997).

Based on the skill gaps identified by this process for all applicable

employees, 11 training programs were chosen and the expected ROI for each

was calculated. These results show that only 6 of the 11 courses selected

showed a positive ROI; therefore, only about $11,800 was spent on this program

(Graber, 1997).

Measuring Return on Investment for a Mandatory Training Program

Marcial (2001) illustrated how a Florida-based government agency

measured the ROI for a mandatory training program on self-mastery. The

program evaluated the impact of using a specific training delivery methodology

and its ability to channel employees to participate in and contribute to the

organization (Marcial, 2001).

It involved a learning map on a high-performance development model

(HPDM), due to the perceived importance of self-mastery, and this particular

learning map had undergone several beta tests and revisions before it was used

by the agency. The learning map included workplace change sheets, how we

learn sheets, teammate skills sheets, development approach sheets, information

guides, and personal opportunity plans. Each person was asked to complete the

questionnaire alone, return to debrief the questionnaire as a group, identify one

thing he or she could contribute or do that could make a difference to the facility,

and tell two co-workers what he or she has learned about HPDM (Marcial, 2001).

15

The data collection methodology was set up to take advantage of all the

data generated in the sessions. A comparison arrangement was established to

isolate the effects of the learning map. To isolate the effects of the learning map

further, the participants were asked to estimate the impact of the program

themselves (Marcial, 2001).

The researchers used the participants’ application of what they learned in

the learning map sessions to convert the data to a monetary value. Monetary

values were assigned to the changes made by the participants using regulations

and methods found within the agency. A database of employee time was readily

available from human resources. An internal specialist provided the

compensation data, including the cost of medical care for injury using billing

codes and allowances paid to providers for treatment (Marcial, 2001).

The benefits-to-cost ratio came out as 1.03, calculated by dividing the total

benefits ($2,819.37) by the total costs ($2,737.10). An ROI of 3% was found by

subtracting the total costs ($2,737.10) from the total benefits ($2,819.37) divided

by the total costs ($2,737.10) (Marcial, 2001).

At the project outset, an ROI of 25% was anticipated. Because this was a

mandated program, the whole cost was an expected expense with no financial

benefits; therefore the attempt to calculate the cost of mandatory training was

valuable. The use of the learning maps for HPDM did not appear to be an

economical delivery method at first, but it became evident that its use was worth

the time invested and that it had the potential to bring about a significant ROI

(Marcial, 2001).

16

Resisting Measurement: Evaluating Soft Skills Training for Senior Police Officers

Police organizations are traditionally governed from the top down in a

military-like hierarchical structure. However, police work often requires the

exercise of independent judgment within limited contexts. McCarty’s (2001)

research illustrated the problems of implementing and evaluating a program

focused on interpersonal skills training in a highly structured, often resistant

organization in New York.

Two methods were used to collect data in this study, which was based on

a Dale Carnegie training program: action plans and questionnaires. Participants

used the action plans to track progress and to collect actual performance data

over a three-month period following the final training session. Participants

received the follow-up questionnaire three months after the final session so that

they could return it with the action plan. The questionnaire provided data

regarding the extent to which the participants had used the training on the job

and the results that came from these applications (McCarty, 2001).

Participants’ estimates of the impact of training were a reliable indicator

when appropriate steps were taken to collect data. Though their judgment was

subjective, the participants had direct experience to guide their estimates and

had first-hand knowledge of other influences that could have had an impact on

performance measures. Participants’ estimates had proven to be extremely

reliable in other studies where they were compared to results from control groups

(McCarty, 2001).

In McCarty’s study, the primary strategy for converting data to monetary

17

value was to ask the participants to make estimates and calculations based on

improvements in their work units. On the action plans submitted, participants

used accepted standards and conversion factors to arrive at the monetary value.

Some of the action plans were incomplete or otherwise flawed, invalidating the

data for purposes of calculating the ROI; even then, there were indications of

performance improvement (McCarty, 2001).

To calculate the ROI, the benefits from the group were compared with the

fully loaded cost of the program for the group as follows: The total benefits were

$333,168 and the cost of the program was $136,530, therefore the ROI

($333,168 - $136,530 / $136,530) was 144%. The high yield for a small number

of contributors was indicative of the type of results manifested when senior

officials who have a large sphere of influence participate in action plan

improvements (McCarty, 2001).

A Preprogram ROI for Machine Operator Training

This proposed program included significant capital expenditures and the

creation of a Canadian training facility (Renaud, 1997). Prior to pursuing the

project, an ROI was developed using a small-scale pilot effort. The ROI was

developed using methods typically reserved for post-program evaluation. The

results of the process can apply to almost any type of setting in which a major

training expenditure is under consideration (Renaud, 1997).

According to Renaud, one of the most difficult tasks in completing this ROI

evaluation was estimating the expected benefits from the program. The pilot

program presented some measurable improvements and this information was

18

used in five tangible benefit areas: training time, machining scrap, turnover,

safety, and maintenance expense (Renaud, 1997).

As a standard practice, supervisors recorded production shortfalls with

new employees until they reached the standard rate for a machine. These losses

were essentially production lost to trainees taking the time allowed to learn to

operate a machine at a standard rate. The pilot program showed a 64%

reduction in this production lost to training, and the supervisors estimated that

trainee losses could be reduced by 50% with a structured training program

(Renaud, 1997).

Many factors contribute to machining scrap; one of the biggest factors is

the lack of training of new and inexperienced operators. The supervisors

estimated that there could be at least a 10% reduction in total scrap costs with

the new training program. The turnover rate in the machining area was eight

employees per month; because of the smaller numbers of employees involved in

the pilot program, turnover reduction data were inconclusive. The supervisors felt

that training could reduce turnover by at least 30%; a 30% savings was

$115,200. This estimate was considered conservative (Renaud, 1997).

Most of the accidents in the machining area were not lost-time injuries.

The pilot program indicated a 25% reduction in accidents, but the supervisors

estimated that accidents could be reduced by 30%. To remain conservative, the

25% value was used, resulting in an annual savings of $14,250 (Renaud, 1997).

Effective training of new employees should result in less maintenance

required on production machines. The pilot program showed a dramatic

19

reduction of 45%; however, the supervisors estimated that the unscheduled

maintenance expenses could be reduced by 10% each year with the

implementation of the training program (Renaud, 1997).

The total projected annual savings were $304,950. The annualized costs

were $131,500. The annual gross savings of $304,950 less the program costs of

$131,950 result in a net savings of $173,450. The expected ROI ($173,450 /

$131,500) for the first years was 132%. The investment in the equipment and the

program development was to be spread over several years (Renaud, 1997).

Getting Results with Interpersonal Skills Training

Because of their soft nature, interpersonal skills training was a particular

challenge when calculating ROI. Russ-Eft (1994) described a very successful,

commercially available interpersonal skills training program being implemented in

a large information service organization in New York.

The organization was facing several challenges during the implementation

of the training program. They had more than 100 locations spread across the

United States. They wanted to bring about a cultural change to make the climate

more supportive and cooperative and to foster improved performance throughout

the organization. The ROI and evaluation study was designed to identify

solutions to these cost-related issues (Russ-Eft, 1994).

The organization decided to use a financial approach to justify the

implementation of the skills program. The parameters were defined and the

evaluation was conducted using surveys designed to evaluate the transfer of

skills acquired. The evaluation instruments gathered ratings of subjects’ on-the-

20

job behavior as well as ratings of organizational climate and job satisfaction. The

on-the-job behavior ratings included 44 items grouped into four categories:

dealing with problems, communicating with co-workers, working with supervisors,

and improving work. Members of the trained and control groups then indicated

the percentage of time that they spent on the job dealing with the items grouped

under the four behavioral categories. Ratings of organizational climate and job

satisfaction were obtained from seven additional items (Russ-Eft, 1994).

The Russ-Eft study used pre-training and post-training ratings of the

behavior of training and control groups, to which people had been randomly

assigned. Ratings were gathered from the members of both groups, their

supervisors, and their colleagues. Pre-training ratings were gathered immediately

before training; post-training ratings were gathered approximately three months

following training (Russ-Eft, 1994).

A series of analyses of variance with repeated measures were conducted.

These analyses compared trained participants with control participants, training-

group supervisors with control-group supervisors, and trained-group colleagues

with control-group colleagues (Russ-Eft, 1994).

The costs of training included trainees’ time away from work, trainers’ time

away from work for preparation and training, the costs of materials used during

the sessions, the time required for designing the sessions, and certification costs.

Costs incurred for the entire population of 85 trainees were estimated at

approximately $70,000 (Russ-Eft, 1994).

The analysis of variance showed overall improvement, retrospectively

21

comparing skill ratings before and after training (i.e., post-training ratings of pre-

training skill) after the training was completed. Significant differences appeared

between trained and control groups due to these overall improvements (Russ-Eft,

1994).

The results indicated that the total benefits were approximately $305,000

for a sample of 42 trainees out of the total population of 85 trainees; resulting in a

net benefit of $235,000 when the costs are subtracted from the benefits. The

traditional ROI formula yields an ROI of 336%. These calculations underestimate

the net benefits somewhat, as none of the indirect benefits was included in the

analysis (Russ-Eft, 1994).

Training’s Contribution to a Major Change Initiative

Stone (1997) illustrated how the ROI was calculated for an extensive

training program for relationship bankers headquartered in North Carolina. The

program evaluated the impact of training in the face of a variety of other change

initiatives, including a process improvement effort implemented prior to the

training. The study illustrated one approach to isolate the impact of the various

factors contributing to improvement (Stone, 1997).

Because the decision to determine the ROI for training was not made until

after the program had been implemented, the process of calculating the ROI was

much more difficult. Several factors can contribute to performance

improvements. In the Stone study, the possible factors were the re-engineering

effort that preceded the training, the support of the deal team, incentives,

coaching by managers, capital market liaison assistance, and other training

22

initiatives. Strategies were thus required to isolate the effects of the training. Four

strategies were considered: control group arrangement, trend-line analysis,

estimates taken directly from participants, and estimates taken directly from the

managers of the participants (Stone, 1997).

The data collection plan included several approaches to converting data to

a monetary value. The specific benefit from each of the six business performance

measures had to be converted to dollar values so they could be compared to the

training program costs. For the customer satisfaction measure, it was decided

that no value would be placed on the actual training; instead, customers would

be asked to indicate the specific benefit they received (Stone, 1997).

The participant and manager questionnaires provided significant

information on changes in behavior; the information obtained indicated the skills

were being used on the job. Based on the input from the team questionnaire, the

five business measures from the training program with the strongest influence

were increased sales of capital market products, improved customer satisfaction,

improved employee satisfaction, new business from existing clients, and new

relationships established. Customers were asked to provide specific information

regarding the impact of the training on the business; 67% of customers

responded that the bankers added value to their business (Stone, 1997).

The ROI for the training was developed only based on participants’ input.

The total fully-loaded cost to train the participants was $698,725. When this

amount is combined with the benefits in the standard ROI formula, the ROI

comes out to 47.2%. Although this value may be lower than anticipated, the

23

return is much higher in reality; the 47.2% ROI value is an understatement of the

actual return that does not consider several factors. When factored in, the actual

ROI could easily approach a value in the 100% to 200% range (Stone, 1997).

ROI Case Studies

Table 1 has a list of additional case studies that are pertinent to using ROI

in a training environment and that hold relevance to this study.

Table 1

ROI Case Studies

Organization Industry Program ROI (%)
Office of Personnel U.S. Government Supervisory 150
Management Training
Magnavox Electronic Electronics Literacy Training 741
Systems Company
Litton Guidance and Avionics Self-Directed Work 650
Control Systems Team Training
Coca-Cola Bottling Soft Drinks Supervisory 1,447
Company of San Training
Antonio
Texas Instruments Electronics Sales Negotiation 2,827
Training
Apple Computer Computer Process 182
Manufacturing Improvement
Training
Hewlett-Packard Computer Sales Training 195
Company Support Services
First National Bank Financial Sales Training 555
Services
Causeway Corporation Financial Total Quality 154
Services Management
Training

24

Table 1 (continued)

ROI Case Studies

Multi-Marques Inc. Bakery Supervisory Work 215
Process Analysis
Training
Midwest Banking Banking Loan Officer 1,988
Company Training
Financial Services Inc. Financial Human Resource 2,140
Services Selection Training
North County Electric Electric and Gas Applied Behavior 400
& Gas Utility Management
Training
Yellow Freight System Trucking Performance 1,115
Management
Training
Healthcare, Inc. Healthcare Sexual Harassment 1,052
Services Training
Apex Corporation Manufacturing Advanced Sales 2,981
and Distribution Skills Training
Eastman Chemical Chemical Empowerment 2,307
Company Training
Nortel Learning Telecomm Finance Training 317
Institute

NYNEX Corporation Communications Information 511
and Media Technology
Training
Texas Instruments Technology Negotiation Skills 2,827
Systems Group Training
First Union National Banking Change Initiative 472
Bank Training
Bell Atlantic Network Telecomm Computer-Based 319
Services Maintenance
Training

25

Table 1 (continued)

ROI Case Studies

Speedy Telecomm Performance 1,600
Telecommunications Management
Company System Training
Cracker Box, Inc. Restaurant Chain Performance 298
Management
Training
Focus Corporation Computer Build-to-Customer- 570
Manufacturing Order Training
Verizon Communication Telecomm Customer Service -54
Skills Training
Slick Manufacturing Government Computer Training 125
Agency (Ireland)
Healthcare, Inc. Healthcare Sexual Harassment 1,052
Services Training
Apex Corporation Manufacturing Advanced Sales 2,981
and Distribution Skills Training
Eastman Chemical Chemical Empowerment 2,307
Company Training
Compiled from In Action: Measuring Return on Investment, Volume 1, by J. J.
Phillips (1994), In Action: Measuring Return on Investment, Volume 2, by J. J.
Phillips (1998), and In Action: Measuring Return on Investment, Volume 3, by J.
J. Phillips (2001).

26

Chapter III: Methodology

Overview

The success of a training program at SP, a pharmaceutical company

located in the Midwest, was examined for this quantitative study. Two specific

objectives of this study were met through the implementation of a comprehensive

data collection and analysis process: (a) to examine the specific impact of the

training program in measurable business contributions to the extent possible, up

to and including the calculation of the ROI for SP; and (b) to examine the extent

to which participants applied on the job what they learned during the training.

At SP, the management had become more interested in measuring the

impact of training and development programs. Four major trends were driving

these actions:

1. Training programs were rapidly getting more expensive to develop and

deliver.

2. The importance of training in meeting strategic objectives within SP

placed the training process at a level where accountability was necessary.

3. A trend toward measurement and metrics at SP was recognized

because of regulatory compliance issues.

4. Executive management, in an attempt to manage resources efficiently

at SP, had brought closer scrutiny to the training and development process and

was requiring accountability for large training expenditures. Collectively, these

trends were driving a need for more accountability and evaluation in training and

employee development at SP.

27

Restatement of the Problem

The problem addressed in this study was the determination if there were

positive financial impacts of a leadership-training program on a business and to

present a verifiable and valid ROI with meaningfulness. SP had a major need to

determine the value and applicability of its leadership training to the company for

current and future leaders. Therefore, a study of this nature was proposed and

was based on the hypotheses and research questions addressing the differences

in various learning relationships across four organizational groups in a targeted

company. There were many studies (Phillips, 1994; Phillips, 1997; Phillips, 2001)

that indicated a positive ROI, but the lack of a statistically sound approach in

noted studies (see Review of Related Literature for examples) hinders verifying

the reliability of the data in the studies.

Statement of Hypotheses

There are five hypotheses defined in this section. All five hypotheses each

involve an analysis of variance (ANOVA) test. These research questions were

developed to test the hypotheses to which they apply, not to prove them

Research Question 1: What differences, if any, in the perceived relevance

of the 15 objectives existed across the four classes at SP?

H10: There will be no difference in relation to the perception of relevance

of the 15 objectives across the four organizational groups at SP (Operations,

Sales and Marketing, Scientific Research, and Support Groups).

H1a: There will be a difference in relation to the perception of relevance of

the 15 objectives across the four organizational groups at SP (Operations, Sales

28

and Marketing, Scientific Research, and Support Groups).


of the six elements of the job existed across the four classes at SP?


of the six elements of the job across the four organizational groups at SP

(Operations, Sales and Marketing, Scientific Research, and Support Groups).


the six elements of the job across the four organizational groups at SP



of the five skill areas existed across the four classes at SP?


of the five skill areas across the four organizational groups at SP (Operations,

Sales and Marketing, Scientific Research, and Support Groups).


the five skill areas across the four organizational groups at SP (Operations, Sales

and Marketing, Scientific Research, and Support Groups).


of the 13 topics in one’s own work or that of the work unit existed across the four

classes at SP?


of the 13 topics in one’s own work or that of the work unit across the four

organizational groups at SP (Operations, Sales and Marketing, Scientific

29

Research, and Support Groups).


the 13 topics in one’s own work or that of the work unit across the four

organizational groups at SP (Operations, Sales and Marketing, Scientific

Research, and Support Groups).

Research Question 5: What are the differences, if any, in the ROI across

the organizational groups at SP? In this context, a significant difference simply

means there is statistical evidence that there is a consistent difference; it does

not mean the difference is necessarily large or important (Sleezer, 1994).

H50: There will be no meaningful ROI across the organizational groups at

SP (Operations, Sales and Marketing, Scientific Research, and Support Groups).

H5a: There will be meaningful ROI across the organizational groups at SP


Description of Research Design

Phillips’ ROI Methodology: This ten-step model provides a process for

collecting data, summarizing and processing data, isolating the effects of

programs, converting the data to monetary value, and capturing the actual ROI.

In the first step, the planning is initiated and the specific business drivers

of the solution are identified. Discussion and decisions revolve around how the

solution will satisfy the business drivers. Business measures are clearly

identified. The objectives are established and revised to ensure that stakeholders

agree on the training to be applied, the behavior changes initiated and the

business impact measures to be influenced.

30

Detailed planning takes place in the next step. The purpose of the

evaluation is clearly defined and baseline data is developed and collected. If the

purpose is to calculate the ROI, the entire ROI process (ten steps) will be

followed. This step includes determining the data collection strategy and

developing the necessary planning documents that specify in detail how steps

three through ten will be carried out.

The third step involves the client organization collecting the Level 1 and

Level 2 evaluation data during the solution implementation (the evaluation levels

are explained in Chapter 5).

In step four, data on the application of training, behavior changes and

business impact are collected. The business impact data is converted to

monetary values to calculate the ROI. Throughout the process, data is collected

at all levels to show a chain of impact up to the highest level that satisfies the

purposes of the study. Step five begins the data analysis phase of the process as

the effects of the solution are isolated to determine the extent that the business

measures were influenced by the solution.

The sixth step is only applied when the purpose of the evaluation includes

calculating the ROI. If stakeholders have determined that there is no interest in

the ROI calculation for a specific initiative, then the business impact and behavior

change data is reported minus the calculation.

Step seven reports data on intangible benefits along with business metric

improvements. Barriers and enablers to implementation and behavior change are

also reported. Any improvement in behavior and business metrics influenced by

31

the solution is reported in step ten. Before the ROI is calculated (step nine), the

costs (step eight) are compared to the benefits that are converted to a monetary

value from step six. Additionally, all of the data from steps three, four, five, and

seven are reported, along with conclusions and recommendations. Conclusions

address information such as what caused the results, what worked, and what did

not work, while recommendations address where to go from there and how the

findings can be used to implement improvement.

In this study, two methods were used to collect data in keeping with the

Phillips’ ROI methodology. A follow-up questionnaire (see Appendix A) was used

to determine the extent to which participants utilized the training and achieved

self-reported on-the-job success. The action plan (see Appendix B) was

implemented during the training to identify areas for individual improvement as a

result of the training program, to link achievements to department-level

contributions, and to convert the contributions to monetary value. Participants

were required to continue using the process to track progress and collect actual

performance data for a three-month period after the last training session.

Participant estimates of training impact were a reliable indicator when

appropriate steps were taken to collect the data. The participants were the

closest individuals to the performance improvement and often were aware of the

other influences that affected the performance measures. For this study,

participants were asked to indicate the degree to which a specific improvement

resulted from the training program. The action plan was the tool used to capture

this data. The action plan was a tested and validated document approach

32

devised by the ROI Institute (Phillips, 1997).

While data can be converted to monetary values in many ways, the

primary strategy that was used in this study was to ask participants to make

estimates and calculations based on the improvement in their work units. The

participants used accepted standards and conversion factors to arrive at

monetary values.

The leadership training program was believed to show positive business

results using the return on investment methodology. A structured approach was

employed, allowing a comparison of the data gathered from questionnaires and

action plans. The structure used in this study, designated by Gay and Airasian

(2000) as a quasi-experimental design, is well suited to situations where it is not

feasible to randomly assign individual participants to groups. This situation arose

from the purposive sampling methods that were applied in this study.

The following criteria were used for this quasi-experimental study: (a) four

training courses took place—a total of 48 employees were trained, (b) two

experienced senior leaders of SP, who were certified through an extensive train-

the-trainer program, conducted the class instruction, (c) the class was designed

for a group of 10-16 students, and (d) all assignments among all classes had the

same objectives.

Sample Size Selection. A population of 200 was identified by SP

management from which to select the sample size. It was determined that only

about 65 current employees of the leadership population of approximately 200

would require the training—based on a screening of performance reviews and

33

succession planning data by SP, which were not a part of this study. Only 48

participants (24% of the total population, but 74% of the target population) would

be available for the training and subsequent study, therefore sample size

calculations were based on the following data. Using power analysis, the

acceptable margin of error was determined to be 5% with a confidence level of

95%. The sample size of 48 would yield a response distribution of less than 4%

based on an anticipated 100% response rate (which was achieved). With this in

mind, it would be likely to achieve meaningful input with a sample size of 48.

Therefore, with a sample size of 48 from the approximately 65 eligible and

required employees, a confidence level of 95% and a confidence interval of 7.32

(based on a sample size of 48 from a population of 65 and a worst-case

percentage of accuracy of 50%), about 73.8% of the target population made up

the sample of 48.

The size of the study sample was critical to producing meaningful results.

A power analysis could be used to determine the size of the sample is large

enough. However, with an unknown effect size or a useful standard deviation

(based on past data), a power analysis was not performed. All research

questions were tested to determine a meaningful impact in rejecting the null

hypothesis.

Operational Definition of Constructs and Key Variables

The independent and dependent variables for each research question are

presented in this chapter under the Statement of Hypotheses in the five research

questions and the accompanying null and alternate hypotheses. As well, the

34

dependent variables for each research question can be found in the referenced

Statement of Hypotheses section.

Research Question 1. There were four organizational groups selected

(Operations, Sales and Marketing, Scientific Research, and Support Groups) with

each group being an independent variable. These four organizational groups

were used consistently throughout the research as the independent variable for

all six of the hypotheses and the associated research questions. Independent

variable data was collected using both the questionnaires and the action plans.

The averaged class score for the research question (based on the

questionnaire responses) was the dependent variable for the ANOVA test in

hypothesis one. For hypothesis one and research question one, the

questionnaire was the source of all data.



each group being an independent variable.


questionnaire responses) was the dependent variable for the ANOVA tests in

hypothesis two. For hypothesis two and research question two, the questionnaire

was the source of all data.





35


hypothesis three. For hypothesis three and research question three, the







hypothesis four. For hypothesis four and research question four, the







hypothesis five. For hypothesis five and research question five, the action plans

were used as the data source.

Description of Materials and Instruments

Questionnaire. The participants received a questionnaire (see Appendix

A) that provided data regarding the extent to which participants used the training

on the job while involved in the program and the results that came from these

applications. The questionnaire had the participants: (a) rate the success of the

course in meeting 15 objectives, (b) rate the relevance of the program elements

36

to the job, (c) indicate the degree to which the use of the 15 skills are enhanced,

and (d) indicate the extent to which one thinks this course will influence the

measures in their own work or that of the work unit. The questionnaire included a

13-item checklist, and it requested examples and details.

The questionnaire approach and format has been validated through

successful and effective use in multiple ROI projects and case studies completed

by the ROI Institute, as well as ROI practitioners worldwide (Philips, 1994;

Phillips, 1997; Phillips, 2001). Used in conjunction with the action plan, the

questionnaire was an invaluable assessment tool.

Action Plan: The requirement for the action plan (see Appendix B) was

communicated prior to the program start date. On the first day of training, the

program instructors described the action planning process in a 15-minute

discussion. The participants received notepads on which to capture specific

action items throughout the training program. They were instructed to make

notes when they learned a technique or skill that would be useful in improving

one of the three measures that they each identified as important. In essence, this

notepad became a rough draft of the action plans.

For this mixed method study, the reasons behind various aspects of the

results were relied on. In this study the why and how of improved leadership

were investigated. The need was for smaller and more focused samples rather

than large random samples. The researcher relied on qualitative methods for

gathering information on: (a) participation in the training, (b) direct experiential

learning and (c) analysis of outcomes through documentation. The actions plans

37

provided qualitative data that was converted to quasi-quantitative data by the

participants.

The action planning process was discussed in detail in a one-hour session

during the second day of training. This discussion included three parts: (a) the

actual forms, (b) the guidelines for developing action plans and SMART (specific,

measurable, achievable, realistic, and time-based) requirements, and (c)

examples to illustrate what a complete action plan should look like.

The instructors distributed five blank action plans (only three are required,

one for each measure) and examples of completed action plans. During the

second day of training, the participants completed the booklets. The participants

worked in teams to complete all three action plans. Each plan took about 20-to-

30 minutes to complete.

During the third day of training, the participants briefly reviewed the action

planning process as a group, with each action plan taking about five minutes to

review. The program instructors then explained the follow-up steps to the group.

Selection of Subjects

Four functionally-defined groups (Operations, Sales and Marketing,

Scientific Research, and Support) were asked to submit the names of candidates

they considered key and potential leaders that would be contributing significantly

to the future of SP. They were considered the role models or leaders within their

respective departments, as determined by the executive leadership at SP. The

individuals that participated varied from new first-line supervisors to director-level

management.

38

From a population of about 65 leaders companywide who were in need of

the leadership training, 48 were selected by executive management to participate

in the training. For the population of 48 participants, the breakdown by function

was Operations, 15; Sales and Marketing, 12; Scientific Research, 10; and

Support Groups, 11. The sample size of 48, with a confidence level of 95% and a

confidence interval of 7.32, allowed for a meaningful number of participants,

adding to the validity of the study.

Procedures

The training program was conducted in several sessions, with each

session lasting two to four hours and delivered one or two days a week for three

to four weeks. An orientation session also was conducted prior to the first

session. The training program included 48 participants from various functional

organizations of SP. Employees attended the training sessions on SP’s time.

This sample represented about 24% (48 participants) of a total target population

(about 65 targeted leaders and potential leaders) at SP.

The research included four phases: process planning, data collection, data

analysis, and the communication of results. This was summed up in an ROI

impact study of the Strategic Leadership training program within SP.

1. Process planning was the most critical phase. Thorough planning

ensured that the process addressed the appropriate objectives and used the

proper data collection instruments.

2. Data collection occurred at two periods—during the selected training

program(s) to measure participant reaction, satisfaction, and learning; and on a

39

post-program basis to gather information on the application of skills and

knowledge as well as the impact the training program had on the organization.

3. The results of the program were shown by isolating the effects of the

training program. The costs were tabulated and the ROI calculation was

developed.

4. The communication of results included several issues that are often

neglected, but are as important as the process itself: (a) the process and

measurements were meaningless without communication; (b) communicating the

results was necessary to make improvements and to show accountability in the

training programs; (c) communication was a sensitive issue and could have been

a source of great benefit or a cause of major problems; and (d) the varieties of

target audience need different information.

Calculating the ROI required a value to be placed on each data element

connected with the training programs. The following are some strategies that

were used to convert data to monetary values.

1. Some output data converted to cost savings or profit contributions and

was able to be reported as a standard value, such as increased sales.

2. The cost of some quality measures were calculated and reported as a

standard value, such as customer complaints.

3. The historical costs of preventing a measure were used when

available, such as with time lost to accidents.

4. External databases contain an approximate value or cost of some data

elements, such as employee turnover.

40

5. Internal and external estimates of the value of a measure, such as

employee complaints.

6. Measures linked to other measures for which costs are developed,

such as employee satisfaction linked to turnover.

7. Supervisors’ or managers’ estimates of costs or values, such as

unscheduled absence.

8. Employee time saved converted to wages and benefits.

9. Participants’ estimates of the cost or value of the data element, such

as work group conflict.

10. Training staff estimates of the value of a data element, such as

harassment complaints.

Converting data to monetary benefits was critical. The process was challenging,

but was methodically accomplished using one or more of the above strategies.

Data Collection Issues. A data collection strategy was designed to meet

the objectives of this study. The questionnaires and the action plans were utilized

to ensure that adequate, quality input was obtained for the evaluation. In both

data collection methods, the focus was on impact and not process.

Consequently, very little effort was made to collect input on the actual training

delivery processes and mechanisms themselves, although some data were

collected. Most of the emphasis was on the impact of the program, which was

obtained with evaluation Levels 3 and 4 for data collection and analysis (Phillips,

1983; Kirkpatrick, 1998).

To remain objective, data were collected only from people who took part in

41

the training program. Although data from the instruction team and others could

have proven helpful, it was essential that input was free from any perceived bias.

These steps helped to ensure that the process was unbiased, objective, and

contained a minimum of errors.

Data Collection Timing. Another important issue to address in this study

was the timing of the data collection. Although the training program was designed

to have a long-term impact, the specific improvements from the training program

would be difficult to capture if assessed years after the training program was

completed. For longer periods, additional variables could influence output

measures, thus complicating the relationship between the training and the

improvement. The training program was time-spaced (one or two sessions a

week for four to six weeks), which provided opportunities for on the job

application of the training on an ongoing basis. Because of the above factors, it

was decided to measure the success of the training program during a three-

month period after the last training session. The data were collected and

analyzed over a three-month period following the training. This data was then

extrapolated over a 12-month period to simulate standard practice.

A standard practice in program evaluation is to capture the annual benefits

after the program has been conducted and compare them to the cost of the

program (Brinkerhoff, 1994; Graber, 1997). In essence, this limits the benefits in

an ROI calculation to the impact of a training program for one year of

improvements. While this could slightly overstate the results in some cases, it

usually understates them in practice. The skill transfer techniques used by the

42

instructors were among the most effective in the training industry at building the

confidence and skills that contribute to long-lasting results.

Questionnaire and Action Plan. The most common follow-up method—

questionnaires—provided a rich source of information on the extent to which

participants applied what they had learned in the program and the success they

achieved with the application. Because of the need for business impact data, the

action plan process provided a capable means of gauging the actual impact of

the training program as its information was applied. The action plan was also a

useful tool to keep employees focused on changing their behavior in the work

setting. Appendix A is an example of the questionnaire and Appendix B is an

example of an action plan.

The action plan’s necessity was communicated prior to the program start

date. On the first day of training, the program instructors described the action

planning process in a 15-minute discussion.

The participants received the questionnaire 90 days after the completion

of the leadership-training program. It provided data regarding the extent to which

participants used the training on the job while the training program was ongoing

and the results that came from these applications.

Effects of Training Isolation. Participant estimates of training impact are a

reliable indicator when appropriate steps are taken to collect the data (Phillips,

2002). The participants were the closest individuals to the performance

improvement and were often aware of the other influences that affect the

performance measures. For this study, participants were asked to indicate the

43

degree to which a specific improvement was caused by the training program. The

action plan was the tool used to capture this data.

Participants’ direct estimates were deemed the most appropriate

technique of evaluation for this study. Their estimates of the impact of training

are a reliable indicator of value (Phillips, 2002). The participants are the

individuals closest to the training and are often aware of the other influences that

have an impact on the leadership training measures. In studies where

participants’ estimates have been compared to the differences obtained from

control group experiments, their estimates were found to be very reliable

(Bernthal, 1994; McCarty, 2001; Russ-Eft, 1997; Westcott, 1994; Zigarmi, 1997).

Data to Monetary Value Conversion. While data could be converted to

monetary values in many ways, the primary strategy that was used in this study

was to ask participants to make estimates and calculations based on the

improvement in their individual work units. Participants used accepted standards

and conversion factors to arrive at monetary values.

Tabulating the costs of the training effort involved monitoring or

developing all of the costs related to the training program. A fully-loaded cost

profile was recommended when tabulating all of the direct and indirect costs

(Marelli, 1993). The return on investment was then calculated by comparing the

monetary benefits and costs. The benefit-cost ratio was obtained by dividing the

monetary benefits of the program by the costs. The return on investment used

the net benefits (costs minus benefits) divided by costs. This formula is

commonly used to evaluate other investments where the ROI is traditionally

44

reported as earnings divided by investment.

Intangible Benefits. Intangible results are those benefits that cannot be

assigned a dollar value or the assigned value is questionable. Even though these

benefits were not used in the ROI calculation, they are important to the goals of

SP. Comments from the participants and action plan reporting indicated various

intangibles that will benefit SP.

Discussion of Data Processing

The data collection strategy was designed to meet the objectives of this

study. The action plans and the questionnaire were utilized to ensure that

adequate, quality input was obtained for the evaluation.

To assure anonymity, the questionnaire was sent confidentially from

Corporate Training and Development to the 48 participants and the participants

were not required to write their names on it. Experience has shown that

participants will provide more data that are valid if anonymous feedback is

ensured. They were under no pressure to exaggerate the data to impress

superiors. The questionnaires and action plans were returned to the Corporate

Training and Development department for review and analysis (note: the

researcher is also the Director of Corporate Training and Development). Then a

report was developed (an internal privileged document) and presented to the

Executive Leadership Council for presentation to executive management at SP

(see Appendix C).

Responses to the questionnaire provided a very good source of data

because of the number of write-in comments and the quality of data supplied.

45

The return rate of completed questionnaires was 100%, and the return rate of

competed action plans was 100%. Descriptive and inferential statistics were used

to analyze the questionnaire and action plan data. The primary analytical test

used was the one-way analysis of variance (ANOVA). Tables and figures were

used to show the distribution of the participants’ selections in absolute numbers,

means and standard deviations.

Because the main thrust of this study was to determine the business

impact of the training program, every attempt was made to uncover specific

business results linked to the training program. The impact of the training

program was presented to indicate the extent of application of the skills and

knowledge obtained. Each participant was asked to select a number of skills they

used the most on the job since taking the training program.

The following tests were conducted in support of the five research

questions and five hypotheses.

Test 1. A one-way analysis of variance (ANOVA) was conducted to find if

there existed a difference in the perceived relevance of instruction across the

four organizational groups where the training took place. Each organizational

group was the independent variable with the averaged class score for question 1

on the questionnaire as the dependent variable.

Test 2. A one-way ANOVA was conducted to find if there existed a

difference in the perceived relevance of instruction across the elements of the job

for which the training took place. Each organizational group was the independent

variable with the averaged class score for question 2 on the questionnaire as the

46

dependent variable.


difference in the perceived degree of enhancement from the instruction across

the skills of the job for which the training took place. Each organizational group

was the independent variable with the averaged class score for question 3 on the

questionnaire as the dependent variable.


difference in the perceived influence that instruction had on the measure of

performance in one’s own work or that of the work unit. Each organizational

group was the independent variable with the averaged class score for question 4

on the questionnaire as the dependent variable.

Test 5. A one-way ANOVA was conducted to examine if there existed a

difference in the ROI across the organizational groups. Each organizational

group was the independent variable and the ROI expressed as a percent was the

dependent variable.

Methodological Assumptions and Limitations

Assumptions. The underlying assumptions for this study were: (a) two

fulltime instructors were sufficient to deliver all the training throughout the entire

initiative, and (b) standard SP leadership training curricula, used previously within

SP, were suitable for use.

Limitations. SP was in charge of assigning who participated in the

leadership-training program; therefore, the researcher had no control over who

took part in the training and only limited control over class size and scheduling.

47

Further, four organizational areas of one company were the focus of this study:

Operations, Sales and Marketing, Scientific Research, and Support Groups.

Therefore, the sample collected may not be representative of organizations in

other companies, limiting the ability to generalize.

Because the sample population obtained for this study was limited only to

key and potential leaders within SP, it is insufficient to use for generalizations

about the entire population of SP. The sample taken was representative only of

the population of key and potential leaders requiring leadership training at SP in

the four representative organizational groups, making it possible to generalize

about this population and these organizational groups alone.

Ethical Assurances

Of the many definitions of the term ethics, no one definition has emerged

as universally accepted. Any time ethics is the topic of discussion, terms such as

conscience, morality, legality, trust, values, responsibility, and integrity will

frequently be heard. Although these terms are closely associated with ethics,

they do not—by themselves—define it.

Assumptions about ethical underpinnings of human behavior are reflected

in the complexities involved in relating one culture to another, the distribution of

scarce resources, the allocation of power, the dynamics of groups, the

codification of ethical constructs, and the rewarding of ethical behavior and

discouraging of unethical behavior. As leadership becomes more complex and

deals with more situations, the application of ethics can also become more

complex (Bonhoeffer, 1995).

48

Two standards are applied in order to help protect the privacy of research

participants. Almost all research guarantees the participants’ confidentiality; the

stricter standard is anonymity, which essentially means that the participant will

remain anonymous throughout the study. Clearly, the anonymity standard is a

stronger guarantee of privacy, but it is often difficult to accomplish—especially

when participants have to be measured at multiple time points (Blackburn, 1996).

Increasingly researchers have had to deal with the ethical issue of a

person’s right to service. Good research practice often requires the use of a non-

treatment control group. When the program may have beneficial effects,

however, persons assigned to the non-treatment control may feel their rights to

equal access to services are being curtailed (Blackburn, 1996).

Even when clear ethical standards and principles exist, there will be times

when the need to do accurate research runs up against the rights of potential

participants. No set of standards can possibly anticipate every ethical

circumstance. Furthermore, there needs to be a procedure that assures that

researchers will consider all relevant ethical issues in formulating research plans.

The participants of this study were employees of SP and agreed to

participate. They were broken into four groups, by organization, each of which

received the learning materials and participated in the training. Each person

completed the informed consent form (see Appendix D). The study was non-

invasive, and the participants were asked only to complete the questionnaire and

the action plan as appropriate for the group to which they were assigned.

With no risks to any participant, no additional safeguards needed to be

49

established. All participants were identified by a randomly generated study

participant number that only the researcher had access to. All information

gathered was protected by confidentiality agreements and not shown to the

organization and other members of SP.

This researcher made every attempt to comply with the guidelines

established in Appendix B of the Dissertation Handbook (Northcentral, 2005).

Approval from the university’s Ethics Committee was granted via electronic

notification (see Appendix E).

Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"

Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (16)

Similaire à Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"

Similaire à Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI" (20)

Dernier

Dernier (20)

Doctoral Disseration: "Examining the Impact on Business Results Through Post-Training ROI"