Session presentation to the 4 June 2014 Annual State-wide Meeting of the Mississippi Chapter of the Association for Talent Development (ATD).

1. Annual State Meeting
Hattiesburg, MS
June 4, 2014
John Kmiec, Ph.D.
Certified ROI Professional
www.linkedin.com/in/jkmiec/
Consulting Employee
Training and Simulation Services and Solutions
Science Applications International Corporation
john.j.kmiec.jr@saic.com / 228.365.2559 / www.saic.com

2. Overview
• Background
• Opportunities
• Needs
• Program Objectives
• Intervention
• Data Collection Plan
• Data/ROI Analysis Plan
• Results
• Lessons
• Resources

3. Background
• Bulk manufacturer of lightweight food and medical grade plastics
• Complex, massive, hazardous operational environment
• Highly skilled, extensively trained, well paid, disciplined workforce
• Rigorous hygienic, chemical and environmental safety standards
• Very expensive raw materials frequently in short supply
• Product waste and rework impeding business performance
(Source: Phillips & Phillips, 2012)

4. Opportunities
• Product Waste and Rework: $272,850/month ($3.27M/year)
– Every 1% of total product produced as waste costs $245K
– Every 1% of total product produced as rework costs $35K
• Stressed employees under the gun to cut costs and improve quality
• Mediocre employee satisfaction and work engagement levels noted
• Most immediate managers lacked supervisory training/experience
(Source: Phillips & Phillips, 2012)

5. Needs
Program Needs
• Product Waste ($245K/1%) and Rework ($35K/1%) costs
$272,850/month ($3.27M/year)
• Stressed work environment barrier to Work Engagement
• Immediate managers unskilled and inexperienced
Evaluation Needs
• Effectiveness of learning intervention?
• Increase Work Engagement?
• Reduce Product Waste and Rework?
• Rollout Decision?
(Source: Phillips & Phillips, 2012)

6. Program Objectives
Reaction (1)
• Relevance, importance and intent to use on the job
Learning (2)
• Successful application of 5 self-coaching skills
• Can create and sustain motivational environment
On-the-Job Application (3)
• Effectively, continuously apply 5 self-coaching skills
• Create and sustain motivational work environment
Business Impact (4)
• Reduce % Product Waste and Rework
• Increase Work Engagement
ROI (5): 15% Target ROI
(Source: Phillips & Phillips, 2012)

7. Intervention
Purpose: To prepare immediate managers to more effectively
create and sustain motivational work environments that
positively impact work engagement and organizational
performance.
– 90-Day (7-Session) Learning Intervention based on the 5-
Skills of Savvy Managers
– Blended learning format (classroom, online, on-the-job skills
practice, journaling, and peer-interaction)
– Participants: Four frontline managers from Production unit
– Test Group (Production Unit) and Control Group
(Maintenance Unit) research design
(Source: Phillips & Phillips, 2012)

8. Intervention
Two In-person Sessions: Introduction/overview and finale:
– Course Introduction and Overview (1 day): Introduce the course,
overview key concepts, and familiarize participants with the
virtual learning environment
– Course Finale (0.5 day): Summarize the course, discuss
challenges and lessons learned, complete course assessments
and planned actions
Five Skills of
Savvy
Managers
Motivational
Work
Environment
Increased
Work
Engagement
Improved
Organizational
Performance
(Source: Phillips & Phillips, 2012)

9. Intervention
Five On-line Sessions (Blackboard): Summarize previous
session content, discuss any assignments, discuss current
topic, and preview next session:
– Self-managing (1.5 hours)
– Reflecting (1.5 hours)
– Acting consciously (1.5 hours)
– Collaborating (1.5 hours)
– Evolving (1.5 hours)

10. Intervention
Savvy Manager Learning Process
Set learning
and
performance
goals
Learn new
skills
(Plan Actions)
Practice new
skills on the job
(Implement)
Reflect on the
outcomes
(Journal)
Seek feedback
on the
experience
(Discuss)
Make
enhancements
(Adjust)
Monitor
progress
(Improve)

11. Intervention
Theory to Practice Assignments
• Use It or Lose It – Planned Action(s)
• Skill Development Journal
• Group Discussion and Feedback
• Learn, Practice, Adjust, Monitor Progress and Improve
Five Skills of
Savvy
Managers
Motivational
Work
Environment
Increased
Work
Engagement
Improved
Organizational
Performance
(Source: Phillips & Phillips, 2012)

12. Intervention
Use It or Lose It Planned Actions
Write down one idea you “got” as a takeaway from
today’s learning, reading, experience. Then, write
down what you plan to do starting tomorrow to
integrate this idea, practice it, and make it a habit.
1. What I got:
2. How I will use it tomorrow:
3. Which savvy skills I am developing:

13. Intervention
Skill Development Journal
Savvy Skill Coaching Questions
Self Manage What do you need to uncover about this
performance issue? Where will you apply your
discipline/control?
Reflect What becomes clear and more apparent? What is
the new picture that emerges?
Act Consciously What specific actions will you take to help you
reach your goal?
Collaborate Who can help you? Where are your resources?
Evolve What did you learn that will help you to grow? How
will you implement new strategies for going
forward?
JobAid

14. Intervention
Group Discussion and Feedback
What other things might we
do to help build our savvy
manager skills on the job?
Set learning and
performance
goals
Learn new skills
(Plan Actions)
Practice new
skills on the job
(Implement)
Reflect on the
outcomes
(Journal)
Seek feedback
on the
experience
(Discuss)
Make
enhancements
(Adjust)
Monitor
progress
(Improve)

15. Intervention
Practice, Discuss, Adjust, Monitor and Improve
Impact Measures (Level-4):
• Work Engagement: Test/Control Group Arrangement
– 32 line employees from Production unit (test)
– 31 line employees from Maintenance unit (control)
– Utrecht Work Engagement Scale (Schaufeli & Salanova, 2008)
• Production business unit controllable Waste and Rework
– Product Waste & Rework: $272,850/month ($3.27M/year)
– Every 1% of total product produced as Waste costs $245K
– Every 1% of total product produced as Rework costs $35K
(Source: Phillips & Phillips, 2012)

16. Data Collection Plan
Reaction (1)
• Reaction questionnaires taken at the end of each of seven sessions
Learning (2)
• Pre- and post program skill self-assessment profiles
• Facilitator assessment of participant discussions, responses to
questions, and completed assignments during the program
• Utrecht Work Engagement Scale at program start, middle and end
On-the-Job Application (3)
• Skill self-assessment profiles (3-months)
• Utrecht Work Engagement Scale (3-months)
Business Impact (4)
• Percent Product Waste and Rework
• Utrecht Work Engagement Scale (6-months)
(Source: Phillips & Phillips, 2012)

17. Data/ROI Analysis Plan
Data Items Isolation of Effects
Monetary
Conversion
Percent Waste Trend analysis
Participant and
management
estimates
Percentage of waste
times $245,000
Percent Rework Trend analysis
Participant and
management
estimates
Percentage of rework
times $35,000
Work Engagement Control Group Not converted
(Source: Phillips & Phillips, 2012)

18. Results
Reaction (1)
• Content relevance (4.16/5.0, 4.0)
• Importance (4.07/5.0, 4.0)
• Intent to use (4.23/5.0)
Learning (2)
• Pre program skill self-assessment profile mean (105.0/125)
• Post program skill self-assessment profile mean (107.6/125)
• Utrecht Work Engagement Scale: Test group higher than control
group by the end of the 90-day learning intervention
On-the-Job Application (3)
• Skill self-assessment profile mean (112.6/125) at 3-months
• Utrecht Work Engagement Scale: Test group higher than control
(Source: Phillips & Phillips, 2012)

19. Results
Business Impact (4)
$367,500 Projected Trend - $119,350 Post Average = $248,150 per month
$248,150 X 12 = $2,977,800 per year less cost for Waste and Rework
0.50 Impact Estimate X 0.85 Confidence X $2,977,800 = $1,265,565
ROI (5)
Net Program Benefit/Program Costs X 100
($1,265,565 - $253,761)/$253,761 = 3.99 X 100 = 399%
Intangibles
• Increased Work Engagement
• Employee Satisfaction
• Improved Teamwork and Communications
• Better Decision-making
(Source: Phillips & Phillips, 2012)

20. Lessons
• Plan must be based on performance needs – drives purpose
• Close coordination with client from start to finish critical
• Do your homework, establish relationships, connect the dots
• “Stuff” happens – be prepared to adjust the plan
• Communicate, communicate, communicate!
– Needs
– Issues
– Progress
– Results

21. Resources
Measuring ROI in Learning & Development: Case Studies
from Global Organizations. Editors: Patricia Pulliam
Phillips, Ph.D. and Jack J. Phillips, Ph.D. ASTD Press,
Alexandria, VA , 2012.
The Savvy Manager: 5 skills that Drive Optimal
Performance. Jane R. Flagello, Ph.D. and Sandra B. Dugas,
Ph.D. ASTD Press, Alexandria, VA, 2009.
Enhancing Work Engagement Through the Management of
Human Resources. W.B. Schaufeli and M. Salanova. In The
Individual in the Changing Working Life (pp. 380–404).
Editors: K. Näswall, M. Sverke, and J. Hellgren. Cambrigde
University Press, New York, 2008.

22. Resources
For a free copy of this slide deck, please contact John
Kmiec at 228.365.2559 or john.j.kmiec.jr@saic.com

23. John Kmiec, Ph.D.
Certified ROI Professional
www.linkedin.com/in/jkmiec/
Consulting Employee
Training and Simulation Services and Solutions
Science Applications International Corporation
john.j.kmiec.jr@saic.com / 228.365.2559 / www.saic.com
Annual State Meeting
Hattiesburg, MS
June 4, 2014