1. ROI of Human Capital Interventions A Practice Toolkit

3. Step 1 Presenting Problem to Real Problem

4. Step 1. Presenting Problem to Real Problem The first task of the practitioner is almost always analysis Starts with a “presenting” problem – not a “hunch” but a problem … as it presents itself

5. Step 1. Presenting Problem to Real Problem Example… Let's take an example – higher staff turnover for Graduate Engineers (let say compared to an industry standard) The first question should not be, what skills and knowledge are required. It should be which metric(s) are affected negatively.

6. Step 1. Presenting Problem to Real Problem Example…

7. Step 1. Presenting Problem to Real Problem 1.1 Calculating benchmark employee cost

8. Step 1. Presenting Problem to Real Problem 1.2 Cost of 'covering' a vacant position (Calculated costs of other employees 'filling in' while the position is vacant)

9. Step 1. Presenting Problem to Real Problem 1.3 Cost to fill a vacant position

10. Step 1. Presenting Problem to Real Problem 1.4 On boarding & Orientation cost

11. Step 1. Presenting Problem to Real Problem 1.5 Cost of productivity ramp-up (During the first 3 months, an average new employee performs at 50% productivity of a tenured top performing employ

12. Step 1. Presenting Problem to Real Problem 1.6 Unit cost for filling a vacancy R 74964 Two important learning points: Know your metrics Negotiate, and even request help from your client to determine the right metrics. They are closer to the business and at the same time you obtain by in for the intervention.

13. Step 1. Presenting Problem to Real Problem 1.7 Total cost of replacing graduate engineers R 74964 x 6 = R 449784 Let's assume that we had to replace five graduate engineers the past six months

14. Step 1. Presenting Problem to Real Problem Now we can move on to analysis We know now which metrics are affected Consultant speciality

15. Step 1. Presenting Problem to Real Problem The basic tasks in analysis

16. Step 1. Presenting Problem to Real Problem Two types of data Quantitative ”The truth is in a number” – however does not say much about context Data ” More about context, how people experience in their own words”. Qualitative

17. Data gathering Before intervention Before intervention Step 1. Presenting Problem to Real Problem Questionnaires Surveys Tests Interviews Focus groups Observation Performance records Knowledge and skills testing Program follow up Project assignments After intervention After intervention

19. Disciplinary records;

20. Safety records;

21. ScorecardsAfter intervention

22. Step 1. Presenting Problem to Real Problem Contextual factors to consider when selecting data collection methods Before intervention After intervention Time for participants to respond; Costs; Amount of disruption of normal activities; What is the level of certainty that you will achieve with the result; Practicality; Culture/Philosophy – example the 10% syndrome.

23. Step 1. Presenting Problem to Real Problem Requirements for effective data collection instruments and techniques Before intervention Construct validity ( methods, techniques and test instruments should be evaluated in terms of its theoretical grounding); Predictive validity (methods, techniques and test instruments should be evaluated regularly in terms of its ability to predict what it purport to predict); Face validity – for line management very important! After intervention

24. Step 1. Presenting Problem to Real Problem Requirements for effective data collection instruments and techniques Before intervention After intervention Reliability: Produce consistent results over time, that differentiates effectively, and are independent of the assessor/data collector, or context.

25. Step 1. Presenting Problem to Real Problem Data consolidation

26. Step 1. Presenting Problem to Real Problem Data analysis

27. Note the following ! Because this toolkit does not aim to transfer knowledge about analysis as such the previous slides provides a brief and high level overview, for the sake of a clearity… for the sake of the toolkit we assume that the analysis provided clarity about the problem….

28. Step 1. Presenting Problem to Real Problem Brief Exposition of Problem Our analysis shows that the HR and line managers involved in selection decisions do interviews in a haphazard and nonscientific fashion Solution Following discussions with consultants, HR experts and line all agreed we need to implement competency based interviewing Lets assume this based on the core example

29. Step 1. Presenting Problem to Real Problem The training department is instructed to develop an appropriate program and put all line and support staff through training – they however indicate that a change in results is required quickly and that ROI for the training need to be justified soon after implementation… Lets assume this based on the core example

30. Presenting Problem to Real Problem So we have addressed this even before we asked any questions about training – and now we can move on to the next step Starts with a “presenting” problem – not a “hunch” but a problem … as it presents itself

31. Step 2a Planning the Intervention (Starting With the End In Mind)

32. Step 2a: Planning the Intervention (With ROI in Mind) When we plan an intervention – considering that we want to measure ROI- we really reverse engineer (or start with the end in mind) – We use the Kirkpatrick Model to achieve this

33. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures for clarification, based on our core example/case study (re graduate engineers)

34. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures a merely examples Remember this is a snapshot – there is a whole learning map and course development process underlying these objectives

35. Step 2a: Planning the Intervention (With ROI in Mind) Back to our case study to show how we plan “backwards” – remember these objectives and measures a merely examples

36. Step 2a: Planning the Intervention (With ROI in Mind) What have we achieved by planning the intervention this way? ROI ? Results A path – trail linking the business need with the intervention Results Results Results

37. Step 2a: Planning the Intervention (With ROI in Mind) We can also go some steps further and plan where we will get the information to evaluate the intervention and how…

38. Step 2b Planning the Intervention (Planning Beyond Training – With ROI In Mind)

39. Step 2b: Planning the Intervention (With ROI in Mind) Usually once we have done the planning and developed the training intervention we are ready to implement…

40. We need to determine what technique(s) we will use to show that the intervention actually had an impact – but our dilemma is this… Step 2b: Planning the Intervention (With ROI in Mind) External Factors Management Attention OR SOMETHING ELSE? Incentives AND YES THERE IS IMPROVEMENT AFTER PROGRAM Systems/Procedures Changes Here is our training program IS IT THE TRAINING PROGRAM?

41. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! Isolating the results/effect of our intervention is the most difficult and also most important element of building the case for value (ROI); Because of this, value at the higher levels is seldom determined; However, the credibility not only of your study to show value, but also the training function is based on this element.

42. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! The complexity of isolating the effect, is a function of the variables that are in play at any moment in time in an organization; The analyst should however always try to prove a causal relationship to an acceptable degree of accuracy; Acknowledging the forces that influence individual, group and business performance is the first step in this process.

43. Step 2b: Planning the Intervention (With ROI in Mind) We have to isolate the effects of our intervention! Remember that we are trying to isolate the effect of out intervention to an acceptable degree of accuracy – this makes very important to involve and agree with the client: What metrics should change as result of the intervention (see step 1) What isolation technique will be used (because they differ in terms of accuracy) Let’s consider the isolation techniques…

44. Step 2b: Planning the Intervention (With ROI in Mind) Let's look at the techniques! Trend line analysis of performance data; Control groups; Participant’s estimation of impact; Supervisor’s estimation of impact; Management’s estimation of impact; Use of previous studies Subordinate’s report of other factors; Estimating the impact of other factors; Use of customer input. With these techniques we try to isolate the effects of our intervention

45. Step 2b: Planning the Intervention (With ROI in Mind) Let's look at the techniques! Trend line analysis of performance data; Control groups; Participant’s estimation of impact; Supervisor’s estimation of impact; Management’s estimation of impact; Use of previous studies Subordinate’s report of other factors; Estimating the impact of other factors; Use of customer input. With these techniques we try to isolate the effects of other interventions and then declare the rest as due to our effort

46. Step 2b: Planning the Intervention (With ROI in Mind) Trend Line Analysis Draw a trend-line using previous performance data as a base; When intervention is conducted, actual performance is plotted and compared to the trend line; Improvement in actual performance over what the trend line predicted can then be reasonably attributed to the intervention.

47. Advantages Disadvantages Show quick results Easy to calculate Easy to understand Inexpensive Not difficult to organize (basically just monitor changes in metrics) A good first order analysis Uses only one factor to isolate the effect of the intervention (time- before and after implementation) Thus less accurate and client could be sceptical to accept only an analysis of the trend line. Trend Line Analysis- Advantages and Disadvantages Step 2b: Planning the Intervention (With ROI in Mind)

48. Example- Not from our central case study… Step 2b: Planning the Intervention (With ROI in Mind) Effect of the intervention Average Pre Program 55 COMPLAINTS Average Post Program 35 Sexual Harassment Prevention Program O N D J F M A M J J A S O N D J F M A M J J A S O TIME

49. Step 2b: Planning the Intervention (With ROI in Mind) Control Groups Technique Similar to classical research the control group method, involves an experimental group and a control group; The so called experimental group will be the target of our intervention; The control group would be a group that is paired with the experimental group, but would not undergo the intervention; Important things to remember with this method are: Paring need to be thought about carefully so as not to dilute the impact of the intervention; Although not as rigid as classical research key factors that could dilute the impact or assessment of the impact of the intervention need to be controlled.

50. Advantages Disadvantages More accurate than most techniques If the pairing and control of variables are agreed with the client the outcome has high level of credibility With this control groups, given the level of accuracy it is possible to benchmark the ROI of certain interventions given a predetermined set of variables More complicated Require more planning, negotiation, control and monitoring As result of above many practitioners are unwilling to use this technique Requires absolute agreement with the client environment that variables will be kept constant for the duration of the ROI evaluation. Control Groups- Advantages and Disadvantages Step 2b: Planning the Intervention (With ROI in Mind)

51. Step 2b: Planning the Intervention (With ROI in Mind) Schematic Explanation of control groups design 1. Pairing 2. Control Variables Control Group M2 M1 Experimental Group Intervention M1 M2 *M = measurement

52. Looking at pairing and control of variables Step 2b: Planning the Intervention (With ROI in Mind) PAIRING Note: Too small then meaningful pairing is not possible – too big then you are not going to be able to control the variables during and after the implementation of your intervention; Remember we go for reasonable accuracy therefore you pair the major elements and also control the major variables!! The above makes it very important to agree upfront with your client environment what how pairing will be done and which variables will be controlled. Select your experimental and control group carefully VARIABLES

53. Step 2b: Planning the Intervention (With ROI in Mind) So what do you typically pair? Size Nature of work Leadership stype Performance level General demography Pair for (Primarily) Negotiate creafully

54. Step 2b: Planning the Intervention (With ROI in Mind) So what do you typically try to control? Size Nature of work Leadership stype Performance level General demography structure Function Recognition and reward Turn over Systems and procedures Changes in: Changes in: + Negotiate creafully Negotiate creafully

55. Step 2b: Planning the Intervention (With ROI in Mind) Remember your decision about pairing and control of variables…. = (f) (your knowledge and judgement) + agreement with the client environment + practicability

56. Step 2b: Planning the Intervention (With ROI in Mind) Estimation of impact by third parties ( this technique can include an estimation of the perceived impact of the intervention, by participants, participants supervisors, participants managers) This technique provides participants with a questionnaire, and based on the improvement ask them to respond to the following: What percent of this improvement can be attributed to the application of skills/techniques/knowledge gained in the training program? What is the basis for this estimation? What confidence do you have in this estimate, expressed as a percentage? What other factors contributed to this improvement in performance? What other individuals or groups could estimate this percentage or determine the amount?

57. Step 2b: Planning the Intervention (With ROI in Mind) Estimation of impact by third parties – how do you increase the confidence level for this technique? Individuals who do not respond to the questionnaire or provide non- usable data on the questionnaire are assumed to report no improvement/impact; Extreme data and unrealistic claims are omitted from the analysis; Only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation; The confidence level, expressed as a percent, is multiplied by the improvement value to reduce the amount of the improvement by the potential error; If value is expressed it is also factored by the confidence value.

58. Advantages Disadvantages Easy to implement; Based on the assumption that participants are capable of determining or estimating impact; Although an estimate, this value will usually have considerable credibility with management because participants are at the centre of the change or improvement; High face validity. Less accurate than control groups Based on perceptions – could be regarded as Return on Expectation rather than ROI Could become victim of “group think” Estimation Of Impact By Third Parties - Advantages And Disadvantages Step 2b: Planning the Intervention (With ROI in Mind)

59. Example from an actual coaching intervention- Not from our central case study… Step 2b: Planning the Intervention (With ROI in Mind) Participants and management agreed which the respective groups will rate All indicators were pre agreed with the client

60. So now we can complete our plan Step 2b: Planning the Intervention (With ROI in Mind)

61. Note – because the techniques related to isolating the impact of other factors are not used often it is only referred to in this toolkit… Step 2b: Planning the Intervention (With ROI in Mind) Trend line analysis of performance data; Control groups; Participant’s estimation of impact; Supervisor’s estimation of impact; Management’s estimation of impact; Use of previous studies Subordinate’s report of other factors; Estimating the impact of other factors; Use of customer input. With these techniques we try to isolate the effects of other interventions and then declare the rest as due to our effort

62. Step 3 Implementation and Data Gathering (With ROI in Mind)

63. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: We opted for a control group evaluation method

64. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: Six months after the intervention the need for replacement of graduates were down by 50% in the experimental group

65. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: In the control group the need for replacement of graduate engineers also down by 10%...possibly because of anticipation of changes in HR approach

66. Implementation and Data Gathering (With ROI in Mind) Referring back to our original example (turnover in graduate engineers) – let us assume the following: In the control group the need for replacement of graduate engineers also down by 10%...possibly because of anticipation of changes in HR approach

67. So because to the method we used we can assign the change in te metric to our intervention, mainly because we controlled major variables. We do this as follows…

68. Step 4 Converting the Change As Result of the Intervention to Value and Calculating ROI

69. Step 4.a Converting the Change As Result of the Intervention to Value Referring back to our original example (turnover in graduate engineers)…this is how it is dome…

70. Because this is such a vast area to know here is some additional input re calculating the value of change in a metric.. Step 4.a Converting the Change As Result of the Intervention to Value Output Increases Time Savings Primary Measurements of Improvement Cost Savings Quality Improvement

71. Step 4.a Converting the Change As Result of the Intervention to Value Always try to link to a metric – alternatively use the method of Estimation Of Impact By Third Parties – See slide 55 and 56 (this method is also referred to as Return on Expectation)

72. Step 4.a Converting the Change As Result of the Intervention to Value See whether one can link tangible values to these?

73. Step 4.a Converting the Change As Result of the Intervention to Value And these….

74. Or determining value on a case by case basis using historical data Actual cost from records Legal fee; Settlement cost Cost R 285,000 Total of 35 complaints Estimated additional from staff Staff time; Management time Step 4.a Converting the Change As Result of the Intervention to Value R 285,000/35 = Cost per complaint Cost of Sexual Harassment Complaint

75. Step 4.a Converting the Change As Result of the Intervention to Value In our case example we did determine the value of the change and can therefore now move on to calculate cost benefit and therefore also ROI

76. Step 4.b Calculating ROI First thing to do now is to determine the cost of the intervention – again back to our main example …

77. Step 4.b Calculating ROI First thing to do now is to determine the cost of the intervention – again back to our main example … Net Program Benefits (R224892- R150,000) Program Costs (R150,000) Benefit/Cost Ratio = Ratio = 0.499928

78. Step 4.b Calculating ROI Net Program Benefits (R224892- R150,000) Program Costs (R150,000) First thing to do now is to determine the cost of the intervention – again back to our main example … X 100 1 Benefits ROI = Benefit ROI = 50% thus for every R1 I spent on this program you got your money back through saving as well as an additional 50 cents.