Workshop on different impact measurement tools

2. Theory of change
Impact Reporting and Investment Standards
Global Impact Investing Ratings System
Social Return on Investment
Demonstrating Value
Randomized Control Trials

3. Measuring Social Value
Dr. Tessa Hebb
Measuring Up, Social Finance Forum
Centre for Impact Investing, MaRS, Toronto
November 8th 2012

4. What is Blended Value
“The Blended Value Proposition states is that all
organizations, whether for-profit or not, create
value that consists of economic, social and
environmental value components—and that
investors (whether market-rate, charitable or
some mix of the two) simultaneously generate all
three forms of value through providing capital to
organizations.” Jed Emerson

5. Metrics and Measurement
Blended value requires measurement of all three
forms of value economic, social and
environmental.
“Measurement should be viewed as a process
whereby the greatest value is achieved through
organizations building up and learning from
data and evidence over time.” (Measuring the Value of
Corporate Philanthropy).”

6. What are social metrics?
Social metrics are measurement tools that can be
used to define and articulate social value, social
outcomes and the results generated by
investment and activities in the social sector.

7. You manage what you measure
“A review of measurement methodologies did not turn up
a “silver bullet” or single numeric against which
performance can be universally gauged. Rather, this
reading reinforced the notion that, to an
extent, measurement is its own reward. It encourages
improvement, management, and the explicit formulation
of assumptions and expectations.” (Measuring the Value
of Corporate Philanthropy)

8. Start with your mission
1. What are the results for which you will hold
yourselves accountable?
2. How will you achieve them?
3. What will they really cost?
4. How do you build the organization you need to
deliver these results?

9. What to Measure (Measuring the Value of Corporate
Philanthropy).
• Links among the mission, programs, and measures
must be clearly defined and articulated in order to
narrow the number of required indicators.
• The measures should be easily collectible and
communicable.
• The measures should be strategically designed and
applicable across the organization at all levels, while
also encouraging of operating units to focus on high-
level strategies.
• Above all, the measures must address progress
toward the mission and illustrate whether and how
the organization’s actions make a difference.

10. Inputs, Activities, Outcomes, and
Impacts
Inputs, activities, outputs, outcomes, and impacts over the
short, medium and long term are the building blocks of
social metrics.
The building blocks are linked to the organizations mission
through its theory of change.
“Activities,” such as the number of staff trained or amount
of goods purchased, and “outputs,” such as the number of
clients served, products distributed, and areas reached may
be the extent of measurement for short-term, one-off
projects. Sometimes simply identifying activities and
measuring output may be all that is feasible.

11. Theory of Change
• “Built around the pathway of
change, a Theory of Change
describes the types of
interventions (a single
program or a comprehensive
community initiative) that
bring about the outcomes
depicted in the pathway of a
change map. Each outcome in
the pathway of change is tied
to an intervention, revealing
the often complex web of
activity that is required to
bring about change.”
(TheoryofChange.org)

12. Logic Model
Input/Resources Activities Outputs Outcomes Impact Indicator
What are the resources What are the What is the evidence of What is the change that What will happen over How would you
that are needed to activities/products that service delivered to the will happen (short term the long term measure the outcome?
accomplish the will generate the intended audience at or medium term) to the How will you know if
activities? outputs? the intended dose? target group or change happened?
(what others will be individuals?
able to see, touch,
count)
Stakeholder 1

13. Conclusion
• Blended value requires economic, social and
environmental values be measured.
• Social metrics help organizations understand
their strengths.
• Social metrics link
mission, activities, outputs, outcomes and
impacts.
• No one size fits all.

14. Contact
Dr. Tessa Hebb
tessa_hebb@carleton.ca
Director,
Carleton Centre for Community Innovation
www.carleton.ca/3ci

15. Impact Map

16. Theory of change
Impact Reporting and Investment Standards
Global Impact Investing Ratings System
Social Return on Investment
Demonstrating Value
Randomized Control Trials

17. Inputs Activities Outputs Outcomes Impact
IRIS
GIIRS
SROI
Demonstrating Value
Randomized Control Trials

18. The Impact Reporting and Investment
Standards (IRIS) is a catalogue of
metrics that can be used to describe an
organization’s social, environmental,
and financial performance.
Pg 18

19. IRIS is…
My Metrics Your Metrics
Full Time Employees = Full Time Employees
GHG Emissions Reduced = GHG Emissions Reduced

20. IRIS Framework
20
http://iris.thegiin.org

21. IRIS Framework
21

22. Inputs Activities Outputs Outcomes Impact
IRIS
GIIRS
SROI
Demonstrating Value
Randomized Control Trials

23. Global Impact Investing Rating System
Environment
377 B Corporations I $1.82B
Revenues I 54 Industries I
$1M Annual Savings
Community
Workers
Governance
Pg 23

24. Pg 24

25. B Impact Assessment
25

26. Inputs Activities Outputs Outcomes Impact
IRIS
GIIRS
SROI
Demonstrating Value
Randomized Control Trials

27. What is SROI?
Social Return on Investment is a framework for
measuring and accounting for the value created
or destroyed by our activities – where the
concept of value is much broader than that which
can be captured by market prices.

28. val·ue/ˈvalyo͞o/noun
the regard that something is held to
deserve; the importance, worth, or
usefulness of something
the material or monetary worth of
something

29. Return on Investment
(Net Profit)
ROI =
Cost of Investment
29

30. ) 9 3+%
9 31"D) * :, ") % * ("D2 2 +3">3$("$9
% ./ +") .3(">) * ./ "<$(.2 2
%<$.3" W/ 3+"./ 3", 29
9 3(3+% +"(3T 9 3+, 3(1"$+, "% () +2 ") 9 3+, 3(1"2
3"2 ) 9 / % 9 1"
2 3"- 04"<() ' ($5 E "./ $."% () +2 ") 9 3+, 3(1"D) * :, "@
+"./ "$+, / % 9 3" 9 .) (3, "2 "./ 3"7XGY ) ("./ 3"- ) * ./ "0* 1.2 3"4) 5 5 2
$% +.) "9 % ..331E 3"
"./
2 3+.22 "@ 3"A 1.2 3"1>1.35 "@ 3"3+, ") 9
, 93, >"./ * % >"./ ">3$("SB 2""W/ :3" 1) %$:"=$:* 3"' 3+3($.3, "./ () * ' / "./ 3"$<<() $% "$+, ") +' ) 2 "
2 / +'
./ 3(3"5 $>"@ , 2 ) +$:E +T ) 2 "% 1.1"$11) %$.3, "D2 "./ 2
3"$, .2 ") ' +' ) 2 ./ 1" 1* % 311"($.3") 9 * ./ "0* 1.2 3"% 5 5 2
% ">) % ) ..331"' () D1"9 5 """""""""""
()
SROI example: Calgary Youth Justice
' () * <") 9 ) 9 3+, 3(1E 3"7XGY ) ("./ 3"- ) * ./ "0* 1.2 3"
"(3T 9 "./ "9 % ! "" B H"+"./ 3"9(1.">3$(".) "! "" KB S" +"$"./ (33">3$("@ 1B
6I I 2 2 6! ! ) $12 """
C :3" "
$@ H6
7 ) %$ :"Q$ :* 3 "4 $ :% :$. 2 +6"C/ (33 "- 3 $ (E K! "- ) * +' "G993 +, 3 (1E KK"(3T 993+, 3(1E
2 * ) "^ "! ) "SK"% () +2 ") 993 +, 3 (1
/ %
Y , 2 $. ) ("
+ % C) . $ :"Q$ :* 3 "
7 XGY +, 2 $ . ) ( 1"Y :* , 3 , 6
"Y % +% - X"H - X"S L ) . 31
4) , 3 - X"!
! c) :2 3"%
% $::1"Z 2 3, [
$=) , 47c VK V` J E KK
I V! OE O V` HI "<3("%
O` $::"9) ("! KK"%$::1
c) :2 3"2
% +=31.2 $.2 +1"
' )
H 08 VK V! O^ E 8
S! V` ! E
KOK V! O! H"<3("2
+=31.2 $.2 +
' )
Z 2 3, [
$=) ,
c) :2 3"$..3+, $+%
% 3"$."% * (."
)
S 0O VK VSHE
OH! V! KE OH
! VS! O"<3("$..3+, $+%3
Z 2 3, [
$=) ,
- ) * ./ "4) * (."c() %311"
J 0! V8OSE ^ I
^ V! S! E 8K
I VJ KE S^
^ V! E I "<3(">) * +' ") 993+, 3(", 2
H^ =3(.3, ".) "- 041
Z 2 3, ["
$=) ,
c() @ ) +"1* <3(=2 ("Z ."
$.2 1) +)
I 0H VK V! HSE K
8J VS8E !
SJ V! E
HKK"<3(">3$(B
"VHI B / ("9) ("c() @ ) +"G992 3(
8` $.2 %
(3F* 2 [(3,
4) 1.") 9"2 $(%
+% 3($.2 +" )
` 0^ VK VO! E K
HI VH8E !
HI VHI K , $>"e"S` I "l "VO! E K"Z ">) * ./ ") +:>[
HI !
Z 2 3, ["
$=) ,
; ++* $ :"7 ) %$ :"Q$ :* 3 "4 ( 3 $ . 3 , ". / ( ) * ' / "- 04 16
2 V8 OSE ^ I
^ V` J ! E KO
J V! O8 E 8 K 4$:% :$.3, ".) "(3:93% +9:$.2 +"m"SB B
I * ."2 ) HP "
c ( 3 13 +. "=$ :* 3 ") 9"1) %$ :"=$ :* 3 "% 3 $ . 3 , 6"Z =3 ( "S"
2 ( )
V! E J ! E H!
` J Y
+.3(31."($.3"SB P
I
>3 $ ( 1[6
C) . $ :"2+=3 1. 5 3 + . "2 "- 3 $ ( "! 6
+ V! ` HE
KKK
7 XGY 3 $ ( "! 6
"- V! 6"VIH" H
IB B
I I
ST>3 $ ( "7 XGY6 V! ! KB KB S
6"V!! S"!
47c"l "42 :3"1) * (% +"<() % "% ((3+.:>"9 5 "(3:2 :3"1) * (% """
.$@ 3"2 311E * () $@ 3B
Y .$& $" 5 5 * +2
." 31" 4) .>" 21"$..(2 * .3, ".) "./ 3"2 :=35 3+.") 9 $+>"% 5 5 * +2
@ +=) "5 ) .>"<$(.+3(1E
"
2+=31.) (1"$+, "' ) =3(+5 3+."$' 3+%31") 9 3(2 "<() ' ($5 1"$+, ""
2 9 +'
4) 5 5 * +2 ) :* +.33(1"$(3"./ 3"/ 3$(.") 9 2
.>"Q "./ 1"<() ' ($5 B 3+"
"W/ ) <<) (.* +2 31"9 (">) * ./ E <<) (.2 "<() ' ($5 ") <3($.2 +1E "
.2 ) "1* +' ) "$+,

31. 7 Principles of SROI
• Involve stakeholders
• Understand what changes
• Value the things that matter
• Only include what is material
• Do not overclaim
• Be transparent
• Verify the result

32. Inputs Activities Outputs Outcomes Impact
IRIS
GIIRS
SROI
Demonstrating Value
Randomized Control Trials

34. Capacity building
‘Snapshot’
Capacity
report
to use
informatio
n
Support for
Capacity
Monitoring to collect
Systems informati Clear
Developme on picture of Information
nt informatio Blueprint
n needs

35. Performance Snapshot example

36. www.demonstratingvalue.org
The workbook, snapshot gallery,
blog, newsletter and more
Twitter @demvalue

37. Rating/Benchmarking system
Library/Catalogue of indicators
Framework
Toolbox
Framework
Set of principles
Study design/methodology
Randomized Control Trials

38. The use of RCTs for Social Impact
Assessment: An Example
Presentation to the 2012 Social Finance Forum,
November 8 and 9, 2012

39. Challenges for Social Impact Assessment
 WHAT outcomes to measure – that will best align with your
program goal
 WHICH methods to use – that will best isolate program
effects and allow the attribution of these efects to your
program
 HOW to quantify and monetize these effects – to
demonstrate financial viability
39

40. Demonstrating Impact
 The capacity to measure the difference a new approach or
program makes.
 To measure the true impact of a new program, we need to
know what would have happened if the program had not
been introduced.........we need a counterfactual:
• to account for natural maturation processes, e.g., children skills are
constantly improving
• to account for factors external to the program, e.g., state of the
economy fluctuates and influences labour market outcomes
• to account for “regression to the mean” phenomenon, e.g., lone
mothers on welfare eventually go back to work

41. Displaced workers re-integrating labour market as
a result of being offered an earnings supplement
60%
Full-Time Employment Rate
50%
40%
30%
Program Group
Supplement Group
20%
10%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Month From Random Assignment

42. Displaced workers being offered supplement
versus those not receiving the same offer
60%
Full-Time Employment Rate
50%
40%
30% Program Group
Supplement Group
20% Control Group
Control Group
10%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Month From Random Assignment

43. Why Randomized Experiments?
 To measure the true impact of a new program, we need to
know what would have happened if the program had not
been introduced.........we need a counterfactual:
 Random assignment of participants to a treatment group
and a control group provides the best method to create a
counterfactual
• Not the only method available
• Not always possible to do RA

44. How Random Assignment Works
Recruitment of participants
A random sample of individuals from the population that is
Note: Both program and targeted for the program intervention is recruited and interviewed
control group members
continue to have access to
Informed Consent
government programs and Potential participants sign an informed consent agreeing to be part
services available to of the experiment and provide information for research purposes
members of their
community. Random Assignment
Volunteers are assigned at random to either
the program group or the control group
Program Group
Control Group
Eligible to receive program
Ineligible
intervention
Informed of eligibility for the program
intervention and the conditions Informed of ineligibility status
attached to recipiency
Do not meet conditions for Not eligible for program intervention,
Meet conditions for recipiency
recipiency, therefore, do not but continue to be part of the study
receive program intervention for research purposes
Receive program intervention

45. Properties of RCTs / Social Experiments
 With a large enough sample, random assignment insures that the
two groups of individuals are identical, on average.
• This is true for all observable and unobservable characteristics
(such as motivation, self-confidence, determination, and all other
personal attributes that can explain why an intervention will succeed
with one individual but not another)
 Unlike nonexperimental methods, properly implement social
experiments are guaranteed to provide internally valid impact
estimates – no selection bias.
 Nonexperimental methods may be equally reliable in any given
application; we simply cannot know a priori that they are reliable, as
we can with experimental methods – you can only match
participants on measured characteristics.

46. The Community Employment Innovation Project
A research and demonstration project testing the value of
community-based employment as an alternative to income
transfers in areas of chronic high unemployment
Two parallel but related studies
• Individuals:
• Aims to preserve employability, through faster re-employment
• Provides opportunities for skill development and strengthening of social capital
• Communities:
• Study of a model which utilizes strengths of local communities to create jobs
• Aims to support their capacity growth and improve the social economy

47. The Program Model
The Offer to Individuals
• EI and IA recipients were offered 3 years of full-time employment, on locally
developed projects in exchange for their entitlements to EI or IA
• Employment was designed to replicate full-time market jobs
• 35 hours per week, at $325 a week, EI/CPP insurable, 15 days annual leave,
medical benefits
• Support Services: some job-readiness and transferable skills training
The Offer to Communities
• 6 communities in the Cape Breton Regional Municipality were offered a free
workforce of 750 workers for up to five years
• Each community was required to elect a representative board, develop a
strategic plan, and approve projects
• Local control given to communities – explicitly links projects to local needs

48. Participants Impact Study
Key outcomes of interest
• Economic well-being
• Employment, earnings, transfer receipt, income, poverty, and
hardship
• Longer-term employability: Skills and experience
• Characteristics of post-CEIP employment, employability
skills, attitudes to work
• Social Capital, volunteering, life satisfaction

49. CEIP impacts on EI sample
100 Program Group
A 53 95 Control Group
percentage 90 Impacts
point 85
impact at 80
75
peak
70
Percentage Employed Full Time
65
60
55
50
45
40
35
30
25
20 No significant
15
10
impacts a year
5 after end of
0 CEIP eligibility
-5
-10 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
-15
-20
Months From Random Assignment

50. IA Sample: Large in-program impacts on
employment, but not sustained
100 Program Group
Nearly 80 95 Control Group
90 Impacts
percentage 85
point impact 80
at peak 75
70
65
Percentage Employed Full Time
60
55
50
45
40
35
30
25
20
15 No significant
10 impacts a year
5 after end of
0 CEIP eligibility
-5
-10 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
-15
-20
Months From Random Assignment

51. IA Sample: Permanent reductions in IA receipt
three years after CEIP comes to an end
100 Program Group
Control Group
Impact
80
60
Percentage Receiving IA
40
20
0 A sustained
1 11 21 31 41 51 61 71
12 percentage
point reduction
A stable -20
in IA receipt
42 percentage
point reduction
-40
in IA receipt
during program
-60
Months From Random Assignment

52. Impacts on Social Capital:
Sustained increase in the size of networks
Network Size
Change in number of contacts
from baseline to 54 months
2.5
2.0
54-month gain
1.5
1.0
0.5
0.0
Help with chores Emotional Specialized Help finding a job
support advice
Program Control

53. Community Engagement:
Sustained increase in volunteering and social contact
 Volunteering is important to both individuals and communities
 Can be an avenue to skill development, improves social
inclusion, and is a large resource for many community organizations
54-Month: Impacts on Formal volunteering with
groups or organizations
50
45
40
35
Percent
30 Program
25
20 Control
15
10
5
0
Volunteered in the past 12 Volunteered in the past 12
months months
EI Sample IA Sample

54. CEIP Theory of Change:
Analytical framework of expected change
Individual Community Response Organizational
Early Engagement Organizing, Planning, Mobilization Mobilization
Mobilization
Years 1-3
Process Process
effects CEIP Projects effects
Provide work experience and
valued community Services
Project
Development
and Interim
Effects
Years 4-5
Skill Gains,
Social
Social Capital Inclusion, Cohesion
Well Being and Community
Improves
Post-program Capacity Improves
Long-run
Effects
Years 6-7 Employment Levels, Market
Conditions Improves

55. Targeted Community Sectors
 The two largest project categories were similar across communities
 Youth projects were most prevalent in New Waterford and Whitney Pier
 Seniors projects were at greatest scale in Sydney Mines
Figure ES.5: Percentage of Work Years Assigned, by Community and Sector Served
100%
90% Health, Environment,
Percentage of total work years assigned to a sector
Beautification
80%
Recreation, Arts and
Culture
70%
Services for the Poor,
60% Unemployed
50% Supports for Seniors
40%
Supports for Youth
30%
Other: Services for
20% the Disabled, CED
Agencies, CEIP
Boards
10%
0%
Glace Bay NewWaterford North Sydney Sydney Mines Whitney Pier
Community

56. Results: Theory Supports Observation
Preponderance of positive change in more successful communities
New Sydney Whitney North
Dominion Glace Bay
Waterford Mines Pier Sydney
Social Capital + + + + +
Social Cohesion + + + +
Social Inclusion + + + - +
Third Sector Relative Size +
Organizational Capacity + + + + +
Economic Outcomes
Social Conditions + + +
Youth Effects + + + + -
Senior Effects + + + + +
Effects on the Poor + + + +

57. 57

58. 58

59. Cost-Benefit Analysis Results (IA Sample)
Net benefit-cost per IA program group member over the full 54-month follow-up
Accounting Perspective
Component of Analysis Individuals Communities Government Society
Monetized components
Participant Impacts
CEIP earnings 34 344 0 -34 344 0
Foregone non-CEIP earnings -10 974 0 0 -10 974
Transfer payments (EI & IA) -11 836 0 11 836 0
Tax payments (taxes and premiums) -3 559 0 2 921 -638
Other household member earnings 2 035 0 0 2 035
Third Sector Organizational Effects
Value from CEIP jobs (to sponsors) 0 20 024 0 20 024
Volunteering (CEIP induced) 0 2 404 0 2 404
CEIP administrative costs 0 0 -4 274 -4 274
Admin costs of EI & IA transfers 0 0 471 471
Net Benefit/Cost per Program Group Member 10 010 22 428 -23 390 9 048

60. Benefit-Cost Ratio
 CEIP is very cost effective when one considers the combined benefits to
individuals and community -- $1.39 in net benefits per dollar for IA
recipients
Total net benefit for every dollar that government spent on CEIP
1.60
1.39
1.40 EI Sample IA Sample
Benefit-Cost Ratio
1.20
Positive Net Present Value 1.02
0.96
1.00
0.81
0.80
0.60
0.43
0.40
0.21
0.20
0.00
Participants Communities Society
Perspective

61. Cost-Benefit Analysis Accounting framework
Accounting Perspective
Component of Analysis Individuals Communities Government Society
Non-Monetized components
Participant Impacts
Reductions in Hardship, Stress + 0 0 +
Improved Social Capital + 0 0 +
Increased Trust in Networks + 0 0 +
Foregone Leisure – 0 0 –
Community Effects
Increased Social Capital of Residents 0 + 0 +
Improved Community Cohesion 0 + 0 +
Increased Social and Civic Participation 0 + 0 +
Foregone Leisure 0 – 0 –

62. 62

63. Valuing Intangibles – Examples
Social Capital – each additional contact is valued at 7% of income;
therefore CEIP impact is worth $3,808 per participant
Trust – each additional percentage point increase in trust is valued at
2.5% of income; CEIP impact is worth $2,401 per participant
Hardship – the reductions in stress associated with lower hardship
during CEIP is valued at $3,379

65. Benefit-Cost Ratio
 Including the intangible impacts improves the benefit cost ratio to
$1.61 in net benefits per dollar spent
 About a 50 percent improvement in overall net benefit to society
Total net benefit for every dollar that government spent on CEIP
1.80
1.61
1.60 EI Sample IA Sample
1.40
Benefit-Cost Ratio
1.20
1.20 Positive Net Present Value
0.96
1.00
0.81
0.80 0.65
0.60
0.40
0.40
0.20
0.00
Participants Communities Society
Perspective

66. Take aways
 To find out what difference a program makes, one needs to find an
appropriate counterfactual. RCTs provide the best approach to set up a
counterfactual and derive an Impact.
 An Impact measure is required to do a proper Cost-Benefit Analysis;
and a sound CBA is needed to report on the financial viability of a
program
 Analytical frameworks and Cost-Benefit Analysis should and can
incorporate social benefits
 Short of convincing private sector investors or government authorities to
transform their accounting or fiscal framework to include environmental
and social benefits, attempts should be made to monetize these
benefits.

67. WWW.SRDC.ORG
67

68. Randomized Control Trials