This was presented by Matt Stempeck and Micah L. Sifry from Civic Hall at the Impacts of Civic Technology Conference (TICTeC 2018) in Lisbon on 18th April 2018. You can find out more information about the conference here: http://tictec.mysociety.org/2018

1. The problem with impact
measurement in Civic Tech
Matt Stempeck & Micah Sifry
#impactproblems

2. The 10 problems with impact
measurement in Civic Tech
Matt Stempeck & Micah Sifry
#impactproblems

3. The 10 problems with impact measurement
1. We're all using different metrics
2. Sharing is irregular
3. Most projects don't reach most people
4. Different constituencies want different impacts
5. We don't evaluate relative to investment made
6. We don't evalue relative to the macro environment
7. Quantitative metrics can miss the objective
8. Qualitative cases are too often biased
9. Causality is hard to prove in social environments
10. Our externalities may eat us all

4. Working backwards
Outcomes ->
Proxies for progress ->
Prove our causality

5. Working backwards
Outcomes ->
Proxies for progress ->
Prove our causality

6. 1. We're all using different metrics
API
calls
Users
Time
spent
Money
raised
"Wins"
Money
saved
Votes Installs Shares Media

7. 1.
http://www.networkimpact.org/civictecheval/

8. 2. Sharing is irregular
Bi-annual or quarterly reports
with metrics and stories
"Impact" page with stories Couldn't find 2015
and 2016 reports

9. 3. Most projects don't reach most people
Platform Reach Percent of US
population
Percent of world
population
Change.org 237,426,190 users 72.8% 3.2%
Jumo 1,000,000 users
when acquired in 2011
3.2% 0.01%
⚡ ⚡ ⚡
Everyblock 145,000 users
when acquired in 2009
0.05% 0.002%
Civic Hall 14,518 event guests
in 2017
0.004% 0.0002%

10. 3. Most projects don't reach most people
Platform Reach Percent of US
population
Percent of world
population
Change.org 237,426,190 users 72.8% 3.2%
Jumo 1,000,000 users
when acquired in 2011
3.2% 0.01%
⚡ ⚡ ⚡
Everyblock 145,000 users
when acquired in 2009
0.05% 0.002%
Civic Hall 14,518 event guests
in 2017
0.004% 0.0002%
Matt's Twitter 6,000 0.002% 0.00008%

11. 4. Different constituencies want different impacts

12. 4. Different constituencies want different impacts

13. 4. Different constituencies want different impacts

14. 5. We don't evaluate relative to investment made
INPUTS OUTCOMES
THE
WORK

15. 6. We don't evaluate relative to the environment

16. 7. Quantitative metrics can miss the objective
2 + 2 =

17. 8. Qualitative cases are too often biased

18. 9. Causality is hard to prove in social environments
y
x

19. 10. Our externalities may eat us all

20. Some imperfect proxies for outside researchers

21. This is one reason we are
building the civic tech field
guide

22. Track impact by platform
bit.ly/organizecivictech
Micah Sifry
micah@civichall.org
Matt Stempeck
stempeck@gmail.com