The document summarizes a presentation by Alistair Croll on using lean analytics for local government projects. Some key points from the presentation include: - Most government projects have an attention or connectivity problem, which is where innovating will spend most time. - Lean analytics lessons include choosing one metric to rally around and rejecting vanity metrics, as well as designing experiments to test hypotheses around potential improvements. - The lean analytics cycle involves picking a key metric, finding areas for potential improvement, designing a hypothesis-driven experiment, making changes, and iterating based on results.

1. April 22, 2013
LEAN ANALYTICS & LOCAL GOV
ALISTAIR CROLL
Tuesday, 23 April, 13

2. Alistair Croll
Co-Author, Lean Analytics
Tuesday, 23 April, 13

3. Lean Analytics
Use data to build a
better business faster.
www.leananalyticsbook.com
@byosko | @acroll
@leananalytics
Tuesday, 23 April, 13

4. Analytics is the measurement
of movement towards your
business goals.
http://www.flickr.com/photos/itsgreg/446061432/
Tuesday, 23 April, 13

5. Most startups don’t know what they’ll be
when they grow up.
Tuesday, 23 April, 13

6. Most startups don’t know what they’ll be
when they grow up.
Paypal
first built for
Palmpilots
Tuesday, 23 April, 13

7. Most startups don’t know what they’ll be
when they grow up.
Freshbooks
Paypal was invoicing
first built for for a web
Palmpilots design firm
Tuesday, 23 April, 13

8. Most startups don’t know what they’ll be
when they grow up.
Freshbooks
was invoicing Wikipedia
Paypal
for a web was to be
first built for
design firm written by
Palmpilots experts only
Tuesday, 23 April, 13

9. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal lawnmower
for a web was to be
first built for company
design firm written by
Palmpilots experts only
Tuesday, 23 April, 13

10. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal lawnmower
for a web was to be
first built for company
design firm written by
Palmpilots experts only
Hotmail
was a
database
company
Tuesday, 23 April, 13

11. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal lawnmower
for a web was to be
first built for company
design firm written by
Palmpilots experts only
Flickr
Hotmail
was going to
was a
be an MMO
database
company
Tuesday, 23 April, 13

12. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal lawnmower
for a web was to be
first built for company
design firm written by
Palmpilots experts only
Flickr
Hotmail Twitter
was going to
was a was a
be an MMO
database podcasting
company company
Tuesday, 23 April, 13

13. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal lawnmower
for a web was to be
first built for company
design firm written by
Palmpilots experts only
Flickr
Hotmail Twitter Autodesk
was going to
was a was a made desktop
be an MMO
database podcasting automation
company company
Tuesday, 23 April, 13

14. Kevin Costner is a lousy entrepreneur.
Don’t sell what you can make.
Make what you can sell.
Tuesday, 23 April, 13

15. Tuesday, 23 April, 13

16. The basic Lean message is
learn and adapt, fast.
Tuesday, 23 April, 13

17. “What information
consumes is rather
obvious: it consumes
the attention of its
recipients.
Hence a wealth of
information creates a
poverty of attention, and a
need to allocate that
attention efficiently among
the overabundance of
information sources that
might consume it.”
(Computers, Communications and the Public Interest, pages 40-41,
Martin Greenberger, ed., The Johns Hopkins Press, 1971.)
Tuesday, 23 April, 13

18. http://www.flickr.com/photos/art_es_anna/288880795/
Tuesday, 23 April, 13

19. Tuesday, 23 April, 13

20. Tuesday, 23 April, 13

21. Tuesday, 23 April, 13

22. Lean Analytics lesson 1:
Most government projects have an
attention or a connectivity problem.
This is where you will spend most
of your time innovating.
Tuesday, 23 April, 13

23. Empathy stage:
Localmind hacks Twitter
• Stage: Empathy
• Model: UGC/mobile
• Real-time question and answer platform tied to locations.
• Needed to find out if a core behavior—answering questions about a place—
happened enough to make the business real
Tuesday, 23 April, 13

24. Localmind hacks Twitter
• Before writing a line of code, Localmind was concerned that people would never
answer questions.
• This was their biggest risk: if questions went unanswered users would have a
terrible experience and stop using Localmind.
• Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about
the area: how busy is it; is the subway running on time; is something open;
etc.
• The response rate to their tweeted questions was very high.
• Good enough proxy to de-risk the solution, and convince the team and
investors that it was worth building Localmind.
Tuesday, 23 April, 13

25. Tuesday, 23 April, 13

26. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Tuesday, 23 April, 13

27. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Tuesday, 23 April, 13

28. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
Tuesday, 23 April, 13

29. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Tuesday, 23 April, 13

30. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Tuesday, 23 April, 13

31. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Time on site, or Poor version of engagement. Lots of time spent on
pages/visit support pages is actually a bad sign.
Tuesday, 23 April, 13

32. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Time on site, or Poor version of engagement. Lots of time spent on
pages/visit support pages is actually a bad sign.
How many recipients will act on what’s in them?
Emails collected
Tuesday, 23 April, 13

33. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Time on site, or Poor version of engagement. Lots of time spent on
pages/visit support pages is actually a bad sign.
How many recipients will act on what’s in them?
Emails collected
Number of Outside app stores, downloads alone don’t lead to
downloads lifetime value. Measure activations/active accounts.
Tuesday, 23 April, 13

34. 2-sided market model:
AirBnB and photography
• Stage: Revenue
• Model: 2-sided marketplace
• Rental-by-owner marketplace that allows property owners to list and market
their houses. Offers a variety of related services as well.
Tuesday, 23 April, 13

35. AirBnB tests a hypothesis
• The hypothesis: “Hosts with professional photography will get more business.
And hosts will sign up for professional photography as a service.”
• Built a concierge MVP
• Found that professionally photographed listings got 2-3x more bookings than the
market average.
• In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for
hosts.
Tuesday, 23 April, 13

36. NIGHTS BOOKED
10 million
8 million
6 million
20 photographers
4 million
2 million
2008 2009 2010 2011 2012
Tuesday, 23 April, 13
Friday, November 9, 12

37. Pick the right experiments
http://www.flickr.com/photos/bootbearwdc/1243690099/
Tuesday, 23 April, 13

38. http://www.flickr.com/photos/circasassy/7858155676/
If it won’t change
how you behave,
it’s a
bad
metric.
Tuesday, 23 April, 13

39. The five Stages of Lean Analytics
Tuesday, 23 April, 13

40. The five Stages of Lean Analytics
The stage you’re at
Tuesday, 23 April, 13

41. The five Stages of Lean Analytics
Empathy
The stage you’re at
Tuesday, 23 April, 13

42. The five Stages of Lean Analytics
Empathy
The stage you’re at
Stickiness
Tuesday, 23 April, 13

43. The five Stages of Lean Analytics
Empathy
The stage you’re at
Stickiness
Virality
Tuesday, 23 April, 13

44. The five Stages of Lean Analytics
Empathy
The stage you’re at
Stickiness
Virality
Revenue
Tuesday, 23 April, 13

45. The five Stages of Lean Analytics
Empathy
The stage you’re at
Stickiness
Virality
Revenue
Scale
Tuesday, 23 April, 13

46. The five Stages of Lean Analytics
The business you’re in
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy
The stage you’re at
Stickiness
Virality
Revenue
Scale
Tuesday, 23 April, 13

47. The five Stages of Lean Analytics
The business you’re in
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy
The stage you’re at
One Metric
Stickiness
Virality
Revenue That Matters.
Scale
Tuesday, 23 April, 13

48. Lean Analytics lesson 2:
Choose one metric around which
to rally support, and reject vanity
metrics ruthlessly.
Tuesday, 23 April, 13

49. Choose only one metric.
Tuesday, 23 April, 13

50. Metrics are like
squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/
Tuesday, 23 April, 13

51. Metrics in practice:
The Lean Analytics Cycle
Tuesday, 23 April, 13

52. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM
Tuesday, 23 April, 13

53. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Tuesday, 23 April, 13

54. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Tuesday, 23 April, 13

55. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without
data: make a
good guess
Tuesday, 23 April, 13

56. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without With data:
data: make a find a
good guess commonality
Tuesday, 23 April, 13

57. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without With data:
data: make a find a
good guess commonality
Hypothesis
Tuesday, 23 April, 13

58. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without With data:
data: make a find a
good guess commonality
Hypothesis
Make changes
in production
Tuesday, 23 April, 13

59. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without With data:
data: make a find a
good guess commonality
Design a test
Hypothesis
Make changes
in production
Tuesday, 23 April, 13

60. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

61. Metrics in practice:
The Lean Analytics Cycle
Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

62. Metrics in practice:
The Lean Analytics Cycle
Success! Pick OMTM Draw a line
in the sand
Find a
potential
improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

63. Metrics in practice:
The Lean Analytics Cycle
Success! Pick OMTM Draw a line
in the sand
Pivot or
give up Find a
potential
improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

64. Metrics in practice:
The Lean Analytics Cycle
Success! Pick OMTM Draw a line
in the sand
Pivot or
give up Draw a new line Find a
potential
improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

65. Metrics in practice:
The Lean Analytics Cycle
Success! Pick OMTM Draw a line
in the sand
Pivot or
give up Draw a new line Find a
potential
Try again improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production
Tuesday, 23 April, 13

66. Lean Analytics lesson 3:
There’s no “finished.” Just more
iterations.
Tuesday, 23 April, 13

67. The B2B stereotype
• Domain expert knows
industry and the problem
domain. Has a Rolodex;
proxy for customers.
http://www.techdigest.tv/2007/02/im_a_pc_im_a_ma.html
• Disruption expert knows
tech that will produce a
change Sees beyond the
current model.
Domain Disruption
expert expert Operations
Tuesday, 23 April, 13

68. The B2B stereotype
• Domain expert knows
industry and the problem
domain. Has a Rolodex;
proxy for customers.
http://www.techdigest.tv/2007/02/im_a_pc_im_a_ma.html
• Disruption expert knows
tech that will produce a
change Sees beyond the
current model.
Domain Disruption
expert expert Operations
Tuesday, 23 April, 13

69. Three typical approaches
Create a popular consumer Dropbox
Enterprise pivot product then pivot to tackle the
enterprise
Take an existing consumer or Yammer,
Copy and rebuild open source idea and make it MapR
enterprise-ready
Convince the enterprise to Taleo,
Disrupt a problem discard the old way because of Google
overwhelming advantages. Apps
Tuesday, 23 April, 13

70. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Tuesday, 23 April, 13

71. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support
Tuesday, 23 April, 13

72. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support
Word of mouth, references, case Bad vibes;
Virality studies exclusivity
Tuesday, 23 April, 13

73. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support
Word of mouth, references, case Bad vibes;
Virality studies exclusivity
Growing direct sales, professional Pipeline, revenue
Revenue services, support recognition, comp
Tuesday, 23 April, 13

74. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support
Word of mouth, references, case Bad vibes;
Virality studies exclusivity
Growing direct sales, professional Pipeline, revenue
Revenue services, support recognition, comp
Channels, analysts, ecosystems, Crossing the
Scale APIs, vertically targeted products chasm; Gorillas
Tuesday, 23 April, 13

75. The Zero Overhead principle
A central theme to this new wave of
innovation is the application of core product
tenets from the consumer space to the
enterprise.
In particular, a universal lesson that I keep
sharing with all entrepreneurs building for the
enterprise is the Zero Overhead Principle: no
feature may add training costs to the
user. DJ Patil
Tuesday, 23 April, 13

76. Lean Analytics lesson 4:
Government can learn from
enterprise-focused startups:
Disrupt a known problem with new
technology.
Tuesday, 23 April, 13

77. Skunk Works for intrapreneurs
• The Lockheed Martin Skunk Works
Tuesday, 23 April, 13

78. Span of control and the railroads
• Daniel C. McCallum
Tuesday, 23 April, 13

79. The BCG matrix
• How businesses think
about products or Question marks! increase
Pivot to
Stars!
companies (low market share, market (high growth rate,
share
high growth rate)
through high market share)
May be the next big thing. virality, What everyone wants. As
• Lean is about moving Consumes investment, but attention
market invariably stops
will require money to growing, should become
up and to the right Growth rate
increase market share.
cash cows.
Milk with
Pivot to Pivot to
revenue
redefine problem/ increase growth
optimization as
solution through rate through
growth slows
empathy
disruption
Dogs! Cash cows!
(low market share, (high market share,
low growth rate)
low growth rate)
Barely breaks even, may Boring sources of cash, to
be a distraction from better be milked but not worth
opportunities. Sell off or additional investment.
shut down.
Market share
Tuesday, 23 April, 13

80. BCG and policy
Widely
popular
Public support
Widely
ridiculed
Tiny Impact on society Huge
Tuesday, 23 April, 13

81. BCG and policy
Widely
popular
Public support
Widely Pork
ridiculed
Tiny Impact on society Huge
Tuesday, 23 April, 13

82. BCG and policy
Widely
popular
Public support
Banning
leaded gasoline
Widely Pork
ridiculed
Tiny Impact on society Huge
Tuesday, 23 April, 13

83. BCG and policy
Widely
popular “I Declare
today
Jebbediah
Springfield
Public support
day”
Banning
leaded gasoline
Widely Pork
ridiculed
Tiny Impact on society Huge
Tuesday, 23 April, 13

84. BCG and policy
Widely
popular “I Declare
today
Jebbediah
Springfield
Public support
day”
Banning
leaded gasoline
No
more
big
soda
Widely Pork
ridiculed
Tiny Impact on society Huge
Tuesday, 23 April, 13

85. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Tuesday, 23 April, 13

86. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep
Tuesday, 23 April, 13

87. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep
Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing
Tuesday, 23 April, 13

88. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep
Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing
Consider the ecosystem, channels, Channel conflict,
Revenue and established agreements resistance, contracts
Tuesday, 23 April, 13

89. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep
Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing
Consider the ecosystem, channels, Channel conflict,
Revenue and established agreements resistance, contracts
Hand the baton to others gracefully Hating what happens
Scale to your baby
Tuesday, 23 April, 13

90. Lean Analytics lesson 5:
When working from within, the
difference between a special
operative and a rogue agent is a
mandate.
Tuesday, 23 April, 13

91. E-commerce enterprise
• Stage: Scale
• Model: e-commerce
• EMI was a big music company trying to understand how its customers bought
content
Tuesday, 23 April, 13

92. David Boyle tackles a small problem
• David Boyle, SVP Insight, EMI Music, ran the insight group
• Had billions of rows of data, but nobody wanted to analyze it
• Instead started a survey project, got a million responses, used this data to sell
the idea of “data-driven” business
• Then got support for the broader data initiative.
Tuesday, 23 April, 13

93. David Boyle tackles a small problem
• Talked to 1M people in
3y
• At any point, surveying
12 people in the world
• Boiled this down to a
few fundamental
profiles
Tuesday, 23 April, 13

94. Lean Analytics lesson 6:
Start with a less important project
(preferably one that involves
intelligence gathering.) Then use
that success for bigger
undertakings.
Tuesday, 23 April, 13

95. Where your lean lives:
It’s learning: Places with lots of data
It’s rapidly iterated: Apps or software
It’s popularizing: Moving up a box
It’s impact-increasing: Moving right a box
It has high uncertainty: De-risk with an MVP
It’s boundable: Lean hates molasses
Tuesday, 23 April, 13

96. Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844
Tuesday, 23 April, 13

97. ARCHIMEDES
HAD TAKEN
BATHS BEFORE.
Tuesday, 23 April, 13

98. Once, a leader convinced others in
the absence of data.
Tuesday, 23 April, 13

99. Now, a leader knows what
questions to ask.
Tuesday, 23 April, 13

100. Ben Yoskovitz
byosko@gmail.com
@byosko
Alistair Croll
acroll@gmail.com
@acroll
Tuesday, 23 April, 13

101. Thank you!
Further discussion:
muni-innovation@googlegroups.com
Contact staff: pn-staff@codeforamerica.org
Tuesday, 23 April, 13