Measuring Success introduces nonprofit professionals to proven techniques on how to move from anecdotal to data-driven decision making and steer your organization to success. Gain insights on how to focus your limited organizational time and energies on the issues that are supported by data instead of anecdotes. Learn techniques for using data to track and measure progress over time, report impact to stakeholders, and manage toward success.
4. Benefits of data-driven decision making
to Customers
Success matters because your mission is important!
If you don’t measure and use data, you are not
achieving the full potential of your mission.
Questions? info@measuring-success.com #7steps
5. Benefits to Management & Staff
Focus limited time and energies on key activities that are
real issues and have impact on our mission
Set goals for staff and hold them accountable
“Cannot manage what you do not measure” – Peter Drucker
“What gets measured gets done”
“High performing organizations use data analytics roughly
three times as extensively as the lower performers in their
field” – Competing on Analytics, HBS Press
Questions? info@measuring-success.com #7steps
6. Benefits to Boards and Donors
Boards Donors
Focus their energies away Ability to show “return on
from emotions & anecdotes investment”
Decisions not made by loudest Attract large gifts and grants
or wealthiest voice by confidence engendered by
data
“Analytical projects aimed at
improved outcomes had a Data analytics was one of the
median ROI of 55% for CRM top three areas of increased
and 139% for financial investment by companies
management” during the recession.
– Competing on Analytics, HBS
Press
Questions? info@measuring-success.com #7steps
7. Popular Press Examples
Professional sports: Oakland A’s (Moneyball)
Maximize team wins while minimizing payroll
By hiring players as indicated by data, not scouts’ gut
Health Care: Intermountain Health Care (NY Times Magazine Nov 9, 2009)
Improve patient outcomes (survival, longevity, quality of life) while
minimizing cost
By selecting treatment/procedure supported by evidence not Doctor’s
gut instinct or unneccessary procedures
Technology: Google
Increase advertising revenues ($21 billion) by targeting ads to right
customers
By mining data on all users of free Google consumer software
Questions? info@measuring-success.com #7steps
8. B. Quick Case Study
Questions? info@measuring-success.com #7steps
9. 1. Chapter-based
community centers
Experiencing some with member drops, financial insolvency.
Challenge to business model & brand consistency.
Association HQ
Local Affiliates or Regions
(N=150)
Individual Members / Users
(N=1000 to 10,000 per affiliate)
Questions? info@measuring-success.com #7steps
10. 2. Association sought “early warning
system” & understanding of what led to
successful outcomes
Questions? info@measuring-success.com #7steps
11. 3. Build measurement and dashboard
system as pilot experiment
Engaged 6 willing chapters
Built:
Customer survey
Employee survey
Financial analysis tool
Member participation tool
Questions? info@measuring-success.com #7steps
12. 4. Focused on rankings and metrics that
were statistically valid
Regression analysis ties activities to outcomes
Comparison
Against peer chapters in other geographies
Against local competition from other organizations
Against own prior measures (longitudinal)
Within demographic segments of member base
(see following slides)
Questions? info@measuring-success.com #7steps
13. 4a. Identify activities associated with outcomes
(multiple regression analysis)
Budget Management versus Value for Membership Dollar
Correlation
Value for Membership Dollar
Average Score 1-5 Scale
Perceived Budget Management & Transparency
(1= strongly disagree, 5 = strongly agree)
Questions? info@measuring-success.com #7steps
14. 5. Like doctor, ran diagnostics on each
chapter every 2 years
Rolled out pilot to all chapters;
readiness factors to participate
Mutually agreed on a plan for
improvement
Some consulting support
Select Measures from Customer Rank Score Priority Goals & Strategy
Survey (of 15) % Str
Agree
Membership Value for the Dollar 7 25% Medium Focus on
quality, budget
perceptions
Professionals welcoming 2 35% Low
Budget Perceived as well 14 16% High Double scores in
managed 2 years.
Questions? info@measuring-success.com #7steps
16. 6. Turnaround: focus lead to improvement from
low to average in 2 years; now aiming for top
Personal Conversations with Customers
Monthly or more often
27%
Questions? info@measuring-success.com #7steps
17. Data encourages prioritization:
80% of board & mgmt team hypotheses about what we
anecdotally “believe” is a problem is not supported by data!
Questions? info@measuring-success.com #7steps
18. 7. Chapter ruled by anecdotes: 80% of
assumptions were not supported by data
Assumed they were strongest with eldest & wealthiest portions of
participant base
Likelihood to recommend to a friend
% Strongly Agree
25-34 35-44 45-54 55-64 65-74 Over 75
50% 43% 50% 48% 45% 40%
Less than $100,000- $200,000- $300,000- $400,000- $500,000 and
$100,000 $199,999 $299,999 $399,999 $499,999 over
40% 42% 67% 63% 42% 46%
Shock, challenged the data, acceptance
Management team focused energies on improvement with eldest
and wealthiest. 2 years later, scores significantly higher there.
Questions? info@measuring-success.com #7steps
19. 8. Rising Tide Lifts All Boats: top, average,
and low performers all improved
Surplus Margin for Early Childhood Program
Surplus as % of Expenses, after allocating all overhead
X X
Questions? info@measuring-success.com #7steps
20. 9. New policy: value, not price
Highest Willingness
Assumed that key driver of To Pay (center)
Demand Function
member retention was price
C. Perceived
Analysis shows not price, but Quality of Services Least Ability to Affect
perceived quality and value-
for-dollar
Result: association stopped
encouraging price B. Commitment A. Financial
subsidization, encouraged To Issue Ability
perceived quality improvement
Perceived value and value for
dollar are tracked carefully and
promoted system wide
Questions? info@measuring-success.com #7steps
21. 10. Outcome improvement: participation
Chapters that embraced this approach outperformed others
significantly in participant enrollment and financial sustainability,
despite the recession
Participant Retention Rate:
91% to 96%
New Participant Rate:
5% to 10%
Net Participation:
96% to 106%
Financial sustainability increased: coverage ratio (% expenses
from fees & membership) grew from 74% to 80%
Questions? info@measuring-success.com #7steps
22. C. 7 Steps to Data-Driven
Decision Making
Questions? info@measuring-success.com #7steps
23. Data driven decision making is parallel to what
we learned in middle school science class
Identify issue
State hypothesis: “I
believe…”
Perceived mechanism/
cause
Design experiment
Examine data
Confirm or reject
hypothesis
Questions? info@measuring-success.com #7steps
24. 7 Stages of Data-Driven Decision Making
1.
Framing
the 2.
Problem Hypothesis
Develop-
ment 3. Data
Collection
4. Data
Analysis 5.
Inter-
pretation 6.
Decision
Making 7.
Commun-
ication
Questions? info@measuring-success.com #7steps
25. 2. Hypothesis Development
Brainstorm hypotheses. Many will turn out to be
incorrect, but that’s ok so long as you can
articulate a plausible mechanism.
Remember, our consistent experience with organizations is
that over 80% of their initial hypotheses as to what is
driving a problem are not supported by the data!
During brainstorm define:
Hypothesis (may need to revise several times)
Mechanism (cause & effect)
How could measure it
Questions? info@measuring-success.com #7steps
26. 2. Hypothesis Development:
Gann Academy
Initial Hypothesis: unevenness in students’ experience
getting their individual learning needs met by the school
Plausible Mechanism: additional academic support
(learning center).
Major differentiator in students’ learning experience (for those
in the same classrooms)
Final Hypothesis: students who received additional
academic support from learning center felt their
individual learning needs were better met by the school.
Questions? info@measuring-success.com #7steps
27. 2. Hypothesis Development:
Gann Academy
How measure?
Parent survey
Question on individual learning needs met by school
Cut data by whether student used learning center
t Scale 10a. Teachers
Understand
Needs & Employ
Effective Learning Learning
Techniques Center Yes Center No
Strongly Disagree 3%
Disagree 9%
Neither Agree Nor Disagree 16% Count 84 240
Agree 44%
Average 3.5 4.0
Strongly Agree 28%
Average 3.8
Questions? info@measuring-success.com #7steps
28. 3. Data Collection
Do… Don’t…
Prioritize hypotheses Just go after what’s
Work backwards: figure out easiest
how will use results of data Overtax your resources –
first before collect it data collection can be very
Consider several methods of time intensive
data collection Go after more data than
Save time and money by you can act upon –
considering simple, pre- Garbage In Garbage Out
built tools allowing some (GIGO)
customization (instead of
building own or buying
highest end product you
won’t use)
Questions? info@measuring-success.com #7steps
29. 3. Data Collection
Tool Pro Con
Survey Data collection phase is To get quality results and high
automated and not resource response rates, need to put lots
intensive of time into asking the right
questions, making it look
Obtain perceptions of quality professional, and assuring
confidentiality
Financial analysis Data your organization already Requires reorganizing and “fine
has slicing” the financials, so need a
strong CFO
Tracking system Integrated into daily efforts Takes a lot of discipline and
(CRM) time to build data
Self-reported from Already have the database Time consuming if database
existing databases does not contain the right
information may end up with
lots of estimates
Questions? info@measuring-success.com #7steps
30. 3. Data Collection : Gann Academy
Good: financial modeling Bad: admission tracking
Clarified board’s needs up Purchased high end
front: 5 year projections, database system
key levers of sustainability Not trained properly
Used already-built but Orphan Excel spreadsheets
simple model (in Excel) Never clarified what
instead of building own management needed for
Worked hard to gather decision making, so
consistent financial data “GIGO”
(counting $ in development
and CFO’s office)
“What if” scenario building
enabled board buy-in
Questions? info@measuring-success.com #7steps
31. 3. Data Collection : Gann Academy
Work backwards. Decide how you will use the data first, as this
will dictate whether and how to ask for the data.
Questions? info@measuring-success.com #7steps
32. 5. Interpretation: draw correct conclusions from
data. Danger in “seeing what want to believe”
You don’t need to be a statistician to correctly interpret data
However, do need to understand basic concepts
Mean
Standard deviation
Counts (sample size)
Benchmarking
Frequently, someone will “find” something in the data that is not
valid but it supports their personal views
Questions? info@measuring-success.com #7steps
33. 5. (Mis)Interpretation: Gann Academy
Relative Strength of Gann Academy’s
Board member who was Attention to Individual Needs vs.
convinced the most severe Most Attractive Alternative School
(higher score is better)
threat on enrollment was from
charter and public schools, not Gann vs. Gann vs. Gann vs.
area private schools Charter
Hypothesis: public schools are or General
Private
better at supporting individual magnet public
Schools
public schools
student needs and thus pose the
schools
greatest risk of attrition for
Gann Academy families.
Count 50 4 51
Data source: teacher attention
Average 3.8 3.4 3.6
to individual needs cut by most
attractive alternative option
Questions? info@measuring-success.com #7steps
34. 5. Interpretation: Benchmarking
Relative to What?
Contextualization is critical:
Peer group (similar organizations
nationally/internationally)
Local competition*
Own longitudinal history
Across demographic groups*
*Depicted on a prior slide
Questions? info@measuring-success.com #7steps
35. 5. Interpretation: Benchmarking
Relative to Peer Group
of other similar high
schools around the
country
Note if had used
primary schools (K-8) as
peer group
Questions? info@measuring-success.com #7steps
36. 7. Communication
Sharing with board, participants, donors, employees
Fear: sharing data will undermine management’s authority
Reality: sharing data engenders trust
Diffuse concerns through transparency
Even if current scores are not as high as would like, donors value a
roadmap toward and metrics for success
Questions? info@measuring-success.com #7steps
37. 7 Stages of Data-Driven Decision Making
1.
Framing
the
2.
Problem
Hypothesis
Develop-
ment 3. Data
Collection
4. Data
Analysis 5.
Inter-
pretation 6.
Decision
Making 7.
Commun-
ication
Questions? info@measuring-success.com #7steps
38. Common Non-Profit Applications
of Data-driven Decision Making
Outcome measurement
Identifying activities that give greatest “bang for buck” in limited
resource environment
Financial sustainability
Setting program fees or membership dues levels
Identifying most susceptible demographic groups
Donor prioritization using predictive models
Board reporting
Program evaluation
Questions? info@measuring-success.com #7steps
39. If your organization makes data-driven
decisions you will be able to…
Focus limited energies and resources on only the 20% of hypotheses that
are supported by the data
Motivate staff and volunteers because they know they are investing in
what works
Align efforts of staff, volunteers, board, donors, and customers
Monitor progress toward goals
Have a dashboard for your performance
Make strategic board decisions more efficiently and gain more buy-in
Distinguish yourself against and outperform your competition by
developing a core competence few non-profits have
Observe improvements in key areas such as: financial sustainability,
donations, participation, mission impact
Questions? info@measuring-success.com #7steps
40. 7 Steps for Data-Driven
Decision-Making
Sacha Litman, Managing Director, Measuring
Success
José Fernández, Director of GuideStar Exchange
Questions? info@measuring-success.com #7steps