In an increasingly challenging donor environment, funders want more meaning reporting of success and outcomes by nonprofits. This webinar provides insights and knowledge that can mean the difference between scaling up - or dialing down - key initiatives.
3. Progress in Cancer:
Measuring Success in Patient
Advocacy Initiatives
A Metrics Study
Lori Melançon
Senior Director, Corporate Affairs
Onyx Pharmaceuticals
4. Measuring Impact in the Evolving World
of Nonprofit Organizations (NPOs)
Increased pressure to measure impact
Fierce competition for funding
Measuring success a daunting challenge
5. Progress in Cancer Study Overview
Objectives
• Evaluate how diverse NPOs in cancer measure impact against goals to
assess trends and best practices
Rx4good Methodology
Primary Research: Group Self-Assessment
▪ 24 NPOs participated in self-assessment of metrics best practices via 8-item
Survey Monkey questionnaire
Secondary Research: Assessment of Websites and Annual Reports
▪ 45-question survey evaluated how 35 NPOs report on diverse measures of
progress against goals
1
2
6. Study Sample
Websites and annual reports of 35 NPOs assessed:
Self-assessments completed by 24 (69%) of the 35 organizations
7. Group Demographics
The majority of the 35 assessed organizations have been
operating for 20+ years with annual revenue of $1M–$5M
Age of Organization Annual Revenue
<$1M 20% (n=7)
$1M - $5M 31% (n=11)
$5M – $10M 23% (n=8)
$10M - $30M 11% (n=4)
>$30M 11% (n=4)
N/A 3% (n=1)
20+ Years 49% (n=17)
10-20 Years 43% (n=15)
5-10 Years 6% (n=2)
1-5 Years 2% (n=1)
9. Survey Revealed NPO’s Perspective on
Metrics Reporting…
75% of NPOs have
changed their view
about metrics during
last three years
• increased
understanding of
value of metrics
• funder
expectations
96%
Metrics are essential for
evaluating success
against goals
33%
The work we do is
difficult to measure
25%
Metrics are important
but we don’t have the
resources to
measure our
programs
8%
We would rather
invest in
programs than in
measuring results
10. Key Obstacles to Measuring Impact
Lack of Resources Determining Outcomes Change
We have limited time, staff and funding to
do all the things we want to do…And it's hard
to increase staff size without
an increase in funding.
Measuring impact takes so much staff
and time that the programs do not get
the attention they deserve.
It is the wrong way around.
Not all of our programs are easy to
track, especially grassroots and
advocacy initiatives.
It can be can be a challenge to determine
the best way to measure impact. It can also
be challenging to determine if behavior
has really changed due to a program
or an event.
“
“
“ “
“ “
“ “
11. NPOs Measure Impact in Many Ways,
Led by Progress Achieved Against Goals
96%
92%
92%
92%
79%
75%
75%
67%
67%
54%
Progress Achieved Against Goals
Numbers Reached, Attended Events
Actions Taken by Stakeholders as a Result of
Programs
Social Media or Web Engagement
Media Coverage
Funding Received
Perceptions Changed or
Reinforced
Sustainability of Programs or Initiatives
Message Delivery
Funding Given
Q: What are the ways in which you measure your impact now? (Check all that apply)
12. Metrics Ranked Most and Least Valued by
NPOs Mirror Those Attributed to Funders
Q: Please rank the following metrics options in order of importance to your
ORGANIZATION/FUNDERS, with one being the most important, 10 least important.
NPO Both Funders
Numbers Reached,
Attended Events
Actions Taken By
Stakeholders as a Result
of Programs
Progress Achieved
Against Goals
Perceptions Changed or
Reinforced
NPO Both Funders
Funding Received
Funding Given
Message Delivery
Media Coverage
Sustainability of Programs
MOST IMPORTANT*
*Ranked 1 or 2 as “most important” by the majority of survey respondents
*Ranked 9 or 10 as “least important” by the majority of survey respondents
LEAST IMPORTANT*
13. Organizations Revealed Best Practices
for Measurement
Ongoing
review of
progress
against goals
Broad
program
evaluation &
follow-up
Digital
reporting
Tracking
research
advances
15. Cancer NPOs Have a Strong
Commitment to Reporting Results…
91%
Publish an annual
report
89%
Report annual revenue &
operating expenses
60%
Clearly state
organizational goals
46%
Report progress against
each goal
16. …and a Strong Commitment to
Transparency
Approximately 80% identify corporate
and individual sources of grants
Majority report funding given and funding
received (65% and 70%, respectively)
Dollar size, number and impact of grants
both given and received are often disclosed
17. Strong Social Media Presence, Yet Less
Often Measured or Reported
97%
97%
91%91%
60%49%
46%Measure impact on progress based on
social media or web engagement
29%Quantify growth of social media engagement
18. Website Research Shed Light on NPO
Priority Metrics
Audience reach/
event attendance
(91%)
Legislative/
policy work
(74%)
Scientific progress
(63%)
Survey patients about
impact of programs
(23%)
Media coverage (43%)
Message delivery (9%)
Website hits (29%)
Unique visitors (31%)
MORE COMMON LESS COMMON
19. Key Takeaways and Study Implications
Metrics increasingly valued by NPOs, driven largely by funder
expectations
▪ Tasked with finding more immediate measures of success
▪ Limited resources and knowledge on how to best convey impact
Variety of reporting techniques; more “outputs” than “outcomes”
▪ Funders and NPOs largely aligned in metrics they deem most/least important
▪ More focused on reporting results than websites/annual reports indicate
NPO demographics have marginal bearing on metrics activities
▪ Effective metrics reporting not necessarily linked to size or revenues
Diverse stakeholders and funding obstacles will continue to challenge metrics reporting,
but best practices and cost-effective strategies exist to showcase impact
21. The Core Challenge As I See It*
“… there is virtually no
credible evidence
that most nonprofit
organizations
actually produce
any social value.”
*“The End of Charity” by David Hunter – Philadelphia Social Innovations Journal
22. The Nonprofit
Marketplace
Hewlett Fdn
THE ROADMAP TO A SOLUTION TO THE CHALLENGES
INSTITUTIONAL
FUNDERS
supply $ &
technical
assistance for
meaningful
information
BENEFICIARIES
are engaged
&
empowered
to provide
meaningful
information
Institutional
Funders use
a set of
standardized
NP reports
of meaningful
information
Beneficiaries
receive
feedback on
how their
meaningful
information
was utilized
23. WHAT IS MEANINGFUL INFORMATION FOR MEASURING
A CHARITY’S SUCCESS?
RESULTS
GOVERNANCE
FINANCIAL
HEALTH
24. How Does This Fit into the Evolution of our
Rating System?
• CN 1.0 – Financial Health
Launched in 2002 with 1,100 charities
• CN 2.0 – Governance
Launched Sept 20, 2011 with 5,500
charities
• CN 3.0 – Results Reporting
Methodology released and data collection
begun Jan 2013, with a goal of 10,000
charities rated by end of 2016
25. Results Reporting Is…
• An assessment of how charities use their results
internally and share them with stakeholders, including
donors
• We are looking to see that you are reporting on results
measures, and showing how your organization learns
and improves based on those measures (i.e. learning and
improving over time is more important than a ‘snapshot’
of results)
26. How Charity Navigator Evaluates Results Reporting
FIVE ELEMENTS:
1. Alignment of Mission, Solicitations
and Resources
2. Results Logic and Measures
3. Validators
4. Constituent Voice
5. Published Evaluation Reports
27. One Element Will Impact Ratings Next Year
Element One:
Alignment of Mission, Solicitations and Resources
28. Element Two:
Results Logic and Measures
• Is the organization’s causal logic, theory of change,
plausible?
• If applicable, is there an indication of how much of the
action is required to produce the pre-defined outputs and
outcomes?
• Is this logic based on reasonable evidence?
• Are there specified measures (indicators) to be collected
and a plan to do so?
29. Element Three: Validators
• Have your charity’s results been vetted by another
organization?
• Not every charity and cause area will have a validator.
That will not diminish your rating.
30. Element Four: Constituent Voice
• How well does your charity collect and publish
feedback from your primary constituents (the
people who are meant to be the direct recipients of
benefits created by the organization’s actions).
• May not apply to every cause area, but will apply
to most.
31. Element Five:
Published Evaluation Reports
• Does your charity publish evaluation reports that cover
the results of its programs at least every five years?
• Are those reports based on recognized techniques to
understand your results?
• Does you charity explain what, if anything, it is changing
as a result of the findings in the evaluation report?
32. WHAT WE HAVE LEARNED SO FAR
Out of Approximately 1,000 Human Service Charities Evaluated So
Far:
• ~ 70% of Charities have a clearly defined program
• ~ 15% of Charities have a clear definition of how much of
an action is required to produce the pre-defined outputs and
outcomes
• ~2% of Charities have at least 1 of the other 3 elements of Results
Reporting (Validators, Constituent Voice or Published Evaluations)
33. A DREAM THAT YOU CAN PLAY A ROLE IN MAKING A REALITY
RESULTS STATEMENTS,
TODAY TOMORROW
?????
35. To Learn More In General About Results Measurement:
Books /Articles/ Studies
1. The Battle for the Soul of the Nonprofit Sector, Berger, Penna and Goldberg, Philadelphia
Social Innovation Journal
2. Money Well Spent by Paul Brest, et. Al
3. Working Hard and Working Well, by David E. K. Hunter
4. Billions of Drops in Millions of Buckets by Goldberg
5. Leap of Reason by Mario Marino
6. The Nonprofit Outcome Toolbox by Dr. Robert Penna
7. Charity Navigator’s webinar on how to use our site
8. Saving Philanthropy Video
9. Independent Sector’s Charting Impact
10. PerformWell Web Site
40. Pharmaceuticals Giving at a Glance, 2013
79%
of all companies run a
corporate foundation
86%
of all companies match
employee gifts
62%
of all companies give
internationally
64%
of all companies make
non-cash gifts
78% of
Pharma
companies
89% of
Pharma
companies
100% of
Pharma
companies
78% of
Pharma
companies
N=261 for full sample. Pharma n=9.
41. A Majority of Companies are Measuring the Societal
Value of their Contributions
18%
35%
47%
Very Experienced
(5 Years or More)
Moderately
Experienced
(3-4 Years)
Slightly Experienced
(2 Years or Less)
76% Measure
Outcomes
and/or
Impacts
N=160. N=119.
24% Do Not
Measure
Outcomes
and/or
Impacts
42. Nearly a Third of Companies Focused On Outcome/Impact
Measurement in a Strategic Philanthropic Program
4
2
• For companies only evaluating grants larger than a specific threshold, the average
threshold was approximately $105,000
• “Other” descriptions include companies that:
• Only evaluate foundation grants
• Evaluate for a combination of strategic programs, grant size, and cause area(s)
• Evaluate for as many grants as possible without any specific direction
Description of Impact Measurement Scope
% of
Companies
All grants, regardless of grant size 23%
Only grants made for a strategic philanthropic program(s) 31%
Only grants larger than a specific threshold 16%
Only grants larger than a specific threshold AND made to a specific cause area(s) 12%
Only grants made to a specific cause area (or cause areas) 8%
Other 10%
43. How much do companies report spending on evaluation?
4
3
48%of companies spent
money in 2013 to
evaluate grants
5%
of 2013 contributions were
dedicated to measuring the
outcomes and/or impacts
associated of their grants
Note: Spending money includes contributions to grantees
(earmarked for evaluation) and payments to consultants,
contractors, and evaluation specialists. N=77.
Note: Median. Includes companies spending money
on evaluation. N=37.
44. Companies Use a Variety of Resources for
Measurement Activities in Strategic Giving Programs
26% 26%
20%
15%
11%
9%
7%
Internal
Resource
(Entirely In-
House)
Consulting
Firm(s)
Internal
Resource
(Informed by
External
Resources)
Universities Publicly
Available
Metrics
Research
Institution(s)
Other
N=105.
45. 66% of Companies Partner with Grantees to Measure the
Outcomes and/or Impacts of their Grants
4
5
% of Strategic Programs Focused on:
Common impact metrics for
Education included:
• Literacy Rates
• Graduation Rates
• College Readiness Scores
• Bachelor’s Degree Attainment
Rates
• Racial Disparity Changes
• Attendance Rates
• Teacher Retention Rates
• # of girls/women supported
• # of people obtaining access to
technology
Common impact metrics for
Health & Social Services
included:
• Emergency Room Visits
• Hospital Recidivism
• Immunizations Administered
• Patient Engagement
• Number of children receiving
immunizations
• Increased farmer yields
• Other donors leveraged for
marginalized community
• Improved health resiliency
Common impact metrics for
Community & Economic
Development included:
• Number of Women Impacted
• Credit Score Changes
• Net Worth Changes
• Number of Loans Administered
• Number of Jobs Created
• Business Growth Metrics
• Business Profit Changes
Education: 41% Health: 26% Econ Dev:16%
46. There’s A Lot Out There To Help!
4
6
PSI – Impact Magazine Urban Institute: PerformWell
Health and Safety Section
Center for Disease Control and
Prevention
Steps Communities
CHANGE Communities
RAND Getting to Outcomes 2004
ISBN 0-8330-35285-2
Nurse-Family Partnership Theory
of Change Logic Model
Foundation Center
Tools and Resources for
Assessing Impact
49. Grants Announcement
• In partnership with a major university specializing in social innovation
and metrics
• Call for entries open late summer
• NPOs with a healthcare focus and a demonstrated commitment to
metrics reporting eligible
• Submit your email address via post-webinar survey for more
information
The Onyx Advancing Innovation in
Measuring Advocacy Outcomes Award
Our research goal was to evaluate how nonprofits in cancer of diverse sizes and missions evaluate their impact against goals. We wanted to glean any trends and best practices that would help the community advance their success in metrics.
The study methodology included primary and secondary research. We targeted 35 groups for both segments of research.
24 groups agreed to participate in the primary research which included an eight item Survey Monkey questionnaire that asked groups their perspective on measuring outcomes.
Secondary research included a review of annual reports and websites of the 35 organizations to assess how the groups communicated their results.
This slide captures the 35 NPOs included in the study sample. As you can see, the study pool was diverse – with groups that focused broadly on cancer to groups that focused on specific types of cancers. Of the organizations whose websites and annual reports were evaluated, 69 percent completed the metrics self-assessment survey.
While the organizations spanned from well-to-newly established organizations with a variety of income levels, the majority have been operating for 20+ years with annual revenue of $1M–$5M.
The research revealed that the majority of NPOs are very committed to metrics, with that commitment increasing during the past few years.
Funder expectations – from foundations, corporations and individuals -- are a primary reason NPOs are more metrics focused.
Although there are barriers to measuring NPO initiatives, the good news is that almost 100 percent believe metrics are essential for evaluating success.
Still, despite this commitment, NPOs note there are challenges, including lack of resources (e.g., time, staff and funding) and uncertainties on how to convey the actual impact of their work.
The study showed that NPOs measured progress in a variety of ways. We were thrilled to see that so many NPOs were assessing actions taken by stakeholders as a result of programs and perceptions changed or reinforced. These are metrics that get at outcomes vs outputs and are the types of metrics that corporate funders such as Onyx are increasingly looking for in the partnerships we have with patient organizations.
We asked the NPOs to rank metrics in order of importance to their organizations and also asked them to rank what they believed their funders found important. For the most part, many of these “priority measures” were aligned. We thought it was interesting that the NPOs ranked sustainability of programs as least important to funders. This could reflect the challenges of gaining funding from year to year from the same funding sources.
NPOs shared some of their strategies for effectively measuring and reporting their impact, including:
Ongoing review of progress against goals: Follow strategic goal setting and evaluation processes (e.g., use of program dashboards, quarterly progress reports); conduct ongoing review and refinement
Broad program evaluation & follow-up: Establish programs on evidence-based research, use consistent evaluation tools so that findings can be aggregated, track audience reach and survey/monitor feedback from program users
Digital reporting: Track all website and social media engagement, secure scheduled web analytics reports and conduct yearly comparisons of numbers
Tracking research advances: Follow-up on accomplishments of research grantees, monitor research/dollars being dedicated to specific disease; patients enrolling in clinical trials and success of research (e.g., follow-on funding)
Now we’re going to turn to the secondary research that we did. This research was our own evaluation of group websites and annual reports so reflects how groups communicate their success.
Transparency is becoming increasingly important and it was good to see that so many NPOs were reporting their funding sources and size of grants.
Though many social media channels were used by NPOs to support their activities, the impact of these social media efforts were less often publicized or measured.
We noticed a distinction between how NPOs reported on their metrics priorities and how they communicated metrics on their websites and in annual reports. Overall, we saw less of an emphasis on outcomes on websites and in annual reports than NPOs stated in their own self-assessment. There could be a variety of reasons for this -- from NPO views that this information in proprietary to time and resource issues. We do think that this may be a missed opportunity because funders – whether corporations or foundations – do look to this information to understand the NPO and its achievements.
