1. Tools for Measuring
Place-Based Systems Change
Christina Garcia, The James Irvine Foundation
Justin Piff, Equal Measure
May 31, 2017
Grantmakers for Effective Organizations
Learning Conference
2. Goals
Participants will:
1. Learn about the “systems mindset” needed to evaluate
(and learn from!) systems change
2. Understand frameworks for evaluating systems change
3. Develop a plan for applying learning at home
3. "As someone whose job it
is to address the end
result, how can you not
care about the
beginning?"
-Dan Barber
Chef and author of The Third Plate
4. Introductions
• Have you invested in place-based initiatives?
• Have you invested in systems change initiatives?
• Are you currently evaluating (or have you
evaluated) these efforts?
5. The Systems Change Umbrella
Collective Impact
Comprehensive Community Initiatives
Cross-Sector Partnerships
Community Coalitions
Community Development
Community Partnerships
6. A Systems Change Mindset
I want change and I
want it now!
(But yesterday would be nice!)
Tackling and evaluating systems change
requires the right mindset!
7. A Systems Change Mindset
What traits or characteristics do
funders and evaluators of
systems change need?
8. A Systems Change Mindset
Conditions for successful systems change evaluation
• Orientation towards learning
• Comfort with “process” and iteration
• Cultural competency and community responsiveness
9. A Systems Change Mindset: What’s the difference?
Just prove our strategy works. How can what we’re seeing improve
what we do?
Don’t deviate from the workplan. That question’s not important anymore,
let’s adjust our approach.
Let’s ask the board of directors how
things are going.
Let’s ask the students if they’re seeing a
difference.
I want to see client outcomes
at scale.
This work is complex, what are some
signs that systems are changing?
The “norm” Systems thinker
11. How do we know when a system has changed?
Events
Patterns
Systemic Structures
Better
Best
Good
We seek
changes
at
all three levels
12. Systems affecting postsecondary success
Events
Patterns
Systemic Structures
Low postsecondary attainment rates
Low rates among:
Students from select school districts
Students of color
Students who attend community college
First-generation college students
Discriminatory disciplinary practices
Assumption of “college going
knowledge”
Outdated/misaligned curricula or
course sequencing
13. Small group exercise
Identify an “event” you’re trying to change:
• Employment
• Academic outcomes
• Health outcomes
• Civic engagement
What patterns/trends do you see among these events?
• Population
• Location
• Timing
What “systemic structures” are contributing to these patterns?
• Individual or organizational relationships
• Policies or practices
• Incentives
• Power structures
• Communication/Information flow
16. Key Question: What does evidence of systemic change look like?
Observable indicators of embedded, sustainable changes to
systemic structures:
• Behaviors
• Relationships
• Policies or Practices
• Flow of information/communication
• Resource allocation
Capturing changes among systemic structures
Or else they won’t “stick”
17. Guidelines for data collection
• Qualitative: Use interviews, focus groups, visuals, and other qualitative
methods when you aren’t sure what the changes look like, or when you
want to understand diverse and nuanced experiences with the system.
• Quantitative: Capture, count, and track evidence – or signs – of systems
changes when you know what you’re looking for.
Capturing changes among systemic structures
18. REGIONAL HUBS OF EXCELLENCE
Area players are connected to do —
and gain — more together.
PARTNER
COMMUNITY
PARTNER
PARTNER
COMMUNITY
19. Understand the formation and development of Linked Learning regional hubs to support learning
Identify best practices, strategies, and lessons that can inform program strategy and
development of field standards
Provide support and insights to the James Irvine Foundation, Jobs for the Future, and Linked
Learning hubs to support the success of the initiative
Multi-case study, mixed-methods approach
DE
Learning-
oriented
Adaptable
Responsive
Linked Leaning Regional Hubs: Developmental Evaluation
20. Leadership & Partnership
Development
• Shared vision
• Partnership
composition
• Governance
• Partner roles and
activities
• Relationship-building
• Commitment-building
• Ways of working
together
• Understanding of
regional issues
Systems Change
• Impact of regional
partnerships on 1)
deepening, 2) scaling,
3) aligning, and 4)
adopting Linked
Learning across
sectors
• Strategies and policy
levers advancing
Linked Learning
• Areas that hub-
initiated practice and
policy changes offer
the greatest potential
for large-scale
changes in student
experiences
Interacting Factors
• Context of each region
in shaping design,
implementation, and
effectiveness of hubs
• Landscape of
education and
workforce partners
influencing the work of
the hubs
• Alignment with
parallel initiatives
and/or pre-existing
partnerships
• Role of investment
and technical
assistance
Linked Leaning Regional Hubs: Developmental Evaluation
21. Aspen Opportunity Youth Incentive Fund
21 communities across the US focused on improving outcomes for “opportunity youth”
Key Question: “How does implementing a collective impact approach contribute to
systemic shifts in communities that improve educational, work, and life outcomes for
opportunity youth?”
Annual partnership-level survey
• 9 “systemic shifts” that demonstrate the system is changing to better support opportunity youth
• Evidence of “systemic shifts” based on partner behaviors
A series of site interviews and visits
22. Quantifying Systems Change for the Aspen Opportunity Youth
Incentive Fund
More effective integration of programs and
organizations into pathways serving opportunity
youth
What changes among the system do
you expect to see?
What would tell you this has
happened?
Organizations implement shared intake forms
Organizations use the same data systems to track youth participation across programs
Staff from different organizations train one another
Staff from organizations of different sectors train one another
Organizations use the same assessment system to assess youth service needs
23. Quantifying Systems Change
57%
65%
40%
65% 67%
48%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Community 1 Community 2 Community 3
Year 1 Year 2
Effective integration of programs and organizations into pathways serving
opportunity youth
24. National Evaluation of StriveTogether
About 20 cradle-to-career partnerships from across the US
Key Question: “How, and to what extent, have communities across the StriveTogether
Cradle to Career Network built their civic infrastructure?”
83 Questions capturing partner perceptions of civic infrastructure in 9 areas
• 5-point scale
• Multiple perspectives re: experience in the community
• What we would expect to see based on StriveTogether’s Theory of Action
More than 2,500 surveys completed over 2 years
25. StriveTogether: Measuring approaches to systems change
70%
65% 63% 64%
60% 59%
55% 53%
48%
2016 2015
Shared Vision Data Use Partnership Partnership Community
Capacity Structure Engagement
Partner Partnership Partnership Practice
Commitment Action Communication Change
26. Common Pitfalls in Assessing Systems Change
Thinking you found the silver bullet
Naval gazing
Celebrating event-level changes while not changing systems
Forgetting to look at patterns – changing overall event
numbers may not address disparate outcomes for key groups
27. Small group exercise
Identify one or two “systemic structures” you want to see change
What evidence would you see that these structures have changed?
• Behaviors
• Relationships
• Polices or Practices
• Communication/Flow of information
• Allocation of resources
• Changes in knowledge or awareness
Whose perspective should inform whether the system is changing?
28. Investing in and evaluating systems change requires a different
mindset
Evaluations should focus on learning and understanding
Monitor events, patterns, and systemic structures
Stakeholders will help define and understand systems change
In Summary
29. GEO’s Systems Grantmaking Resource Guide: http://systems.geofunders.org/
Introduction to Systems Thinking, Daniel H. Kim: https://thesystemsthinker.com/wp-
content/uploads/2016/03/Introduction-to-Systems-Thinking-IMS013Epk.pdf
Three Keys to Unlocking Systems-level Change:
https://ssir.org/articles/entry/three_keys_to_unlocking_systems_level_change
Resources and Tools
Notes de l'éditeur
Justin
Justin (funder asked me to put this in)
Justin
This quote is from an unlikely source and a great example of systems thinking and how “place matters” How and where we grow and cultivate our food, including the crops we surround it with, can literally make it taste better.
Justin
Christina
Christina
Provide examples of what each looks like from a grantmaking and evaluation perspective.
Christina to lead, Justin to take notes on flip chart
Use flip chart to have folks identify the traits and briefly discuss/explain.
Participants can offer suggestions for evaluators, funders, or both.
Christina
Briefly explain each of these as they would apply to both the funder and the evaluator.
Christina
Walk through the different sets of statements, pointing out how the 2nd statement of each set exhibits the systems mindset necessary for systems change evaluation.
Justin
Here’s a framework often used to understand systems at the root of the issues we aim to address.
We borrowed this from Daniel Kim’s “Introduction to Systems Thinking” b/c it provides a useful framework for unpacking how to evaluate the kind of systems change many of us are interested in.
Events, What we see – these are the kinds of problems you’re trying to address (employment, homelessness, kindergarden readiness, etc.).
Patterns, What our data reveal – We engage in systems change because the system isn’t working for some subpopulation/subset of people. Here, we need to unpack the “events” more to understand what patterns we’re dealing with (e.g., people of a certain status, age, socie-economic standing, georgraphy/neighborhood).
Systemic structures – these are hardest to see, but these are what we really want to change when we talk about “systems change” – these are the unseen dynamics creating patters and ultimately exhibiting as the “events” we see. Things like ………………………………………..
Justin
Once we’ve identified the systemic structures at play, we need to determine how to assess and understand them.
Changes at all three levels are needed. Most philanthropies focus on the “events,” but we need to make sure that systemic structures are changing to ensure we’re not just applying a “band-aid” solution. But, if we only focus on “systemic structures,” we miss out on understanding whether these changes are actually having an impact. We’re going to talk more about how to evaluate changes among systemic structures.
Justin
Walk through example.
Justin
8 mins.
Work on, on your own. (10 minutes)
Walk participants through the exercise and pass out worksheets.
Using Handout: Pick an “event” you’re addressing. Identify the patterns you’ve observed and systemic structures influencing these. Get input from all participants in the group – there are no wrong answers. Worksheet will provide additional probes/discussion questions for the group to consider about the nature of systemic structures.
Have a couple folks report out depending on time.
Christina…..introduce concept of including diverse stakeholder perspectives, including program/initiative beneficiaries, then direct folks back to tables to share and get input.
Following up on exercise, have folks get additional input from table mates re: possible systemic structures affecting the event. (10 minutes)
Have a few folks from the larger group report out to the whole group (5 minutes)
Large Group Discussion: Reflecting on the exercise, ask folks to share the value of having multiple participants in their discussions? Did each bring a unique perspective to understanding the “issue?” We often engage in systems change because we think discrete, clear/cut strategies won’t work. As such, we need to understand the “system” from a variety of perspectives, including those influencing and influenced by the system. Each brings a different understanding of the issues at play, and each perspective is needed to change “the system.”
Ask participants for examples of the types of people/stakeholders they would include in understanding how the system is functioning?
Justin
Tee up next section.
Now that we can identify systemic structures, we need to determine how to best understand/assess changes.
We will provide a few examples of how we’ve approached this.
Justin
5 minutes
Need to operationalize “systems change” by articulating the evidence we would see of a changed system. And, we also need some confidence that these changes will “stick” – observing one new behavior one time won’t cut it. So, we look for changes that are embedded and sustainable by nature – these have the power to affect “events” at scale.
Note that most “events” have multiple systemic structures driving them. Measuring systems change needs to take these multiple factors into account (which is also why it’s so important to engage diverse stakeholders in identifying systemic structures).
Justin
Christina
Developmental evaluation approach:
Is effective for initiatives that are innovative, in the early stages of development, and occur in complex shifting environments
Focuses on learning, rather than accountability
Is flexible and responsive
Adapts learning questions and methods as the initiative evolves
Identifies and shares emerging insights in a timely way
Multi-case study approach
Focuses on understanding each hub in its own context
Focuses on identifying cross-cutting themes, strategies, and lessons
Methods
Interviews and discussions with anchors, key regional partners, JFF, and Irvine
Site visits
Anchor and key partner survey
Field observations
Document review
Other methods, as needed (e.g., interviews and discussions with field experts, social network analysis, and spatial analysis)
Christina
Animate one column at a time left to right.
Georgia
Justin
Justin
Here’s how we quantify based on indicators of systems change to understand initial states and progress.
Bridget
JP
Describe what we did here.
Two year’s of data – remarkably consistent trends.
Will hone in more on four aspects of this process.
Justin (Christina can editorialize)
Naval gazing – a constant focus on systems change can leave you spinning your wheels – focusing so much on systems that you forget to look at impact.
Celebrating event-level changes while not changing systems -- Need to make sure that it’s changes among systemic structures leading to changes we want, not band-aid or “one offs”
Forgetting to look at patterns – changing overall event numbers may not address disparate outcomes for key groups – Critical to make sure that we look at patterns – our overall numers on things like employment, high school graduation, etc. may be improving, but are they improving equitably. It’s often underrepresented groups most affected by dysfunctional systems, so need to maintain an eye on who’s benefiting (and who’s not) from our strategies/interventions.
Thinking you found the silver bullet – systems are complex – there likely is no silver bullet.
Justin
Small group exercise to have participants identify the “indicators” of systems change based on their work during the first exercise
Using Handout -- Have folks consider how to create evaluation and measurement approaches capturing both events and systems changes and relationship between changes in events and changes in the systems or increased clarity about those systemic issues.
Depending on time, have a couple folks share what they came up with. Maybe one example.
May also ask: What approach would you use to capture these changes?