Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Data for improvement
Similar to Data for improvement (20)
Recently uploaded
Recently uploaded (20)
Data for improvement
- 1. DATA FOR IMPROVEMENT Practical strategies to strengthen your impact Presented by Dale A. Blyth and Brandi Olson
- 2. SESSION GOALS: Get clear on why data matters more today Know where you are going with data Use data to learn, adjust, and tell a better story along the way
- 3. Name Organization Relationship status with data No interest in a relationship Never Met Flirting Dating Broken Up It’s Complicated Committed to Making it Work
- 4. GET CLEAR ON WHERE YOU ARE GOING.
- 6. EVIDENCE SUGGESTS WE NEED TO MOVE THE FIELD From a “PROVE it or lose it” approach To an “IMPROVE it to Move it” Approach
- 7. DATA IS USED IN MANY WAYS As a tool for decision making As a tool for accountability As a tool in battle for resources As a tool to tell your story As a tool for improvement
- 8. THREE KINDS OF DATA USE 1. Positioning (make the case) 2. Performance (improve how you do it) 3. Proof (demonstrate your impact) From Charles Smith, Weikert Center
- 9. TODAY We want to focus on PREFORMANCE – Using data as a tool for improvement.
- 10. What is a specific program area that you want to improve?
- 12. What questions do you have about using data to navigate this area of improvement?
- 14. 1. Don’t just admire the problem. • Move from Defining to Analyzing and Understanding the problem • Work to Frame the problem in new ways that suggest strategies
- 15. 2. But also don’t just leap into action • Restrain the “just do it” approach • Sometimes doing something is not better than doing nothing • Reflect and Respond – neither is sufficient by itself
- 16. 3. Do set measurable goals. • Name specific pieces you can work on • Could set goals around • Aspects of participation • Youth engagement • Quality components • Skills and outcomes desired
- 17. 4. Do improve intentionality of your practice. • Intentional goals • Intentional strategies • Intentional implementation • Intentional measurement • Intentional reflection • Intentional adjustments
- 18. 5. Create a culture with a dynamic improvement approach. • One that recognizes and uses multiple types of data • One that includes expertise and experience • One that engages young people as co-creators
- 20. TOWARD A FORMULA FOR UNDERSTANDING YOUTH PROGRAM IMPACT RI = A x P x D x Q x YE Where RI is Real Impact Experienced A is Access to Program P is Participation D is program Design Q is Quality of Practice as Delivered YE is Youth’s Engagement in Program
- 21. 3 - Strengthen DESIGN 1 – Increase ACCESS 4 - Improve QUALITY 5 - Enhance YOUTH ENGAGEMENT YOUTH OUTCOMES 6 Upgrade YOUTH WORKER EXPERTISE SIX PATHS TO INCREASING THE IMPACT OF YOUTH PROGRAMS 2 - Increase PARTICIPATION
- 23. LIKE GPS, data can help you pinpoint your location.
- 24. LIKE A COMPASS, data can help identify where you are going.
- 25. LIKE A MAP, data can help you navigate around obstacles.
- 26. Use data from all of your tools to navigate on your journey.
- 27. And sometimes your most meaningful data source comes your own observations
- 29. 1. Look at the data and make sense of it (what is the story?)
- 30. 2. Ask questions (find the mystery)
- 31. 3. Dig deeper (look for clues)
- 32. 4. Adjust the plan (change the storyline)
- 33. SCI-FI AFTERSCHOOL • 11 sites • 750 4th-8th grade participants • Goal: youth in our programs will identify an adult they can trust
- 34. Is there an adult here who… No Mostly No Mostly Yes Yes Is interested in what you think about? 6.05% 45 6.45% 48 35.62% 265 51.88% 386 You can talk to when you are upset? 7.8% 58 8.06% 60 29.84% 222 54.3% 404 Helps you when you have a problem? 4.6% 34 7.8% 58 33.3% 248 54.3% 404 You will listen to and respect 2% 15 4.17% 31 25.8% 192 68% 506 Supportive Adults-SAYO
- 35. Youth overwhelmingly have a trusted adult at SciFi Afterschool. Is there an adult here who you will listen to and respect? Is there an adult here who helps you when you have a problem? Is there an adult here you can talk to when you are upset? Is there an adult here who is interested in what you think about?
- 36. Some youth are not connected to a caring adult. Is there an adult here who you will listen to and respect? Is there an adult here who helps you when you have a problem? Is there an adult here you can talk to when you are upset? Is there an adult here who is interested in what you think about?
- 37. WHAT IS THE STORY HERE? 1. Look at the data and make sense of it (what is the story?) 2. Ask questions (find the mystery) 3. Dig deeper (look for clues) 4. Adjust the plan (change the storyline)
- 38. Youth overwhelmingly have a trusted adult at SciFi Afterschool. Is there an adult here who you will listen to and respect? Is there an adult here who helps you when you have a problem? Is there an adult here you can talk to when you are upset? Is there an adult here who is interested in what you think about?
- 39. PILSEN YOUTH CENTER • Neighborhood afterschool program • 65 elementary youth • Goal: reduce the number of fights between youth
- 41. Total fights in a 2 week period
- 42. 35 fights 40 30 20 16 12 Total fights in a 2 week period Conflict resolution instruction starts March April May Instruction time changes
- 43. QUESTIONS?
- 45. RESOURCES http://www.brandiolsonconsulting.com Slide deck available on website Logic Models, Data-Resources, Impact + Measurement Newsletter http://www.extension.umn.edu/youth/co ntact/dale-blyth/ blyth004@umn.edu
Editor's Notes
- Welcome
- Brandi
- Brandi Introductions-Dale and Brandi start Everyone else: name, organization, relationship status: never met, flirting, dating, no interest in ever being in a relationship, broken up, it’s complicated, committed to making it work for the long haul Make a graph of responses
- Dale Current landscape in youth development…why does data matter so much? Used to look at this through the eyes of artists and photographers but increasingly we also expect to be able to look at this through the eyes of data (pair of Google glasses) >> assessing the chance of rain >> calculating the best route to take >> assessing the best place to find minerals or water >> discovering where others have found the best photos We can now geocode, time code, compare and contrast in ways never before available This growth in the availability of data is growing exponentially and the expectations that data will exist and be useful in taking action are also going up dramatically The same is true of the social landscape No longer enough to just address outcomes – increasingly we need to have data on access, quality, participation, engagement, program outcomes and longer term impact. This leads to a critical shift we and other practice fields need to continue to make. In short, we are increasingly in an
- Why do we need this shift and how will it help. Evidence –based programs often do not go to scale well Implementation Science and Improvement sciences are showing the many factors that affect impact of a program – and most of these have to be improved over time if there is going to be sustained, measureable impact In a field as diverse and multi-dimensional as Afterschool youth work – we need to increase quality and intentionality not just fidelity This requires data, data integration or alignment (acrooss programs as well as across types of data within programs (#M Process)
- Dale Explain how current trends in the field demand/push different kinds of data
- Dale Think about question, share answer with someone nearby. Share out a few examples. What do you want to improve in your program? How do you know what improvement looks like? How do you currently measure/understand improvement?
- Brandi How Dale and Brandi think about data (it’s a tool for navigating): Once I had a group of college freshmen girls on a 2 week backpacking trip in the upper peninsula of michigan. They took turns navigating each day, using a map and compass. On this particular day mid-way through the trip, we were hiking to a remote lake, our destination for the night. It was so remote that we knew of no trails to get us there, so we were bushwacking through the woods—exactly what it sounds like, using only our compass as a guide. About 3 hours into the day, the girls stumbled upon a trail…compared to the thick woods that they had been navigating, it was lovely—wide, clear of debris, but it wasn’t on our map. They were so excited to have found a trail that the leaders immediately started hiking down it. Knowing that the lake was directly east of where we were, one of the navigation leaders voiced her concern a few minutes in that the lake was east, and this trail seemed to be going straight south. The other leader, who was a little more gregarious, insisted that the group keep hiking just a little ways, that maybe it would turn east soon. Everyone else, happy to be out of the woods eagerly followed. For almost 1 mile, we hiked down this trail, the leaders never once looking at their compass. When the trail clearly started turning again—a look at the compass confirmed—they had been hiking for almost a mile—in the wrong direction. They knew clearly where they were going, but they weren’t clear enough on how to get there. When opportunity came along to take a different route, they were easily pulled in a different direction, and didn’t use the tools available—their map and compass. They wasted a lot of time and energy running down the wrong trail. It’s why maps, and the tools to navigate them are so important! We think using data in your organization is a lot like using a compass on the trail…it’s a tool to make sure you don’t get lost, or head in the wrong direction. Another way to think about data is in terms of the story that you want to tell about your impact…(Dale)
- Dale Share questions out loud with the room (Brandi can write down questions as people respond)
- Dale
- Dale The “do something, anything!” approach is often more harmful than doing nothing.
- Dale
- Dale Increase intentionality around program design and experiences
- Dale And make sure you know how to use it in combination with your expertise and experience. Big goal today is to help you think about how to use data…for navigation. GPS systems Can be annoying as they re-calculate your route but they are doing data integration Can lead you to ignore other data – from your spouse or son who lives in the area and knows how to get there
- You aren’t the pioneers in this field… If you are planning a trip, you will rely on your previous experience, but also talk to others who have taken similar trips.
- This framework is based on a broad interpretation of the research literature many of you have so richly contributed to over the years.
- We used to primarily think of adding programs, and what we have learned is that adding a program isn’t the only way to improve impact. This model identifies 5 possible pathways to assuring greater actual program impact Strengthen program design/features used Increase youth’s access Improve the quality of practice Enhance youths’ engagement Upgrade youth worker’s expertise Each can use data, both existing and new, to increase impact It assumes that some of these pathways can impact more than one factor (e.g., improving quality can increase youth engagement or increasing a youth worker’s abiltity can increase one or more factors
- Brandi Quickly tell GPS story in the Upper Peninsula… Dale and I want you to not just have data, but know how to use it in really meaningful ways.
- Brandi Challenge: you don’t even know where you are on the map. You have a general idea of where you are going—like California, but you aren’t exactly sure what highway you are on as you wander through Montana. EXAMPLE: Need a quick organizational + data use example WHERE ARE YOU AT WITH RESPECT TO CONSISTENT QUALITY OF WHAT YOU CURRENTLY DO?
- Brandi Challenge: refer back to original hiking story. We were going south, but we needed to be going north. EXAMPLE: Non-profit that connects high school girls with summer internship opportunities in STEM field, with goal of increasing women in STEM industries. A few years ago, a large corporate funder came along and offered a substantial grant, if the program would also include boys in the program and broaden their focus to increasing racial diversity in the field. The org agreed, but made a commitment that 70% of their internship spots would still go to young women. They tracked their participation data closely. Over the course of 2 internship cycles, they found that they were receiving nearly twice as many applications from boys as from girls and were having a difficult time meeting their goal of
- Brandi
- Sometimes you need a map, sometimes, a gps, sometimes a compass
- A map and compass are only as good as the expertise of the user…if the map says you should find a dry creek to cross, but what you see is a raging river…what you see is always right.
- Let’s get practical…how does all of this work in real programs and how will it work in your program? Data for improvement is about engaging in a continuous improvement process. Strategies for how to check and adjust
- When you have an epic backpacking trip, you have a great story to tell, twists and turns and unexpected change of events along the way…it makes for a great story. Thinking about how to write a continuous improvement story is a helpful framework for using data to make improvements.
- When you have an epic backpacking trip, you have a great story to tell, twists and turns and unexpected change of events along the way…it makes for a great story. Thinking about how to write a continuous improvement story is a helpful framework for using data to make improvements. Ask questions (find the mystery) Dig deeper (look for clues) Adjust your plan (change the story line)
- When you have an epic backpacking trip, you have a great story to tell, twists and turns and unexpected change of events along the way…it makes for a great story. Thinking about how to write a continuous improvement story is a helpful framework for using data to make improvements. Ask questions (find the mystery) Dig deeper (look for clues) Adjust your plan (change the story line)
- When you have an epic backpacking trip, you have a great story to tell, twists and turns and unexpected change of events along the way…it makes for a great story. Thinking about how to write a continuous improvement story is a helpful framework for using data to make improvements. Ask questions (find the mystery) Dig deeper (look for clues) Adjust your plan (change the story line)
- Case study- explain background info
- Brandi Sci Fi After school uses the Survey of Afterschool Youth Outcomes (SAYO) to measure their outcomes related to supportive relationships with adults and youth’s sense of competence in science and math. This is their data. Trying to find the story here is difficult, but it gets easier if we look at the data differently.
- Brandi Looking at the same data in this graph…what story is the data telling? What questions are raised? (is there a mystery?) This is a great way to look at data for showing impact…it’s a great storytelling picture. However, it in looking for areas to improve, you might want to focus on the data in a different way.
- Brandi In groups of 3-4 people who are sitting near you, spend a few minutes looking at this graph. Use the handout: Look at the data and make sense of it (what is the story?) Ask questions (find the mystery) Dig deeper (look for clues) Adjust the plan (change the storyline) (spend 10 minutes on this discussion) Ask participants to share their responses, especially spending time on the questions that they generated.
- When you have an epic backpacking trip, you have a great story to tell, twists and turns and unexpected change of events along the way…it makes for a great story. Thinking about how to write a continuous improvement story is a helpful framework for using data to make improvements.
- Brandi Go back to green slides…what if the data showed that you are doing well in 3 of 4 areas, but the 4th area is weak
- An afterschool program started thinking more intentionally about the ways of relating that youth engage in, and realized that a lot of conflict between youth results in fights during the program. They set a program goal that youth would use conflict resolution skills to resolve conflict peacefully. They identified a conflict resolution process and defined the component skills that were involved. Program staff set aside specific time each week to intentionally teach conflict resolution skills, and they wanted to understand if their strategy was effective. They decided to start tracking the number of fights that occurred during gym time. They reasoned that if the conflict resolution skills instruction was working, they would start to see a decrease in the number of physical and verbal fights. The staff team came up with a concrete definition of what counted as a fight—what it had to look like and sound like (verbal name calling, physical contact, interruption to activity and required adult intervention), and a simple recording system (a checkbox chart that the gym staff kept) They agreed to track this information and look at it every other week for 3 months.
- Dale? This is the record keeping system that they used
- The program started their new conflict resolution instruction at the beginning of March. Working in the same small groups, discuss this data using your handouts… Look at the data and make sense of it (what is the story?) Ask questions (find the mystery) Dig deeper (look for clues) Adjust the plan (change the storyline) (10 minutes)
- During the first month of implementing the plan, the total number of gym fights increased. The staff explored why this was happening. They realized that the conflict resolution learning activities were taking place as the first program activity of the day when attendance is lowest, as youth trickle in during the first hour of the program. They readjusted the schedule and put the conflict resolution activities later in the afternoon when the majority of youth were present. After the scheduling change, they started to see a steady decline in the number of fights happening during gym time. Because nothing changed in the structure of the gym activities, they reasoned that the number of opportunities for conflict during gym remained constant, yet the number of physical and verbal fights was decreasing. The staff concluded that the approach was having a positive impact on youth’s conflict resolution skills, and they continued to implement it through the rest of the year.
- Dale and Brandi What questions do you have? If not many questions, end with reflective question: What will your next step be from here?
- Data is a tool for improvement of impact. When working with youth, the journey is never really over and the story is never really finished…it’s never ending. Data can help you better understand where you are, how you are going to get there, and keep you from getting lost… Sometimes data reveals the storyline, sometimes it allows you to change the storyline… The most interesting stories and journeys are the ones that have unexpected twists and turns…
- What other resources should we include? Do we want to link to SAYO?