The final report (presented to the foundation board) of an evaluation of the Ball Foundation's partnership with Rowland Unified School District, prepared by Catherine Awsumb Nelson, Ph.D., independent evaluator.

1. The Ball-RUSD Partnership
Final Evaluation Report
September 23, 2011
Catherine Awsumb Nelson, Ph.D.
1

2. Key Questions
I. What did the partnership look like?
a) Design
b) Participation
c) Quality
II. What impacts did it have?
a) Non-participants
b) Personal transfer of participants
c) Broader impacts
III. What does Ball leave behind?
IV. What is the potential for sustainability?
V. What was learned about investing in district
transformation?
2

3. 2010-11 Data Collection
DATA SOURCE n Details
Individual interviews 22 Executive Cabinet (5), SIL design team (2), CoP2 (2), CoPs (7),
IC (6)
SIL case studies 5 schools At each of 5 schools, fall and spring interviews with principal,
30 new team member, and returning team member, plus
interviews document analysis
Event observations 9 SIL (3), Literacy Network (3), IC (1), Efficacious Instruction
workgroup (1), Sensing team (1)
Event feedback 10 SIL (5), Literacy Network (4), IC (1)
surveys
Milestone 8 Design teams from LN, SIL, and IC, full Executive
monitoring meetings Cabinet
Middle and end of year
Staff survey 364 131 partnership participants (58% response)
233 non-participants (35% response)
Student survey 4949 Overall response rate 56% for Grades 4-12
Participant tracker
3 895 Certificated staff only

4. Ball approach to district transformation
 Transform, not reform
 Focus is on systems change
 Assumption that most necessary expertise is already
in the district
 Key strategies:
 Capacity building
 Coherence making
4

5. Why is “coherence” so important?
 “I hope we are strong enough  “The „here‟s another binder‟
to keep it going. I worry about mentality is what we fight here
it. Not a reflection on the work all the time. So many things
Ball has done but the they want you to know and
district…we do amazing things learn about. We dip our toes
in this district, but we tend to into so many things and they all
dabble. You have to keep blur together. I don‟t think we
things alive yourself if they are need to try so many things at
working for you because the once. With kids, when you
district moves on. We do a lot bombard them with stuff, it
of good things but don‟t stick doesn‟t work. When you teach
with anything long enough to for depth, spend the quality
make it great.” (CoP member) time on a unit, that is when
learning happens. (SIL
member)
Fewer than 10% of respondents agreed with the statement “Once we start a
new program in this district we follow up to make sure it is working.”
5

6. Critical Features of the Rowland
Partnership
 Ball provided structures and time for inquiry and
collaboration around instruction at three levels
 Classroom
 School
 District
 Collaboration supported through design and
coaching
 Focus on changing adult learning to change student
learning
 Working with the willing
6

7. Flipping the adage
The old adage is time is money.
In education, money buys time, which creates opportunity for learning. That is
the great gift Ball has given Rowland, along with design and coaching that help
the professional time align with what we know about what creates impact in the
classroom.
7

8. U.S. Schools Lag International Competitors
in Providing Professional Learning Time
 The United States is far behind in providing public school
teachers with the kind of high intensity, job-embedded
collaborative learning that research shows is most effective in
changing practice and improving learning
 U.S. teachers report little professional collaboration in
designing curriculum and sharing practices, and the
collaboration that occurs tends to be weak and not focused on
strengthening teaching and learning.
 Compared to other nations that outperform the United States
on international assessments, American teachers spend
much more time teaching students and have significantly less
time to plan and learn together.
“Professional Learning in the Learning Profession:
A Status Report on Teacher Development in the U.S. and Abroad.”
(National Staff Development Council, 2009)
8

9. Building Instructional Capacity in RUSD
Structure Function Ball supports #
Communitie Small groups of educators •$ for professional books 120-
s of Practice collaborating around a and training 150
specific literacy topic •Literacy Network Days
•Cluster days
•CoP Garden
•Developmental
framework
School Teams of 6-8 •Design and facilitation for 110-
Instructional administrators and monthly cross-district 150
Leaders teachers tasked with meetings
leading professional •Learning Walks
learning at each site
Instructional Representatives across •Meeting design and 25+
Cabinet district and role groups facilitation
charged with identifying •Support for workgroups
and supporting district wide
instructional priorities
9
Executive Existing district leadership •Coaching 5

10. District Context
 Budget cuts
 Accountability pressure
 Demographic shifts
 Hollowing out of district instructional support capacity
10

11. Partnership Participation,
2009-2011
Both
years, 16.8%
2010
only, 8.9%
Never, 65.9%
2011
only, 8.4%
11

12. Intermediate and secondary participation
increased
40%
35%
35%
% of classified employees participating in
33%
2010 2011
30% 28%
25%
Partnership
21%
20% 19%
15% 14%
10%
5%
0%
Elementary Intermediate High
12

13. Quality of the Work
13

14. Ratings for Quality of Professional Learning
remarkably stable
% of participants rating partnership professional
2011 2010
learning as “Good “ of “Excellent”
Ensuring that all voices are heard 86% 86%
Having a positive impact on student learning 78% 77%
Making it safe to raise difficult issues 78% 74%
Building on existing professional expertise within the
district 78% 76%
Focusing on issues directly relevant to my practice 76% 75%
Being grounded in data and/or other evidence of
student learning 71% 67%
Striking a good balance between content and process 69% 71%
Using time well 57% 58%
14

15. Measures of overall intervention quality increased across the
board, with the biggest increase in potential for sustainability
Ball Partnership work has helped RUSD move
57%
different parts of the system toward a common
71%
focus.
District leaders have demonstrated that they are 61%
committed to the Partnership work. 61%
Ball Partnership work is well integrated into the 33% 2010
day-to-day work of the district. 45% Participants
Ball Partnership activities help us deeply examine 69%
our approach to teaching and learning. 78%
Working with Ball helped RUSD respond more
49%
strategically to Program Improvement status and
59%
budget cuts than we otherwise would have.
Important aspects of the work will continue once
42%
Ball personnel and funds are no longer in the
65%
district.
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% agree/strongly agree
15

16. Impacts on non-participants
16

17. All avenues for non-participant awareness increased this
year, with informal communication about CoP work growing the
most
Heard about Ball Partnership activity in a 46%
staff meeting 56%
A colleague talked to me about work he/she 42%
was doing with Ball 47%
I heard informally about the work a teacher 22%
Community of Practice was doing 39%
Participated in a meeting led by my school's 31%
School Instructional Leadership team 36%
Read about Ball Partnership activity on the 24%
district website or in printed materials 33%
Participated in a Learning Walk @ my site led 25%
by my school's School Instructional… 29%
I was told about the work of the Instructional 13%
Cabinet 21% 2010
2011
Work from a teacher Community of Practice 18%
group was shared in a staff development… 20%
I haven't heard anything about the 10%
Partnership this year 6%
0% 10% 20% 30% 40% 50% 60%
17 % of non-participants who heard about partnership through…

18. 40-50% of non-participants noticed
improvements in site meetings
100%
% of non-participants saying site meetings over the last
90%
80% Much more so More so
70%
two years have changed
60%
50%
14% 15% 20%
40% 14% 12%
12% 13%
30%
20%
35% 38% 40% 36%
29% 30% 32%
10%
0%
Driven by Inquiry Uses Build Build cultural Focused on Opportunities
what based evidence of professional proficiency teaching and for
teachers student learning learning collaboration
want/need to learning community
learn
18

19. Participant Impacts
19

20. Reported levels of personal transfer are
unchanged from last year
100%
Revolutionized my practice Major transfer Moderate transfer
% of participants reporting level of transfer
90%
80%
70%
36% 34%
60% 40% 38% 33%
50%
38%
40%
30%
37% 37% 34%
31% 32%
20% 26%
10%
9% 9% 9% 9% 9% 7%
0%
My classroom How I prepare How I support How I work How I structure How I connect
instruction for and reflect classroom with other staff adult learning my work to
on my instruction broader district
classroom priorities
instruction
20

21. % of participants rating impact major/moderate
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
21
Collaboration in
43%
43%
professional learning
Quality of professional
41%
40%
learning
CoPs as a model of
44%
38%
collaborative inquiry
Norms of reflective
practice/rigorous
41%
dialogue 34%
Opportunities for teacher
46%
33%
leadership
Coherent vision of
48%
32%
effective instruction
Use of
effective/research-based
49%
30%
practices for instruction
SIL teams developing
41%
26%
site-level coherence
Improved decision-
making processes in the
34%
20%
district
Major
District better structured
Moderate
to support effective
Participant ratings of broader district impacts
47%
16%
instruction
IC developing district-
level coherence in
44%
14%
instruction

22. For all impact areas with a direct comparison from
last year, ratings of Major impact increased
2011-
Participants
rating impact Increase from
"Major" 2010
Increased collaboration in professional learning 43% +17%
Increased quality of professional learning 40% +12%
Establishing norms of reflective practice and rigorous
dialogue about instruction 34% +17%
Use of effective/research-based practices for
instruction 30% +13%
Improved decision-making processes in the district 20% +10%
District better structured to support effective instruction
22 16% +6%

23. District instructional support capacity
Staff are held accountable for realizing the
39%
district's vision of quality instruction
District priorities are clearly focused on
35%
supporting and improving instruction
The district has a coherent vision of quality
34%
instruction
District decisions are grounded in data 30%
Key resources of time, money, and personnel
28%
are clearly connected to instructional…
It is clear where and how decisions about
20%
instruction get made
District-level decisions are made with adequate
13%
input from school-based personnel
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% of participants who agree/strongly agree
23

24. Capacity Building Structures
24

25. CoPs took off in 2nd half of year
100%
96%
94%
93%
90% 89%
88%
% of members saying their CoP had done "Mostly" or
82%
81%
80%
76%
70% 71%
69%
67%
60% Used a cycle of plan/act/reflect
58%
50%
"Completely"
43% Jointly examined artifacts of student
40% learning
36%
30% Put in place ways to capture our
learning
20%
18%
Shared learning with colleagues not
10% in our CoP
0%
January March June
25

26. Communities of Practice participant ratings
of rigor up sharply over last year
We routinely used Process Learning Circles and/or 2011
66%
specific conversational processes like ordered 2010
59%
sharing
All of our members stayed engaged and 77%
accountable to each other 63%
We pushed each other to be rigorous about what
81%
works and why in the literacy practice we were
57%
focusing on
We routinely looked at evidence of student work
81%
from our own classrooms as we talked about how
54%
well a specific practice worked
We routinely agreed to try specific things in our
90%
classroom and then discuss with the group how
76%
they worked
My CoP had a clear question or purpose to focus 90%
our inquiry 84%
0% 10% %20%CoP members agree/strongly agree 90% 100%
of 30% 40% 50% 60% 70% 80%
26

27. Greatest SIL accomplishments were in team development;
whole staff learning also significantly impacted
Learned to function well as a team 48% 43%
Built our own competencies as instructional
23% 63%
leaders
Made staff meeting time more focused on
36% 42%
learning
Made staff meeting time more collaborative and
29% 48%
inquiry based
Built shared understanding among our team
16% 57% Completely
about what efficacious instruction looks like
Mostly
Become seen by all staff as leaders of learning in
19% 48%
the school
Used cultural proficiency as a lens to analyze
13% 49%
instruction in our school
Built shared understanding among the whole staff
12% 44%
about what efficacious instruction looks like
Used learning walks to reflect on instruction in our
27% 29%
school
27 0% of members rating extent to which their70% accomplished goal (May SI
% 10% 20% 30% 40% 50% 60% team 80% 90% 100%

28. SIL impact ratings up across the board this
year
Change
from last
% of SIL participants who agree/strongly agree 2011 year
Because of the SIL work, people across the district are starting to use
more similar language about instruction 65% 26%
The SIL work has significantly influenced our site-level professional
development approach and agenda 57% 11%
Our SIL team had enough representation to effect change in our site 49% 17%
Expectations for implementing the SIL work at our site were clear 43% -3%
Our SIL team will be a driving force in our school's instructional
improvement efforts going forward 56% 11%
SIL has given teachers more of a leadership role over instruction in
this school 51% 8%
We made progress this year in making instruction more public in this
school 59% 11%
28

29. Instructional Capacity assessments of their
own effectiveness vary widely across goals
Launching a system wide pilot of the new data… 95%
Building our own understanding of RTI (within IC) 91%
Determining training needed for new data system 74%
Setting a direction for RTI district wide 67%
Setting district wide instructional priorities 63%
Understanding IC's role 55%
Monitoring the implementation of the PI plan 35%
Supporting staff in implementing instructional… 31%
Increasing district wide awareness of IC 23%
Monitoring the effectiveness of PI plan… 16%
0%
20%
40%
60%
80%
100%
% of IC rating group group “Effective” or “Extremely Effective”
29

30. Challenges to district-level coherence-making:
Imbalance between mandate and resources
Capacity
Mandate:
Build instructional
coherence through
developing and
Lines of
communication
supporting Authority
instructional
priorities
30

31. Summary of Impacts:
What does Ball leave behind?
31

32. Areas of most and least change this year
Greatest progress Least progress
 Rigor/depth of learning  Cross strand
for partnership connections
participants
 New structure for district
 Quality of professional
coherence continued to
learning for ALL district
staff struggle
 Concrete agreements  Staff assessment of
around Efficacious district instructional
Instruction capacity
 District  Levels of personal
ownership, confidence transfer stayed flat
in sustainability
32

33. Summary
of Impacts
 Idea of design
 Expectation that decisions will be
collaborative and inclusive
Transitions with
 Expectation that professional learning
momentum
will be collaborative
 CoPs as a vehicle for teacher directed
inquiry into practice
 Learning Walks starting to de-privatize
practice in some sites
 SIL starting to re-shape site-based
learning
33

34. “No going back”
 “One of the most striking places I saw the impact
was when we had this presenter from county on EL
issues. The way they presented was just so foreign
from how we do things- it showed how far we have
come. It was just, throw up a power point and then
we will take your questions- boom. Instead of taking
a piece and really working it the way we do now.
(Principal)
 “I think many of us have passed the point of no
return this year. We don‟t want to go back. There is
no way we are going back to professional
development that is not collaborative and self-
initiated.” (Teacher)
34

35. Summary
of
 Haven or silo? Impacts
 Struggle to balance accountability with
Challenges to the emerging
“learning as a journey”
 Divergent conceptions of assessment
 Traditional conceptions of teacher
system
“leadership”
 Search for plug-in solution still evident in
some areas
35

36. What does Ball leave behind?
 Morale maintained during difficult time
 Cuts made with more intentionality
 Capacity for the design and facilitation of adult
learning (in a much broader base of staff)
 District owns new structures for learning and
leadership
 Norms about adult learning
 Agreements about efficacious instruction
36

37. Looking Forward:
Sustainability Potential
37

38. Majority of participants are optimistic that
most impacts will be lasting
Collaboration in professional learning 65%
Use of effective/research-based practices for… 62%
Norms of reflective practice/rigorous dialogue 59%
Coherent vision of effective instruction 59%
Quality of professional learning 58%
Opportunities for teacher leadership 56%
CoPs as a model of collaborative inquiry 54%
SIL teams developing site-level coherence 53%
IC developing district-level coherence in… 44%
District better structured to support effective… 41%
Improved decision-making processes in the… 33%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% of participants saying impact will "Definitely" or "Probably" last
38

39. Large increase in confidence that impacts
are sustainable
Change in
Sustainability
Rating
Communities of Practice as a model of collaborative
professional inquiry +36%
Increased collaboration in professional learning +35%
School Instructional Leadership (SIL) teams as a means to
develop site-level coherence in instruction +34%
Increased quality of professional learning +31%
Improved decision-making processes in the district +14%
The Instructional Cabinet as a means to develop district-
level coherence in instruction -1%
39

40. Moving Forward in RUSD:
Efficacious Instruction
40

41. RUSD Learning
STUDENTS: Paradigm
principles
Learner centered
ADULTS: Teaching for Linked paradigm shifts in
Learning for Effective adult and student
Effective Learning: learning
Teaching Democratic Both grounded in brain-
Relationships mind principles
Inquiry
Clarity
Collaboration Process of strands is
Invested
Data Cognition Learning for Effective
Ownership Feedback Teaching
Expert learners Content for strands is
Teaching for Effective
Learning
41

43. Process for creating and enacting the
framework embodies “capacity”
 Bottom up
 Incorporated research and practitioner knowledge
 Back and forth between the strands
 “Not a thing”
 Ongoing opportunities for meaning-making vs.
“Rollout”
43

44. Baseline findings about Efficacious
Instruction in RUSD
 Teacher clarity is the strongest domain, relationships
and engagement are the weakest
 No significant differences in how Hispanic students
experience instruction
 Quality of instruction as experienced by students
drops slowly from 4th to 8th grade, bottoms out in 9th,
then climbs again until 12th
 Compared to students, teachers overestimate the
quality of relationships, underestimate quality of
feedback
44

45. What was learned about district
transformation?
 Slower than reform
 People want to “thingify”
 Monitor the balance between ownership and
coherence
 Changing power relationships at the top is hardest
 Broadening the teacher role isn‟t easy either
 People do need to be taught skills of collaboration
and inquiry
 Capacity and buy-in are easier to build in the
process of doing authentic work
45