John Cronin's presentation at the December 2015 TASA assessment summit

1. Welcome!

2. What research says about community-based
accountability.
John Cronin, Ph.D.
Senior Director of Education Research
NWEA

3. Damn
little Thank you very much.

4. What I hope to accomplish.
• Define community-based accountability.
• Suggest that a “management letter”
accompany your accountability report.
• Describe three types of metrics your
report should always include.
• Offer observations based on my review
of available community-based
accountability reports.

5. What is accountability?
To be “answerable” means that accountability is a
dialogue between the stakeholders and the leaders of
their schools. Part of that dialogue is understanding the
goals and objectives of the community. Part of that
dialogue is discussing how your schools are doing in
reaching those goals.

6. What is accountability?
To be “answerable” means that accountability is a
dialogue between the stakeholders and the leaders of
their schools. One part of that dialogue is understanding
the goals and objectives of the community. Another part
of that dialogue is discussing how your schools are doing
in reaching those goals.
The most important part of the dialogue is what you’re
doing to improve performance based on this
information, which is leading.

7. What conditions are required for
community-based accountability to work?
• Agreement on who the community is and your
constituencies.
• Agreement on what the school is answerable for.
• Transparency in regard to how the school reports
performance on these objectives.
• Strategy to improve performance, and corrective
action when strategy fails.

8. The management letter…
Is an opportunity to show leadership.
It is a conversation with your stakeholders about the
performance of the school system. In it you explain
your successes and failures, and discuss your key
goals and strategies for improving the educational
experience of students.
Warren Buffet’s annual report to the Berkshire
Hathaway shareholders provides a great model for
content and tone.

9. The management letter is…
Your narrative. It is the centerpiece of the
accountability report
The report on indicators is…
Like financials in an annual report. They
demonstrate transparency and, if well
presented, allow readers to make their own
judgments

10. The management letter…
Is not simply for the community…
It requires the district’s leader(s) to think
seriously about your work and communicate
coherently about the state of the district and
your strategy. It forces you to reflect and clarify
your thinking.

11. Our bad news from 2014 comes from (Burlington-Northern Santa Fe) and is
unrelated to earnings. During the year, BNSF disappointed many of its
customers. These shippers depend on us, and service failures can badly hurt
their businesses. (transparency about failure)
BNSF is, by far, Berkshire’s most important non-insurance subsidiary and, to
improve its performance, we will spend $6 billion on plant and equipment in
2015. That sum is nearly 50% more than any other railroad has spent in a
single year and is a truly extraordinary amount, whether compared to
revenues, earnings or depreciation charges. (our strategy to address failure)
An illustration of good tone from the Berkshire
Hathaway Management Letter

12. Though weather, which was particularly severe last year, will always cause
railroads a variety of operating problems, our responsibility is to do whatever
it takes to restore our service to industry-leading levels. That can’t be done
overnight: The extensive work required to increase system capacity sometimes
disrupts operations while it is underway. (recognizing responsibility and
tempering expectations)
Recently, however, our outsized expenditures are beginning to show results.
During the last three months, BNSF’s performance metrics have materially
improved from last year’s figures. (reporting progress)
From the Berkshire Hathaway Management
Letter

13. In a year in which most equity managers found it impossible to
outperform the S&P 500, both Todd Combs and Ted Weschler (Berkshire
Managers) handily did so. Each now runs a portfolio exceeding $7 billion.
They’ve earned it.
I must again confess that their investments outperformed mine. (Charlie
says I should add “by a lot.”) If such humiliating comparisons continue, I’ll
have no choice but to cease talking about them.
Todd and Ted have also created significant value for you in several matters
unrelated to their portfolio activities. Their contributions are just
beginning: Both men have Berkshire blood in their veins.
From the Berkshire Hathaway Management
Letter

14. You know from news reports that the costs of poor performance on the state
pension fund have passed to schools. This has stressed our budget and one
consequence was that we froze hiring at the high schools. Because of
retirements and transfers at some of the high schools, we did not replace AP
teachers who retired or transferred and those courses were dropped.
That was necessary for a year, but unacceptable for the long-term. This year
our principals reviewed teacher assignments and are reinstituting courses in
areas where we have a teacher with appropriate credentials. Because those
teachers will be dealing with new content and larger class sizes, we’re funding
summer training for them in both the AP content and in teaching strategies
that better enable adaption of instruction with larger student groups.
We can’t promise that this will entirely solve the problem, and our pass rate
on AP exams may take a hit while our new AP teachers master their
assignment. I’ll keep you apprised in my monthly newsletter as to how this is
going.
How this would read in a community-based
accountability management letter

15. Observations from the review of community-
based accountability reports

16. Accomplishments!
•The reports are data rich!
•They are reflect a broader picture of schools than
standardized tests and school report cards.
•They report on aspects of education that the community
values.
•They provide measurements of performance.

17. Comments from review of four Community-Based Accountability Reports
1.They are very detailed and somewhat overwhelming to process.
2.They put more focus on “achievements” and not much on “improvement”.
1. Is the business “profitable”? Is the school “performing”?
2. Is the business “improving”? Are the metrics headed getting better year
over year?
3. Is the business “accelerating”? Is the rate of improvement increasing or
decreasing?
3.Some focus more on “scoring schools” (which is what you were trying to avoid)
then informing.
4.They generally don’t address questions of equity.
5.They generally don’t identify or address weaknesses in schools.
6.They generally don’t contain meaningful narrative.
7.They rationale underlying indicators is sometimes unclear (why did the district
choose to measure the number of classrooms which use non-traditional delivery
methods and why should the community care)
8.They often lack “leading indicators” that predict possible outcomes at the high
school level. (e.g. – what are the signals of drop-out risk in fifth grade, how many
students show those signals, and what is being done about them?)
9.They lack any mention of accountability. This business underperformed and here’s
what we’ve done.

18. The number of indicators in the reports is
overwhelming
Suggestion – The management letter should bring
focus. Make sure every indicator reflects an
outcome that the community values.
The scorecard section of the report can be an
appendix.

19. The reports focus more on “scoring” schools
than “informing” stakeholders.
Suggestion – Establish goals rather than categories
and report whether schools have “achieved the
goal”, and whether they are “improving”, and
“accelerating”.

20. Exemplary (3
points)
Recognized (2
points)
Acceptable (1
point)
Unacceptable
(0 points)
Percentage of
students
reading at or
above grade
level in third
grade
100-90% 80-89% 70-79%
X
Less than 70%
From a community-based accountability
report

21. Our first grade reading rate continues to be below goal and we were
alarmed by the declines in reading that we were seeing in 2009
through 2013. Two years ago, we brought in a part-time specialist to
help at-risk readers and teachers added more guided reading
practice to the schedule. The last two years show we’ve improved,
and that improvement is accelerating thanks to the hard work
kindergarten and first grade teachers. To get closer to goal this year,
we’re paying for teachers to take additional staff development and
offering release time for teachers to observe and coach each other
as they implement new practices.
How this could be communicated in a
management letter?

22. The reports are heavy on reporting status
and light on reporting trends
Suggestion – Report your key indicators
longitudinally and focus on improvement and
acceleration/deceleration in those indicators.

26. Grade Students Athletes % Fine Arts %
9 3276 1487 50% 2256 76%
10 3190 1215 41% 1656 56%
11 2967 932 31% 1233 42%
12 2795 8703 24% 1028 35%
Percent of students participating in athletics
and fine arts programs

27. The reports generally do not address issues
related to equity.
Suggestion – Identify and include metrics in your
report that show how you are addressing the
particular needs of at-risk and minority populations
and how you are addressing achievement gaps.

28. Calculus AB 2012 2013
Enroll
ed
% Taking % 3+ Enrolled % Taking % 3+
All students 347 257 74% 389 300 77%
Enrollment and student performance in AP
Calculus AB

29. Calculus AB 2012 2013
Enroll
ed
% Taking % 3+ Enrolled % Taking % 3+
All students 347 257 74% 389 300 77%
Minority
students
100 40 70% 130 50 70%
Comment –
Our goal has to improve both enrollment and minority
participation in our AP program and our schools have
accomplished that. We’re concerned that while minority
participation increased, the proportion of minority students
actually testing declined this past year. Our principals reported
that cost of the exams has been a barrier for some students, so
we’ve committed to paying the cost of the exam for all
students identified as eligible for free and reduced lunch.
Enrollment and student performance in AP
Calculus AB

30. The reports report satisfaction but show little
interest in the dissatisfied.
Suggestion – Include follow-up questions to
illuminate who and what may contribute to
dissatisfaction and discuss how you are addressing
those issues.

31. Very
Satisfied
Somewhat
Satisfied
Dissatisfied Unsure
2014 30% 55% 9% 3%
Survey Question –
How satisfied are you with the education
that the school system provides your child?

32. Very
Satisfied
Somewhat
Satisfied
Dissatisfied Unsure
2014 30% 55% 9% 3%
Under 30k 12% 60% 22% 6%
30-70k 20% 62% 15% 3%
Above 70k 40% 50% 6% 4%
Survey Question –
How satisfied are you with the education
that the school system provides your child?

33. Top five sources of dissatisfaction – high schools
parents who responded somewhat or dissatisfied
1.Fees for athletic and extracurricular participation
(15%)
2.Lack of communication/responsiveness by some
teachers (12%)
3.Cutbacks in AP course availability (8%)
4.Poor teaching (6%)
5.Elimination of jazz choir program (4%)

34. The reports lack leading indicators that are
predictive of success at the next level.
Example – The report will include dropout rates, but
lacks reporting on elementary and middle school
indicators that are empirically tied to this metric and
would be key to prevention.

35. The reports show little evidence that a poor
result leads to some decision to change it.

36. Florida
District
Highly
Effective
Effective Needs
Improvement
Developing Unsatisfactory VA Score Florida
Ranking
1 44.4% 55.6% 0.0% 0.0% 0.0% 0.39 109
2 25.0% 75.0% 0.0% 0.0% 0.0% 0.37 121
3 90.9% 9.1% 0.0% 0.0% 0.0% -0.14 2802
4 60.7% 39.3% 0.0% 0.0% 0.0% -0.14 2797
5 81.2% 18.8% 0.0% 0.0% 0.0% -0.16 2831
6 37.3% 54.2% 1.7% 0.0% 6.8% 0.12 880
7 81.3% 18.8% 0.0% 0.0% 0.0% 0.22 402
8 41.7% 55.6% 1.4% 1.4% 0.0% -0.34 3274
9 52.2% 47.8% 0.0% 0.0% 0.0% 0.16 664
10 27.0% 66.2% 1.4% 0.0% 5.4% 0 1764
11 7.1% 72.6% 9.5% 10.7% 0.0% -0.08 2445
Teacher Evaluation Ratings in Eleven Florida
Schools 2013

37. Thank you!
E-mail: john.Cronin@nwea.org
Organization Website: www.nwea.org
Contact: