Presentation to NYCOSS superintendents summit

1. Accountability through engagement
with families and community
John Cronin, Ph.D. – Vice President of Education Research - NWEA
Presentation is available at
https://www.slideshare.net/secret/jstTMvwWCcfwl6

2. Community-Centric Accountability
A shift in focus to focusing on the success
of students in partnership with their
families and their community.

3. What is accountability?
Accountability is a dialogue between the
stakeholders and the leaders of their schools.
1 Part of that dialogue is understanding and aligning
the goals and objectives of the parents, the schools,
and the larger community.2
Another part of that dialogue is discussing how your
schools are doing in reaching those goals.3 The most important part of the dialogue is what
you’re doing to improve performance based on
this information, which is leading.4

4. 01 02 03 04
Four Principles That Guide Communications
Understand
what they
want to
know
Be
transparent
Share your
strategy and
corrective
action when
strategy fails
Identify
your
audience

5. 01 02 03 04
How to be transparent without getting killed
Have a
strategy to
address it
Accept
responsibility
and temper
expectations
Demonstrate
progress
Admit
failure

6. Everyone is entitled to their
opinion, but not to their own
facts.
—Daniel Patrick Moynihan

7. Use facts to present new perspectives

8. $0 $20,000 $40,000 $60,000 $80,000 $100,000 $120,000
Associates Degree
Law Enforcement
Education
Psychology
Computer Science
Medicine and Nursing
Agriculture
Mathematics
Physics and Chemistry
Philosophy
Architecture
Business
Engineering
Liberal Arts
Journalism
Biology
Communications
Social Sciences
Economics
Visual and Performing Arts
History
English
Students degree choice by parent income
Source: Kim Weeden, National Center for Educational Statistics
Students from high income families choose career
oriented majors less often.

9. Source – U.S. Department of Labor, National Longitudinal Survey 1979-2010
Graph is reference from Money Matters, National Public Radio, March 19, 2014
What careers
contribute to
upward and
downward
mobility?

10. Career skill
Proportion of
careers
studied Average Salary
Analytical Skills 16% $ 77,415
Leadership 10% $ 64,772
Critical Thinking Skills 8% $ 62,423
Creativity or Resourcefulness 14% $ 62,004
Problem Solving & Troubleshooting 24% $ 56,475
Time Management 8% $ 55,808
Communication 53% $ 50,110
Detail Oriented 35% $ 48,652
Writing 8% $ 45,935
Math 20% $ 44,447
Interpersonal Skills 39% $ 43,542
Organizational Skills 20% $ 40,875
Physical Stamina 53% $ 33,294
Customer Service Skills 31% $ 31,631
Technical or Mechanical Skills 12% $ 27,802
Average salary by skill requirement

11. College major by SAT score
Psychology
English
Engineering
Business
Visual/Perf Arts
Undecided
Social Sciences
Security
Physical
Sciences
Other
Mathematics
Liberal Arts
Legal StudiesJournalism
History
Health
Professions
Eng Technicians
Education
Computer
Science
Biological
Sciences
Architecture
440
460
480
500
520
540
560
580
600
620
440 480 520 560 600
MeanMathematicsSAT
Mean Reading SAT
Bubble size indicates relative
number of students intending
to pursue a this major

12. The ECO intervention

13. The experiment
The experiment was conducted with 25,000
students who scored in the top 10% on ACT
or SAT and whose families fell in the bottom
third of the income distribution.
The experimental group received packets
telling them how to identify schools that
matched their interests, school graduation
rates and costs, and instructions for
applying for fee waivers. Parents received a
letter of introduction and materials to
support their student in selecting a college.
The control group received nothing.
The ECO Intervention
Hoxby, C., & Turner, S. (2013). Informing students about their college
options: A Proposal for Broadening the Expanding College
Opportunities Project. Washington, DC. The Brookings Institute.

14. Results
Students were more likely to apply and enroll to a
“peer” institution with higher graduation rates, student
aid, and higher student related spending.
The intervention cost $6 per student.
The ECO intervention

15. All parents agree that…
✚ Students should develop good study skills and learning habits
✚ Students should develop strong critical thinking skills
✚ Students should develop strong verbal and written communication skills
✚ Schools should offer a strong core curriculum in reading and mathematics
✚ Schools should emphasize science, technology, engineering and math
✚ Schools should offer a strong education in life skills
15
Zeehandelaar, D. and Winkler, A. (2013, August) What parents want: Educational Preferences and
Trade-offs. Thomas B. Fordham Institute

16. Categories of parents
✚ Pragmatists – vocational emphasis, hands-on learning, development of personal talent, accommodation for
struggling students – less emphasis on college attendance
✚ Jeffersonians/Traditionalists – citizenship emphasis, learning a strong code of moral conduct – less emphasis on
high academic standards
✚ Test Score Hawks (minority)- High test scores/preparation for college – less emphasis on life skills and character
development
✚ Multiculturalists (politically liberal)– Emphasis on interacting with people of diverse backgrounds, development of
social skills - less emphasis on preparation for college
✚ Expressionists (charter school) - Emphasis on art/music, development of personal talent, love of learning – less
emphasis on life skills, high academic standards
✚ Strivers (higher income) - Emphasis on college readiness, advanced coursework, high academic standards – less
emphasis on social skills, vocational programming
16
Zeehandelaar, D. and Winkler, A. (2013, August) What parents want: Educational Preferences and
Trade-offs. Thomas B. Fordham Institute

17. Proportion of parent archetypes
Pragmatists, 36%
Jeffersonians,
24%
Test Score
Hawks, 23%
Multiculturalists,
22%
Expressionsists, 15%
Strivers,
12%

18. Which archetypes (if any) best describe
the parents of your school system?
How can understanding your parents
archetypes improve your communication?

19. What kind of data do parents want?
95%
95%
93%
92%
91%
90%
88%
84%
79%
77%
0% 20% 40% 60% 80% 100%
Monitoring general progress
Knowing when to be concerned
Determining readiness for next grade
Knowing if I need to seek extra help
Monitoring standards
Communicating with teacher/admin
Measuring high quality teaching
Helping my child with homework
Adjust content to student needs
Providing activities for home
Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions
Study. Survey conducted by Grunwald Associates LLC

20. Questions parents want answered from
assessment
⎼ Core question – Do you know and care for my
child?
⎼ What are my child’s strengths and weaknesses?
⎼ Is my child on track for the next grade?
⎼ Is my child on track for college?
⎼ Is my child showing improvement?
⎼ Should I be concerned?

21. The reports focus more on “scoring” schools
than “informing” stakeholders.
Suggestion – Establish goals rather than
categories and report whether schools have
“achieved the goal”, and whether they are
“improving”, and “accelerating”.

22. Observations from the review of data in
community-based accountability reports

23. Four types of metrics:
Achievement – How are students learning
Growth – How much progress do students
make within a year
Improvement – Are we improving
achievement and/or rate of growth of
students over time
Acceleration – Is the rate of change
accelerating, decelerating, going negative

24. 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2011 2012 2013 2014 2015 2016 2017 2018
Percentage of students qualified for Algebra II placement
Achievement in math for 2017
shows slightly more than half of our
8th graders qualified for HS algebra II
placement.

25. Your annual report should be your “almanac”
when it comes to student achievement facts.
Any member of your leadership team who
represents educational data should be
thoroughly familiar with it.
In any data discussion, the discussion should
begin with agreement on what the facts actually
are.

26. 35% 37% 41%
46%
52% 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2011 2012 2013 2014 2015 2016 2017 2018
Percentage of students qualified for Algebra II placement
Achievement in math showed an improving
pattern through 2015 and improvement actually
accelerated during that time period.

27. 35%
37%
41%
46%
52% 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2011 2012 2013 2014 2015 2016 2017 2018
Percentage of students qualified for Algebra II placement
Achievement in math showed an improving
pattern through 2015 and improvement actually
accelerated during that time period.

28. 35% 37%
41%
46%
52% 53% 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2011 2012 2013 2014 2015 2016 2017 2018
Percentage of students qualified for Algebra II placement
Achievement in math improved but decelerated
between 2015 and 2017. This might be a leading
indicator of trouble ahead.

29. 35% 37%
41%
46%
52% 53% 54%
47%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013 2014 2015 2016
Percentage of students qualified for Algebra II placement
Achievement in math declined in
2018.

30. • How many of your teachers are respectful towards
you?
• If you walked into class upset, how many of your
teachers would be concerned?
• If you came back to visit class three years from now,
how many of your teachers would be excited to see
you?
• When your teachers ask how you are doing, how
many of them are really interested in your answer?
• How many of your teachers would you be excited to
have again the future?
Teacher-Student Relationships
Panorama SEL survey

31. 0%
5%
10%
15%
20%
25%
30%
35%
40%
None of my teachers A few of my teachers Half of my teachers Most of my teachers All of my teachers
If you walked into class upset, how many of your teachers
would be concerned?
Wilson District

32. 0%
5%
10%
15%
20%
25%
30%
35%
40%
None of my teachers A few of my teachers Half of my teachers Most of my teachers All of my teachers
Teacher-student relationships
October December

33. Teacher-student relationships over time
2.2
2.5
3
3.7
3
3.4
3.7
3.9
October December February May
2017 2018
Tell a partner how you would interpret this data to a group
of parents.

34. Exemplary (3
points)
Recognized
(2 points)
Acceptable
(1 point)
Unacceptable
(0 points)
Percentage of
students
reading at or
above grade
level in third
grade
100-90% 80-89% 70-79%
X
Less than 70%
From a community-based accountability
report

35. 50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
2009 2010 2011 2012 2013 2014 2015
Percentage of students at or above
grade level at the end of 1st grade
Goal Franklin School
Our first grade reading rate continues to be below goal and
we were alarmed by the declines in reading that we were
seeing in 2009 through 2013. Two years ago, we brought in a
part-time specialist to help at-risk readers and teachers added
more guided reading practice to the schedule. The last two
years show we’ve improved, and that improvement is
accelerating thanks to the hard work kindergarten and first
grade teachers. To get closer to goal this year, we’re offering
teachers additional staff development and offering release time
for teachers to observe and coach each other as they
implement new practices.
How this could be communicated in a
management letter?

36. Show multiple perspectives

37. Expanded Eligibility Standard Eligibility
Average
Grade
Mean AP
test score
Percent 4 or
greater
Average
Grade
Mean AP
Test
Score
Percent 4 or
greater
2015 3.87
2016 3.20 3.78
2017 3.15 3.67
2018 3.05 3.60
Results of Expanded Eligibility for Advanced
Placement History at Canyon High School
Many times, one data point may not communicate an
accurate impression of how a program is working

38. Expanded Eligibility Standard Eligibility
Average
Grade
Mean AP
test score
Percent 4 or
greater
Average
Grade
Mean AP
Test
Score
Percent 4 or
greater
2015 3.87 4.2 80%
2016 3.20 3.6 3.78 4.1
2017 3.15 3.7 3.67 4.3
2018 3.05 3.9 3.60 4.3
Results of Expanded Eligibility for Advanced
Placement History at St. Ignatius High School
Many times, one data point may not communicate an
accurate impression of how a program is working

39. Expanded Eligibility Standard Eligibility
Average
Grade
Mean AP
test score
Percent 4 or
greater
Average
Grade
Mean AP
Test
Score
Percent 4 or
greater
2015 3.87 4.2 80%
2016 3.20 3.6 55% 3.78 4.1 82%
2017 3.15 3.7 60% 3.67 4.3 85%
2018 3.05 3.9 70% 3.60 4.3 85%
Results of Expanded Eligibility for Advanced
Placement History at Canyon High School
Many times, one data point may not communicate an
accurate impression of how a program is working

40. Expanded Eligibility Standard Eligibility
Average
Grade
Mean AP
test score
Percent 4 or
greater
Average
Grade
Mean AP
Test
Score
Percent 4 or
greater
2015 3.87 4.2 80%
2016 3.20 3.6 55% 3.78 4.1 82%
2017 3.15 3.7 60% 3.67 4.3 85%
2018 3.05 3.9 70% 3.60 4.3 85%
Results of Expanded Eligibility for Advanced
Placement US History at Canyon High School
In 2016 we expanded eligibility for our Advance Placement History
courses in an effort to make college credit accessible to more of our
students. The results show that the students who qualified through
expanded eligibility did not perform as well as those who qualified
using the prior criteria.
In truth, I’m ecstatic about the results, which showed a clear majority
and increasing proportions of the students who qualified under the
new criteria scoring 4 on the test and achieved college credit.
Expanding the number of students in AP benefited many students
who otherwise wouldn’t have had the opportunity and did not
diminish the performance of the other students taking the program.
All credit to Joe Hernandez and Merideth Johnson who took a big
risk in agreeing to expand the program and did an amazing job of
assuring the success of this effort.

41. Expanded Eligibility Standard Eligibility
Average
Grade
Mean AP
test score
Percent 4 or
greater
Average
Grade
Mean AP
Test
Score
Percent 4 or
greater
2015 3.87 4.2 80%
2016 3.20 3.6 55% 3.78 4.1 82%
2017 3.15 3.7 60% 3.67 4.3 85%
2018 3.05 3.9 70% 3.60 4.3 85%
Results of Expanded Eligibility for Advance
Placement US History at Canyon High School
By the way, Joe and Merideth were surprised that grades in their
course were lower, while success on the test improved. They’ve
decided to take a second look at their grading practices to see if
they should be adjusted.

42. You learn more from the dissatisfied than
satisfied.

43. Allies Bedfellows
Opponents Adversaries
Trust
Agreement
Peter Block’s framework
for assessing relationships

44. Very
Satisfied
Somewha
t Satisfied
Dissatisfi
ed
Unsure
2014 30% 55% 9% 3%
Survey Question –
How satisfied are you with the education that the
school system provides your child?

45. Very
Satisfied
Somewha
t Satisfied
Dissatisfi
ed
Unsure
2014 30% 55% 9% 3%
Under 30k 12% 60% 22% 6%
30-70k 20% 62% 15% 3%
Above 70k 40% 50% 6% 4%
Survey Question –
How satisfied are you with the education that the
school system provides your child?

46. Top five sources of dissatisfaction – high
schools parents who responded somewhat or
dissatisfied
1. Fees for athletic and extracurricular
participation (15%)
2. Lack of communication/responsiveness by
some teachers (12%)
3. Cutbacks in AP course availability (8%)
4. Poor teaching (6%)
5. Elimination of jazz choir program (4%)

47. Suggestion – Include follow-up questions to
illuminate who and what may contribute to
dissatisfaction and discuss how you are
addressing those issues.

48. The choice of metrics matters. Poorly
chosen metrics can distort reality.

49. Which school achieved the best growth in math?
School Mean years of growth
M.L. King Elementary 1.34
Hamilton Middle School 1.46
Armstrong High School 2.14

50. 0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
AverageScaleScoreGrowthbySchool
Grade in School
Average Scale Score Growth in Math by School
The bars represent the median growth for the NWEA™ norm
group by school.
When statisticians refer to a “year of growth,” the median scale
score growth is the common definition of a year. As you can see,
what constitutes a year of growth by that definition varies
immensely by grade.

51. 0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
AverageScaleScoreGrowthbySchool
Grade in School
Average Scale Score Growth in Math by School
The lines represent the standard deviation of growth, by grade,
across the schools in the NWEA norming group.
The standard deviations are relatively even across grades,
which means that a school showing growth that is 1 standard
deviation above other schools achieves roughly 2 to 2.5 scale
score points greater growth than the median school.

52. 0.0
0.5
1.0
1.5
2.0
2.5
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
YearsofGrowth
ScaleScoreGrowth
Average Scale Score Growth in Math by School
The line graph depicts the years of growth associated with a school that
performs 1 standard deviation better than the median school for each
grade.
In grades 1–3, being 1 SD better translates to about 1.3 years of growth. In
grades 8–10, it translates to 1.5 to 2 years of growth.
Given an elementary school and a high school performing at the same
level relative to their peers, the high school will look better if years of
growth are used as the metric.

53. 0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
-5
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
YearsofGrowth
ScaleScoreGrowthinReading Average Scale Score Growth in Reading by School

54. Concluding thoughts

55. Two key vehicles for communication
1
2
The superintendent’s (or principal’s)
letter to stakeholders.
The community accountability
report

56. The principal or superintendent letter
Is an opportunity to show leadership.
It is a conversation with your stakeholders about the
performance of the school system. In it you explain
your business, your successes and failures, and
discuss your intended actions for improving the
educational experience of students.

57. Implications of family-centric
accountability
• Less emphasis on one-size fits all
metrics (proficiency, growth percentiles).
• More emphasis on preparing students for
success in career or college
(academically, socially, life skills) .
• More emphasis on metrics that are
fundamental to a student’s future success.

58. The tone of the letter…
• is optimistic
• is plain yet thoughtful
• is personal and conversational
• is realistic in addressing successes and
transparent about problems
• is effusive in praising of subordinates, critical
of yourself.

59. Identifying and responding to your
audience

60. 01 02 03 04
Four Principles That Guide Communications
Understand
what they
want to
know
Be
transparent
Share your
strategy and
corrective
action when
strategy fails
Identify
your
audience

61. 01 02 03 04
How to be transparent without getting killed
Have a
strategy to
address it
Accept
responsibility
and temper
expectations
Demonstrate
progress
Admit
failure

62. 01 02 03 04
Four ways educators create trust
They believe
stakeholders
are
intelligent
They can
explain their
work
They credit
subordinates
before
themselves
They show
love for
their
mission

63. Eight ways educational leaders create
trust
✚ Love for their mission.
✚ They believe their stakeholders are intelligent
✚ They can explain their work
✚ They are transparent about failure
✚ They have a plan and take action
✚ The accept responsibility and temper expectations
✚ They demonstrate progress
✚ They credit subordinates with their successes.

64. The letter…
is not simply for the community…
It requires you to think seriously about your
work and communicate coherently about
the state of your schools and your strategy.
It forces you to reflect and clarify your
thinking.

65. A simple process
• Involve the community in setting the
vision for schools and your goals
• Use these objectives to drive local
performance.
• Report performance and progress on
those goals annually.

66. One of our commitments is to help more students graduate prepared for
college. Advance Placement is a key piece of this strategy and our goals
were to increase participation in the program, increase the number of
students sitting for AP tests, and improving average scores on AP exams.
Our data show that we are doing well on two of the three. In the past three
years we’ve increased the number of students participating in the program
by 20% across our high schools, and the average scores on the exams
have generally improved. In English for example, the average score
improved from 3.5 to 3.7 and In Calculus AB the average score improved
from 3.1 to 3.5. That’s impressive progress and all credit goes to Marsha
White, who led this initiative and the teachers at Wilson and Jefferson high
schools who made this happen. (crediting others for success)
Unfortunately, the storyline isn’t perfect. While more students are taking
the courses, the improvement in the number of students sitting for the tests
has not kept pace. (Admit failure)
How this would read in a community-based
accountability management letter

67. You know from news reports that the costs of poor performance on the state
pension fund have passed to schools. This has stressed our budget and one
consequence was that we froze hiring at the high schools. Because of
retirements and transfers at some of the high schools, we did not replace AP
teachers who retired or transferred and those courses were dropped. I
should’ve seen that problem coming (accepting responsibility and
tempering expectations).
This year our principals reviewed teacher assignments and are reinstituting
courses in areas where we have a teacher with appropriate credentials.
Because those teachers will be dealing with new content and larger class
sizes, we’re funding summer training for them in both the AP content and in
teaching strategies that better enable adaption of instruction with larger
student groups (demonstrating progress).
We can’t promise that this will entirely solve the problem, and our pass rate on
AP exams may take a hit while our new AP teachers master their assignment.
I’ll keep you apprised in my monthly newsletter as to how this is going.
How this would read in a community-based
accountability management letter

68. Reflect on one of your initiatives that
fell short from the past year.
Retell the story of that initiative to a
partner. Explain your goal, what your
strategy was, where your strategy
worked and didn’t, and how you intend
to correct course.

69. The annual report of learning…
provides a transparent common set of facts
(mastered by all leaders) that provide the
basis for discussing student performance.

70. Thank you!