A case study of the application of HPM and Perimeta to school leadership in three English Academies...<a><img src="http://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a>Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.

1. Evaluating the wider outcomes of
schooling: the Oasis ECHO project
Raising standards
through creating learning communities
Ruth Deakin Crick, Stephen Barr, Howard Green

2. ! Background
• The Education Charter
• The Echo Project
• Three Oasis Academies

3. ! What to do?
"Here is Edward Bear, coming
downstairs, bump, bump, bump, on
the back of his head, behind
Christopher Robin. It is, as far as he
knows, the only way of coming
downstairs, but sometimes he
feels that there really is another
way, if only he could stop bumping
for a moment and think of it"
Alan Alexander Milne
The opening paragraph of “Winnie-the-Pooh”, 1926

4. • Systems thinking – managing complexity
• Developing resilience – becoming learning
organisations
• Responding to emergence – unpredictable
outcomes
• Taking responsibility for our own development
• Engaging all stakeholders in a networked
improvement community

5. ! Key Questions
• How can we measure the full range of what we
value in schools?
• How can collect and manage evidence to
understand the big picture as well as the detail?
• How can we lead and manage complexity?
• How can we used data for real-time leadership
decisioning as well as for evaluating performance

6. ……
Learning is a core process at multi-levels in learning
Macro Layers and feedback loops
organisations
Community
Learning
Leaders
Learning
Teachers
Learning
Students
Learning
Blockley, D. (2010) The Importance of Being Process, Civil Engineering and Environmental Systems, 27(3)189-199

7. ! The complex problem of
creating learning communities

8. ! Imagine ……

9. What sort of design should
we use to model our
learning communities?

10. A Hierarchical Model: Defined by Purpose

11. How
data
3. Inputs
Sources
of
evidence
Students
Teachers
2.3 Developing the
learning of leaders
so that they lead the
learning of teachers
and students
effectively
Leaders
3.11 Contributing actively to a community that
3.12 Feeling included and involved best and
learns together, shares what works with my
children’s education to do Academy and knowing
knows what it needs at the to improve
that there is an open door for contact when I
need it
3.13 Being effective role models as leaders of
3.10 Providing evidence that training and
development opportunities have helped me to
improve my classroom practice and/or my
effectiveness as a leader and manager
3.9 Contributing actively to a community where
shared leadership is promoted
2.2 Developing the
learning of teachers
so that they realise
their full potential
3.7 Feeling that my successes are recognised
3.8 Contributing actively to a community which
focuses on service to others
2. Outputs
2.1 Developing the learning
of students so that they
realise their full potential
3.6 Achieving results that meet my aspirations
and expectations
3.4 Knowing that we are being helped to reach
our full potential
3.5 Maintaining a good rate of progress at key
transitions, for example, primary to secondary
school
3.3 Knowing that I am becoming an effective
learner
3.2 Feeling part of a community that focuses its
activities on learning and achievement
3.1 Feeling valued and included in our community
PURPOSE
Outcome
1.1 Establishing and sustaining a group of high achieving learning communities
that enables everyone to realise their full potential and refuses to put limits on
achievement
2.4 Engaging parents/
carers effectively in the
learning activities of the
Academy and in
supporting the learning of
their children
Parents/ carers

12. What type of data do we
need to collect to measure
the core processes which
fulfil our purpose?

13. Stakeholders
Type of data
Focus
1
Y7 students
Numerical scores per dimension
Learning power
2
Y8 & 10
Students
Numerical values for each question
students as learners and
values & citizenship
3
Y9 students
Stories rated by researchers
Personal transformation
4
Y11 leavers
Numerical values for each question
Aspirations, trajectories
5
Parents/carers
Numerical values for each question
Participation
6
Teachers
Numerical values for each question
Impact of professional
learning on student
outcomes
7
Leaders
Numerical values for each question
Leaders as learners
8
KPI s
Numerical values of performance
Behavioural
indicators
9
Y11
Numerical values of performance
Attainment

14. ! The nature of uncertainty
Fuzziness
Incompleteness
Randomness

15. ! .. combined with performance measures that
cope explicitly with uncertain evidence
Evidence that A
is successful
Lack of Evidence
Evidence that A
is not successful
0
1
Performing well with little uncertainty
Performing poorly with some uncertainty
We don’t really know what is going on!

16. Data collected to evaluate our core processes
Means & SDs
Yes
Uncertain
Perimeta model
No
A
Leadership Ownership and
Decisioning
P D
A S
Improvement Protocols
Rapid Prototyping
Harnessing collective intelligence across the system

17. So what does our
Perimeta model tell us
about Parents/carers ..…
an example

18. ! Dashboard summary

19. ! Perimeta model results

20. ! Drilling down into the evidence
to understand key issues
Outcome
Outputs
Inputs
Establishing and sustaining learning communities
Developing the learning of students so that they realise their full potential
Feeling valued and
Feeling part of a
Knowing that I am
Knowing that we are
included in our
community focused on
becoming an effective
being helped to reach
community
learning & achievement
learner
our full potential
Statement
I was helped and supported when I struggled
My teachers taught me well
The Academy made good use of technologies
I was valued and included
I was often challenged to do better
There were great facilities
I felt part of a learning community
I developed responsibility towards others
I had a wide range of opportunities for learning
I felt safe
POS
0.98
0.90
0.90
0.89
0.89
0.89
0.89
0.89
0.89
0.89
Statement
I am looking forward to the move to Sixth
Form, Further Education, training or work
I led activities during my time at the Academy
No others above 10%
NEG
0.18
0.17
Statement
I supported activities that would improve things
for other people
I led activities
I am looking forward to the move to Sixth Form,
FE, training or work
My parents/carers felt involved
Everyone worked very hard to make it a great
place to learn and to do well
I enjoyed my time
I could not have gone to a better school
I was offered opportunities for leadership
I never felt that anyone was putting limits on
what I could achieve
My teachers accurately predicted my GCSEs
UNC
0.38
0.34
0.33
0.15
0.15
0.15
0.14
0.14
0.13
0.13

21. ! The PerimetA toolkit supports
collaborative action to improve learning
Focusing on
PERFORMANCE vs
broad outcomes
KEY FEATURE:
“Italian Flag”
evidence model &
rich uncertainty
calculus
KEY FEATURE:
Excel interface for
linked data and
predictive models
Understanding
COMPLEX
PROCESSES
Weighing up
uncertainty of
EVIDENCE
Informing robust
CHOICES
Enabling robust
& collaborative
ACTION
KEY
FEATURE:
Hierarchy of
interactions
focused on
outcomes

22. So how do we respond to
what Perimeta is telling
us as a learning
community?

23. Professional Enquiry through Rapid Prototyping
to address key challenges highlighted by
Perimeta….
Plan
Act
Do
Study
…..within a disciplined evaluation framework

24. Measurable System Improvement: New Knowledge and Know
How
P D
A S
P D
A S
P D
A S
P D
A S
P D
A S
P D
A S
P D
A S
P D
A S
P D
P D
A S
A S
P D
P D
A S
A S
Experiential knowing, hunches, ideas
Bryk, A. Gomez L. Gunrow A. (2010)
Getting Ideas into Action: Building
Networked
Improvement Communities in Education,
Carnegie Foundation for the
Advancement of Teacher Education,
Carnegie, Stanford.

25. How can we use
technology to enhance
our ability to develop
organisational learning
through Perimeta

26. ! An academy with systemic
analytics
X
Academy
Learner
Dispositions
Teacher
Dispositions
Reflective Enquiry
Blogs
Strategic Progress
Stories of Change
& Evidence

27. ! An academy network with systemic
analytics
Evidence Hub
Data
Data
Data
Analytics
Integration
Data
2
7

28. ! Systems Learning & Leadership Evidence Hub
http://sysll.evidence-hub.net
Share
and
debate
evidence…
About…

29. ! Systems Learning &
Leadership Evidence Hub
http://sysll.evidence-hub.net
Replay Oasis/Univ Bristol/Open Uni event: http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
2
9

30. ! Learning Analytics Hub (3.0)
Online Tools designed to measure core
processes and to
stimulate self-directed change
Learning Analytics
System Simulations
Recommendation Engines
Data Sets + Data Streams
Buckingham Shum, S. & Deakin Crick, R. (2012) Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics,
Proceedings of LAK12: 2nd International Conference on Learning Analytics & Knowledge, April 29 - May 2, 2012, Vancouver, BC. ACM: New York. ISBN
978-1-4503-1111-3.

31. ! Conclusions
• The ECHO Project as a proof of concept:
The ECHO Project has been an attempt to weave
all the evaluation strands together in order to
create a whole and more balanced snapshot of the
school’s performance for improvement planning …
initial analyses suggest that conceptually,
professionally, practically and technically there is
real promise in this approach … use of the
Perimeta model enables links to be established
between extended self-evaluation, performance
management and school improvement planning.

32. ! Conclusions
• Further research is needed, plus drilling down
into the pilot project data, particularly the
relationship between qualitative and
quantitative data, e.g. what is the relationship
between Year 11 leaver responses (Strand 4
questionnaire) and GCSE results (Strand 9
data)?
• Traditional statistical analysis has shown that
the questionnaires and interview schedules
can be refined and reduced in length

33. ! Conclusions
• Evaluation of ‘teachers as learners’ and
‘leaders as learners’ is relatively new and
would benefit from review and further
development, e.g. addition of a 360degree perspective to the processes
• Approaches and processes developed
through ECHO offer the potential to extend
and enrich the insights of senior
leadership teams (SLTs) into next steps
for improvement and for the Oasis group
as it seeks to identify shared issues that
would benefit from coordinated resourcing

34. ! Conclusions
• Perimeta feedback offers an opportunity to
celebrate and share what’s going well, to
address areas of weakness and, uniquely,
to explore further aspects of the school
where there is uncertainty from the
evaluation evidence.
Perimeta is designed to draw people into exploring
and managing their own data. It is essentially a
decision-making tool. It measures for the purpose
of stimulating positive change according to locally
defined need.

35. ! Next steps
• Reporting to and feedback from 3 pilot
Academies
• ECHO questionnaires and interview
schedules to be refined, put online and
rolled out across the group
• New questionnaire for Academy
Councillors to be piloted in 2013/14
• Questionnaire for wider community
involvement to be developed and piloted in
2013/14

36. ! Next steps
• Further development of Perimeta as a model
directly available to schools
• Further research on issues arising from the
ECHO Project
• SLTs encouraged to develop the rigorous and
systematic use of qualitative as well as
quantitative evaluation evidence for
performance review and improvement planning
• OCL to build similar approaches into
performance review and improvement planning
across the group

37. ! Next steps and, finally, our thanks
• Embedding of the ECHO work within Oasis,
via OCL and, possibly, OCHE/CLICT
• Wider dissemination of the ECHO results

38. With thanks to
Students, staff and parents/carers at the ECHO
especially, students, staff and parents/
Academies carers at the 3 ECHO Academies; staff in
the Graduate School of Education at
Bristol; ‘Perimeta people’; University
Graduate School of Education – staff at OCL/ of
Oasis Centre; Jerusalem Trust for funding
Systems Centre – University of Bristol
University of Bristol Research Committee
Oasis Community Learning /Oasis Centre
The Jerusalem Trust
Bristol