This presentation was given by Anthony Arundel at the Public Conference “Innovation in education : What has changed in the classroom in the past decade?”. Measuring innovation in education and understanding how it works is essential to improve the quality of the education sector. Monitoring systematically how pedagogical practices evolve would considerably increase the international education knowledge base. We need to examine whether, and how, practices are changing within classrooms and educational organisations and how students use learning resources. We should know much more about how teachers change their professional development practices, how schools change their ways to relate to parents, and, more generally, to what extent change and innovation are linked to better educational outcomes. This would help policy makers to better target interventions and resources, and get quick feedback on whether reforms do change educational practices as expected. This would enable us to better understand the role of innovation in education.

1. Anthony Arundel
(Dominique Bowen Butchart & Sarah Gatenby-Clark)
Surveying administrative
innovations in tertiary education:
experience from Australia and NZ
OECD 2019
Study funded by the LH Martin
Institute, University of Melbourne

2. Online and mailed survey
OECD 2019
• 39 Australian
universities
• 6 New
Zealand
universities

3. Reasons for this survey
• Experiment with measuring innovations developed
by administrative staff in universities
– Builds on recent research on measuring
innovation in the public sector
• Collect data of value for benchmarking (across
universities and over time)
– Methods that managers use to innovate
– Outcomes and obstacles
OECD 2019

4. Comparison with OECD “Measuring
Innovation in Education”
OECD AUS-NZ Survey
Compare innovation in
education to other sectors
Main interest is comparing innovation in
universities to other public sector
organizations
Identify use of specified
educational practices over time
Innovations defined by type only (service,
etc), asks about a most important innovation
Construct metrics to evaluate
effect of educational practices
on outcomes
Focuses on the process for administrative
innovations: how innovations are developed,
plus outcomes and obstacles
Evaluate drivers of innovation
in educational practices (not
yet)
Drivers plus the effect of a ‘inclusive
innovation culture’
Develop an appropriate
methodology (not yet)
Part of ongoing work on measuring innovation
in the public sector

5. OECD 2019

6. Survey methods
• Questions cognitively tested in 13 face-to-face interviews
with Senior Managers at seven universities.
• Questionnaire sent to 1,516 senior managers in 10
functional areas (Library services, governance, IT
services, etc.)
– Not sent to senior executive or middle level managers
• 573 respondents (37.8% response rate)
• Responses from all targeted universities (45 in total)
• Questions refer to the respondent’s “area of responsibility”
• Reference period of two years.
OECD 2019

7. + Innovation culture
Revisions
- Risk aversion
OECD 2019

8. OECD 2019
Function
of the
unit
General
information
Innovation
environment
Innovation
methods
Use of
design
thinking
methods
Two year
reference
period for all
questions

9. OECD 2019
Most
important
innovation
Abandoned or
under-performing
innovation
Obstacles to
innovation
Most important innovation =
greatest expected impacts on the
respondent’s area of
responsibility, university,
students, or staff

10. Percent respondents reporting
innovations by type
28%
48%
50%
71%
91%
91%
0% 20% 40% 60% 80% 100%
Marke ng / communica on
Organisa onal
Services / products
Processes
Any innova on
Any innova on (Australian public
sector)
OECD 2019

11. Examples of the most important
innovations
• New form of therapy for university students.
• Enrichment program for high-achieving high school
students.
• Customized website to provide career development
strategies to international students.
• Online suite of resources to assist students in
managing social media.
• Mobile app to allow students to manage their
courses, lectures and tutorials from a smartphone.
OECD 2019

12. OECD 2019
Source of the idea for the most important
innovation, percent of respondents
3
6
6
24
30
31
0 5 10 15 20 25 30 35
Academics or students at your
university
Consultants, businesses or other
universi es
Other
Staff that report to the
respondent or other managers
Respondent
VC, Council or Senior Execu ve

13. ‘Percent respondents giving a ‘high
importance’ rating to innovation drivers
OECD 2019

14. OECD 2019
Methods to support
innovation

15. Widespread use of ‘good practice’
innovation methods
• 52% of respondent’s staff involved in
brainstorming meetings to develop ideas for
innovations.
• 61% of respondents delegate responsibility for an
innovation to an individual.
• 73% of respondents report collaborating on their
most important innovation.
• Majority of respondents use design-thinking
methods.
OECD 2019

16. Percent respondents reporting use of
design-thinking methods
Most of these methods involve ‘co-creation’
with end users of the innovation

17. OECD 2019
Does your workplace
have an inclusive
innovation culture?

18. OECD 2019
49% of respondents agree that their “Senior
Executive support a positive innovation culture that
includes all staff.” (inclusive innovation culture)

19. OECD 2019
Percent respondents by
university agreeing that
their Senior Executive
supports an inclusive
innovation culture

20. Share of staff involved in three innovation support
methods by agreement with an inclusive
innovation culture
OECD 2019
41%
39%
46%
51%
52%
58%
0% 10% 20% 30% 40% 50% 60%
Working groups to develop or
implement an innova on
Training for how to use an innova on
Mee ngs to brainstorm ideas for
innova ons
Agree that there is an inclusive innova on culture Disagree

21. Correlation between an inclusive
innovation culture & the use of design
thinking methods
Design-thinking method Odds
ratio
p
Conduct project user or focus groups 1.9 .029
Surveys of potential users 2.2 .004
“Ease-of-use” surveys 2.7 .001
Pilot tests of an innovation 1.9 .033
Post-implementation studies to identify problems 2.1 .012
OECD 2019
Results from logistic regressions that control for innovation type, reasons for
innovating, restructuring, number of staff, and function
Comparison between ‘agree’ with an inclusive culture versus ‘disagree.
Evidence for a dose-response effect for all methods.

22. OECD 2019
Outcomes (can be positive or
negative)
1. University’s brand or reputation
2. Simpler or faster processes
3. Increase in revenue
4. Employee working conditions
5. Student experience
6. Teaching and learning
7. Research
8. Reduction in costs
(Results for positive
benefits problematic –
lack of time?)

23. Correlation between an inclusive innovation
culture and “major positive effects” from the most
important innovation
Positive effect on: Odds
ratio
p
Simpler or faster processes 1.9 .08
Employee working conditions - ns
Student experience 2.2 .099
Reduction in costs - ns
University’s brand or reputation .38 .017
OECD 2019
Results from logistic regressions that control for use of design thinking
methods, number of staff, source of the idea, type of innovation.
High number of ‘too early to tell’ and ‘not relevant responses’

24. OECD 2019
The use of design-thinking methods and
collaboration strongly increase the
probability that the most important
innovation is a novel process or service

25. OECD 2019
What doesn’t
work?
• Abandoned or
underperforming innovations
• Presence of resource
obstacles
• Negative outcomes for the
most important innovation

26. OECD 2019
The absence of an inclusive
innovation culture doubles the
probability of an abandoned or
underperforming innovation
A budget cut increases the
probability of abandonment or
underperformance by 60%.

27. The odds of reporting each of three obstacles that are
measures of a lack of resources (skills, funding and time)
decreases substantially in the presence of an inclusive
culture (Odds of 0.32, 0.24 & 0.12).
OECD 2019

28. OECD 2019
The main factor increasing the
reporting of all resource obstacles
is when innovation is driven by a
crisis requiring an urgent
response (Odds of 2.2 to 2.8).

29. Factors correlated with one or
more negative effects from the
most important innovation (MII)
• The absence an inclusive culture
increases the odds of a negative effect
from the MII by 2.5 times.
• When the idea for the MII is obtained from
the Senior executive versus the
respondent, the odds of a negative effect
is increased by 1.9 times.
– (respondent better informed or tries harder?)
OECD 2019

30. Conclusions
• We can measure administrative innovations,
innovation activities and obstacles reported
by university managers.
– Beneficial outputs are harder to measure
(insufficient time?)
– Problem of self-reported outcomes
• What works: collaboration, inclusive culture,
design-thinking methods, other innovation
support methods, sufficient resources
– Similar success factors as in the private sector
and in other public sector innovation surveysOECD 2019

31. Conclusions: An inclusive
innovation culture
– Large impact on the use of ‘best
practice’ innovation support
methods such as design-
thinking.
– Substantially decreases the
probability of an abandoned or
under-performing innovation
and negative effects from a
most important innovation.
– No effect on novel innovations.
– Positive but not robust effect on
a few beneficial outcomes of
the most important innovation.
OECD 2019

32. Further information
University survey results:
https://www.oecd.org/sti/102%20-
%20ARUNDEL%20innovation%20in%20universities.pdf
Overview of measuring innovation in the public
sector:
Arundel A, Bloch C, Ferguson B. Advancing innovation in
the public sector: aligning innovation measurement with
policy goals, Research Policy, 2918,
doi.org/10.1016/j.respol.2018.12.001. (Open access)
OECD 2019