Webinar hosted by James Smith and Kim Robertson puts a spotlight on data sovereignty and the importance of listening to Indigenous perspectives on evaluation in Indigenous higher education.
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
NCSEHE webinar: Indigenous Perspectives on Evaluation in Indigenous Higher Education
1. CRICOS Provider Code 00301J
TOWARDS DATA SOVEREIGNTY:
National Centre for Student Equity in Higher Education (NCSEHE) 6 April 2018
A NATIONAL CONVERSATION ABOUT STRENGTHENING EVALUATION IN
INDIGENOUS HIGHER EDUCATION CONTEXTS IN AUSTRALIA
2. Building Legacy and Capacity
workshop series
Setting the scene
The logic behind the concept:
• NCSEHE Board: strategic project to build legacy beyond the current funding period and
leverage the NCSEHE funded research.
• Deepening some of the insights generated during the 10 Conversations at the NCSEHE
2016 National Forum.
• Developing a format that enables productive conversations about complex issues.
• 2017 workshops focused on career education in low SES and regional/remote schools as
well as supporting regional and remote students in higher education.
The objectives of the workshops are to:
• Define a collective knowledge base informed by research and practice.
• Engage in strategic and action planning to guide institutional practice and
future research.
• Develop evidence-informed policy advice.
3. Six high-level questions frames the group discussion:
1. What do we know about the nature of the problem?
2. What does success look like?
3. What do we know from current practice and research: What works? What doesn’t work?
Why and why not?
4. What principles underpin good practice and why?
5. How could the Australian Government, State Governments and Universities, better
support effective approaches?
6. What are the gaps in knowledge to promote positive change?
Building Legacy and Capacity
workshop series
Structure of the workshop
4. Expert Workshops:
• 10-12 subject matter experts approaching the topic from difference perspectives:
researchers, equity practitioners, policy makers and community partners.
• Advance a national conversation at the intersection of equity research, practice and
policy for the benefit of the sector.
Dissemination of insights generated during the workshop:
• Webinars.
• Publication: extended version of pre-reading document, including Good Practice Guide
and recommendations for policy and research.
• Professional illustration to complement text.
• Presentations at the EPHEA Conference and to the Department of Education
and Training.
Building Legacy and Capacity
workshop series
Approach: Expert workshops and multi-modal discussion
5. Peter Anderson Queensland University of Technology and WIRA
Jason Brailey RMIT
Nathan Cassidy Universities Australia
Bronwyn Fredericks Central Queensland University
Kathryn Gilbey University of Southern Queensland
Cheryl Godwell Charles Darwin University and NATSIHEC
Kim Grey Department of Prime Minister and Cabinet
Leanne Holt Macquarie University and NATSIHEC
Maria Raciti University of the Sunshine Coast
Kim Robertson Charles Darwin University (Workshop Co-Facilitator)
James Smith Menzies School of Health Research (Workshop Co-Facilitator)
Karen Treloar TEQSA
Sue Trinidad NCSEHE
Nadine Zacharias NCSEHE (Workshop Series Co-ordinator)
Towards Data Sovereignty: A national conversation about
strengthening evaluation in Indigenous higher education
contexts in Australia
Workshop 3 participants
6. Acknowledgement of Country
We would like to acknowledge the Larrakia
people, upon whose land we present from today.
We also acknowledge the lands from which
participants join us today.
We pay our respects to Aboriginal Elders
past, present and emerging from these lands,
and we extend this acknowledgement to all other
Aboriginal and Torres Strait Islander people
here with us today.
7. A SNAPSHOT OF WHAT WE KNOW ABOUT
EVALUATION IN INDIGENOUS HIGHER EDUCATION
8. What do we know about evaluation in
Indigenous higher education?
While considerable data was available
through departmental program-based
reporting to monitor progress, there was
not always sufficient evidence to assess
the overall success or otherwise of specific
programs. In some cases, there were no
independent evaluations of programs for
the Panel to draw on.
(Behrendt et al 2012, p154)
9. Why is this important?
There are unique considerations in the Indigenous HE space:
• On Stony Ground — emphasised importance of Indigenous
governance and participation in Australian universities
(Moreton-Robinson et al 2011).
• Behrendt Review (2012) — recommended the development of
an Indigenous HE monitoring and evaluation framework.
• HEPPP 2014 NPP Grant — ‘building an evidence base about
Indigenous pathways and transitions into HE’.
• Aboriginal and Torres Strait Islander Higher Education Advisory
Council (2016) — recommended development of an Indigenous
HE Performance Framework.
• Release of the Universities Australia Indigenous Strategy
2017-2020.
10. What else do we know?
Key points of interest:
• High level of scrutiny of evaluation in
Indigenous affairs contexts.
• Indigenous Advancement Strategy
evaluation framework developed.
• Existing models — e.g. Aboriginal and
Torres Strait Islander Health Performance
Framework?
• Fragmentation of Indigenous policies and
programs in higher ed at national level.
• Equity vs Indigenous.
• Indigenous leadership, standpoints,
knowledges and cultural standards are
important.
• Emergence of data sovereignty
discussions nationally and globally.
11. Key points of interest:
• Collecting baseline data from all Australian
universities in 2018.
• Strategy is aspirational.
• KPIs are the link to executive attention.
• Indigenous workforce issues are urgent! Generation
about to retire: Indigenous expertise will be lost.
• Document is the thorn in the side of the exec to get
attention and keep the agenda going.
• Indigenous people don’t just want to sit in Indigenous
studies any more. Want to be across the institution.
• You can’t collapse reporting to quantitative
measures, but needs qual reporting as well.
• Resources a challenge: Indigenous areas are asking
for their fair share (i.e. not just Indigenous
supplementary $).
Reporting against the Universities
Australia Indigenous Strategy 2017-2020
12. 1. Growing Indigenous leadership.
2. Addressing white privilege and power.
3. Valuing Indigenous knowledges and prioritising Indigenous epistemologies.
4. Incentivising cultural competence.
5. Embracing political challenges as opportunities.
6. Recognising sovereign rights.
7. Increasing funding and resources.
8. Leading innovative policy development, implementation and reform.
9. Investing in strategy development.
10. Investing in cultural transformation, change and quality improvement.
11. Improving Indigenous student outcomes.
12. Promoting cultural standards and accreditation.
13. Reframing curricula to explicitly incorporate Indigenous knowledges and practices.
14. Investing in an Indigenous workforce.
Enablers of, and drivers for,
strengthening evaluation
2017 NCSEHE Equity Fellowship
13.
14. Yarning about evaluation
• Drivers/enablers not unique to evaluation — but also common to other areas of
university business.
• Data sovereignty: Indigenous control and decision-making at all stages.
• Executive decision making: Indigenous people not always at the table. Indigenous PVCs
often not directly involved in important decisions.
• Acknowledge difference between ‘equity’ and ‘sovereign rights’.
• Cultural experts should guide evaluation work. There is an opportunity to build the
evaluation capacity/skills of the Indigenous HE workforce.
• Evaluation often perceived as the ‘poor cousin’ — program/intervention planning and
implementation given priority.
• There are very complex lives behind the numbers. Stories and other forms of qualitative
evidence can assist. Context is important!!!
• Combining different information (e.g. ISSP + UA) increases the capacity to develop more
detailed institutional snapshots.
What is the problem?
15. Yarning about evaluation
What is the problem?
• Segregation of ‘black business’ — often not seen as core work. Whole of University
approach is required.
• We need evaluation tools that measure commitment, not just compliance.
• What’s the end game? What will be the outcome of the data collection?
• Do we need a repeat of On Stony Ground? Could be used to see what has changed? What
has worked? What hasn’t?
• Cross-cutting evaluation: how do we get different evaluations to inform each other?
• Equity and Indigenous.
• Different education sectors (e.g. early years, schools and universities).
• Other sectors (e.g. health).
• How do we evaluate the contribution that Indigenous people are making to the
mainstream HE agenda?
17. • ‘Success’ is subjective — trying to define success often generates more questions than
answers.
• Indigenous epistemologies and ontologies need to be valued. These should be reflected
through Indigenous methodologies, methods and in the ‘evaluation terminology’.
• Proximity to influence is important. Indigenous people need to be involved in deciding
what gets evaluated, how and why. Indigenous leadership should underpin institutional
evaluation.
• Evaluation design is important. This should start early with the end goal in mind.
Expected impacts and outcomes can be mapped out over time.
• Evaluation should be forward looking and inform changes in practice.
• Better longitudinal data would be useful to frame future strategy development.
Defining success
Meaningful evaluations of Indigenous higher education
18. • Collecting data: we need to collect data, and build data systems, that correspond to the
identified needs and aspirations of Indigenous students.
• Using data: we need clear progress markers that serve a clear purpose at institutional
and national levels.
• Indigenous student voices are privileged in program design and evaluation (i.e. student-
centred approach).
• Summative evaluations should not be used punitively by institutions (i.e. to cut
programs). Rather they should inform quality improvement.
• Tensions between compliance reporting and impact/outcome evaluation are resolved
by, for example, placing a greater focus on program efficacy and effectiveness.
• Benchmarking against mainstream indicators is not always useful because it can
perpetuate a deficit discourse. Indigenous specific benchmarks would be a better
alternative.
• There should be integrated funding models and reporting requirements across
institutions and Government.
Defining success
Meaningful evaluations of Indigenous higher education
19. 1. Recognise sovereign rights (who speaks for whom?).
2. Demonstrate and celebrate Indigenous expertise, diversity, excellence and difference.
3. Commit to systems and continuous quality improvement.
4. Foster Indigenous leadership within and outside of the sector.
5. Clearly define the purpose of evaluation (Indigenous self-actualisation).
6. Recognise cross-cutting nature of Indigenous evaluation.
7. Focus on use and impact.
8. Build and design evaluation expertise among Indigenous and non-Indigenous HE staff.
9. Adopt a forward focus.
10. Ensure evaluation methodologies are ‘fit-for-purpose’ (adaptive and acknowledging
diverse goals of Indigenous HE).
11. ‘Do no (more) harm’ (evaluation not always a safe space; be careful what you ask;
resistance against power sharing with Indigenous HE stakeholders is real).
12. Strive for innovation that builds on Indigenous knowledges/practices.
Good practice principles
20. Recommendations for future practice
1. Create a structure of responsibility and accountability that supports a whole-of-
university approach.
2. Leverage the current policy shift from access/recruitment to retention/completion to
change the balance between funding, the institution and students (and their
supporters).
3. Put Indigenous consciousness and praxis at the centre of decision making to facilitate
agency within the institution.
4. Leverage expertise of Indigenous staff across the sector, rather than tie them to an
institution.
5. Hold a strategically focused Indigenous higher education conference.
6. Universities could consider making a ‘contribution’ to NATSIHEC to build capacity,
disseminate findings, advance major projects, and provide regular communication.
21. Recommendations for policy
1. Pay greater attention to recommendations from past reports (i.e. development of a
performance, monitoring and evaluation framework/strategy).
2. Bring together Indigenous higher education policy making, programming and funding
into the same Australian Government Department to increase cohesion.
3. Enhance cross-portfolio communication and direction, including across the silos of the
Indigenous Advancement Strategy (IAS).
4. Generate actionable insights from annual reports to government: review, synthesise,
feed back to the sector and all key stakeholders, incl. TEQSA.
5. Consider a legislated commitment to Indigenous higher education in university
legislation as is the practice in Victoria. The founding Acts of Victorian universities
include the following statement: “The objects of the University include:
(i) to use its expertise and resources to involve Aboriginal and Torres Strait Islander people of Australia in
its teaching, learning, research and advancement of knowledge activities and thereby contribute to —
(ii) realising Aboriginal and Torres Strait Islander aspirations; and
(iii) the safeguarding of the ancient and rich Aboriginal and Torres Strait Islander cultural heritage”.
Government
22. Recommendations for policy
1. Higher Education institutions to take Indigenous workforce issues much more
seriously. Urgent action required!
2. Ensure Indigenous people are involved in decision-making at all levels of Institutions
1. Continued consultation and partnership development between UA and NATSIHEC to
input into measures of Indigenous evaluation
2. NATSIHEC to provide guidance to TEQSA on what the expectations are to support
Indigenous evaluation (particularly in relation to national standards and accreditation):
conversation to commence under the current MOU
3. Review the ways in which TEQSA chooses ‘experts’: how does it recognise Indigeneity
and Indigenous expertise?
Other Players – NATSIHEC/TEQSA/UA
Institutions
24. Q1. Are there any examples of Indigenous methods/methodologies in
evaluation? Do these conflict with any requirements under the
governance of universities as public institutions?
• Lots of texts about Indigenous methods/methodologies - although primarily relating to
Indigenous research not necessarily evaluation (e.g. Linda Tuhiwai Smith — NZ; Maggie Kovach
— Canada).
• Some emerging work relating to developmental evaluation (e.g. Michael Quinn Patton — US;
Nan Wehipeihana — NZ) and realist evaluation (e.g. Gill Westhorp and Emma Wiliams — Aus)
applied to Indigenous affairs contexts (but seldom applied in higher education).
• Qualitative methods such as yarning, storytelling, use of narratives and metaphors are
considered important Indigenous methods in research and evaluation. They are currently
under-utilised in Indigenous higher education. Watch-out for the release of Professor James
Smith’s Equity Fellowship Report (it raises various aspects of this topic).
• Quantitative data is also important, but best combined with qualitative methods to provide a
greater level of context.
• No perceived conflicts with governance requirements of universities as public institutions.
• Keep an eye out on data governance/sovereignty discussions happening nationally and
globally.
25. • Absolutely!!! ...but also important to consider local context.
• There are research projects across Australia (both institutional and national in focus)
examining the Indigenous student experience in different ways, and at different junctures. See
for example:
• OLT project being led by Prof Denise Wood at CQU in conjunction with JCU, UniSA, UoN and CDU —
report to be released after June 2018.
• Book and journal issue on pathways into higher education for remote students (perhaps focused more
on outreach programs for prospective students) —
https://www.springer.com/gp/book/9789811040610; and
http://www.cdu.edu.au/sites/default/files/the-northern-
institute/cdu_ni_learning_communities_journal_2015_17_read-online.pdf.
• NCSEHE project led by Prof James Smith examining Indigenous VET to HE pathways —
https://www.ncsehe.edu.au/identifying-strategies-for-improving-vet-to-he-transitions-for-
indigenous-learners/.
• NCSEHE project led by Dr Jack Frawley on Indigenous self-efficacy research —
https://www.ncsehe.edu.au/mediareleases/motivations-for-success-strengthening-self-efficacy-to-
improve-indigenous-student-outcomes/.
• NCSEHE project being led by Ms Fiona Shalley examining Indigenous student completions in two
regional universities — https://www.ncsehe.edu.au/2017-research-grants-program-projects/.
• There is evidence of Indigenous students being successfully involved in student survey design
and implementation in Canada — e.g. Western University, University of Saskatchewan.
Q2. Is there value in a sector-wide approach to capturing Indigenous
student voice for the purpose of benchmarking Indigenous student
experience (as opposed to local university level data collection)?
26. • Continue to collaborate with Indigenous colleagues and peak Indigenous bodies to evaluate this
work.
• NATSIHEC is leading some work relating to STEM…the report should be released shortly.
• Please publish outcomes of this work (there is very little quality evaluation evidence in this
space…despite increased attention being paid to Indigenous students and STEM – see for
example https://www.pmc.gov.au/news-centre/indigenous-affairs/funding-applications-
invited-deliver-indigenous-girls-stem-academy.
• There is no set rule about how to measure this, but think creatively and well done on what you
have achieved so far!!!
Q3. [We] are part of a grant which brings together some Indigenous
people and some mathematicians and statisticians, using Indigenous
pedagogies in mathematics and statistics classes. Some of those
classes have few or no indigenous students enrolled.
Do you have advice on how to evaluate the effectiveness of this
strategy, given so few Indigenous students directly affected (though
many more might be eventually, since many of the maths students are
teaching students). We do acknowledge the importance of also
affecting non-Indigenous students. We would love advice how to
measure, and what to measure.
27. • Increase the Indigenous workforce (this needs urgent attention). This needs to occur across the
university not just in ‘Indigenous offices/centres/areas’.
• Increase Indigenous leadership and governance across all areas of the university.
• Strategically use the skills and expertise of Indigenous academics and the growing number of
HDR students.
• Involve Indigenous students as part of the strategy development.
• Watch out for the upcoming NATSIHEC report which has a section on WOU approaches.
• Evaluate how these approaches are working (i.e. good quality process and impact evaluations).
• Encourage the sector to share stories about what is working and why (note: some universities
appear to be reluctant to share evidence outside of their own institution).
Q4. What were the practical strategies for creating a 'whole of
university' approach within tertiary institutions.
National Centre for Indigenous Studies at ANU would be highly
supportive of the idea to hold a national conference on these issues.
28. • Absolutely!!!
• There are some good examples about internet being used in creative ways to engage
Indigenous students/communities.
• Regional universities have developed solid expertise in this area.
• Check-out the Broadband for the Bush Alliance — http://broadbandforthebush.com.au/.
• There hasn’t been a great deal of thinking about how to embed internet into evaluation
processes/methods in Indigenous higher education contexts (usually its been approached from
a program implementation perspective). Innovation is required in this regard!
Q5. Is there a role for the Internet, as a way to get the university out
beyond the campus? Have in mind Philip Townsend's work on Mobile
Devices for Aboriginal Pre-service Teachers
http://blog.highereducationwhisperer.com/2018/04/mobile-devices-
for-aboriginal-and.html
29. Professor James Smith
& Ms Kim Robertson
C/- Office of the Pro Vice Chancellor — Indigenous Leadership
Charles Darwin University
james.smith@menzies.edu.au
kim.robertson@cdu.edu.au
