Evaluation is a key part of the improvement cycle and the reporting requirements of any course or program. It answers the questions: how did we do, and how might we make it better?
Quantitative evaluation methods can be relatively straightforward, meeting the need for numbers to share a broad view of what’s happening. But sometimes numbers alone struggle to tell the whole story of the impact and the experiences of individuals and groups.
3. Evaluating without numbers
Evaluation is an important part of the improvement cycle and answers the question – how did we
do, and how might we make it better? Quantitative evaluation methods are relatively easy to do,
and meet the need for numbers to tell the story.
In this presentation I will talk about my experience of a narrative or story based evaluation
method, which has recently been published in the journal Studies in Educational Evaluation.
4. The background
• 2007 Promoting Excellence Initiative funded by ALTC
(Australian Learning and Teaching Council) and the
networks began
Networks were state based, representing Universities
within each state
• 2009 now OLT (Office of Learning and Teaching)
funded the networks
5. OLT requirements
The OLT was a government agency that
“promotes and supports change in higher education institutions
for the enhancement of learning and teaching”
In 2011 the government determined to cease funding the agency,
so in 2011 at a final conference the OLT offered to distribute
$50 000 to each network. To do this the OLT required:
• A budget
• A plan of activities
• Evaluation plan
6. Our proposal
Budget – lots of things to spend money on
airfares, catering, meetings, showcases
Plan of activities – meetings, showcases, forums
Evaluation –
7. Evaluation
• The challenge was to demonstrate network value in a volatile
funding environment
• Would the numbers do it? We could count how many
meetings, how many came, how many attended the
showcases and forums. Does that show impact and value?
8. Evaluation of networks
• Usual practice is an end of project approach focussing on quantitative
methods we could still do that
• While the literature suggests that learning and teaching networks are
valuable – members struggle to express what the benefits are to them
• Research led us to narrative based approaches to evaluation
8
9. What do you think of when you hear the phrase
narrative based method?
10. This is what we did…
Developed three evaluation questions to address the aims and purpose of the
network –
• What were the experiences of participants in NSW/ACT PEN (the
network)?
• How effective were network activities in relation to the achievement of the
network aim?
• What factors will help/hinder future network sustainability?
11. Process of story building – a three phase approach
1. Individual reflective stories
2. A collective story
3. A value creation story
12. Phase 1: Individual reflective stories
Three times during the course of the project each network
member was invite to respond to the following trigger questions:
1. What did the previous network mean to my institution?
What could an on- going network mean to me as a network
member
2. What does the network mean to my institution and to me?
3. What did the network mean to my institution and to me?
13. Analysis of the stories
1. Read all the stories
2. Extract significant statements, words of phrases
3. Group like themes together
4. Formulate meaning under each theme by returning
to the stories
5. Integrate the themes into a description of the
members experiences.
14. Phase 2: Constructing a collective story
The purpose of this phase is to identify the
meta-themes that emerge in the middle of the
story. Two steps were undertaken to do this:
1. All themes were analysed from phase 1 to
identify meta themes
2. Individual stories were included under the
meta themes
15. Four meta themes emerged
1. Consolidation of existing connections and helping to foster
new collegial and cross-institutional links
2. Travelling on a common journey creates a sense of
community
3. Collaborative ways of working deepen learning and teaching
conversations to facilitate personal and professional capacity
building
4. Advancement of institutional practices and processes results
when institutional and personal professional outcomes
overlap
16. Phase 3: composing a value creation story
This addresses the evaluator questions such as the
network’s purpose, level of engagement.
The value categories were:
• Immediate value
• Potential value
• Applied value
• Realised value
• Reframed value
17. How did story building contribute to authentic
evaluation?
• Increasing visibility of the networks impacts by
collecting evidence from multiple sources using a
variety of methods
• Exposing affective and relational dimensions of
network life
• Collating this evidence in a form that could easily be
shared with others
18. Some conclusions
• The data gathering and analysis was time
consuming
• The process captured the qualitative elements
of the experience
• The story can demonstrate current and future
impacts of networks
