1.
Using Theory of
Change to evaluate
information literacy
initiatives
Information
School,
University of
Sheffield.
LILAC 2022
Dr Pam McKinney
Sheila Webber

2.
Presentation outline
1. Introductions and our experiences of using Theory of Change
2. Overview of Theory of Change
3. How a Theory of Change is constructed
4. The role of reflection in Theory of Change
5. Developing an evaluation plan
Plus (not presented) - Examples of using Theory of Change - see also the
bibliography at

3.
Introducing ourselves - Pam
● Background as an educational developer, working at CILASS:
Centre for Inquiry-based Learning in the Arts and Social
Sciences
● Theory of Change methodology applied at project and
programme level to evaluate the impact of the Centre
● I led Theory of Change creation process with stakeholders,
and supported the development of evaluation plans
● Supported reflective evaluation process, and jointly published
with project leaders using evaluation data
● Now: lecturer at the University of Sheffield Information School
Logo:
Sabine
Little
Pam McKinney/Sheila Webber 2022

4.
Introducing ourselves - Sheila
● Senior Lecturer in the Information School,
University of Sheffield
● Information Literacy:
○ Teaching, researching & writing about IL for
many years;
○ Currently part of UNESCO's MIL initiative &
FOIL
● Theory of Change:
○ Leader for one of the projects in CILASS;
○ Running ToC workshops with Pam;
○ Using ToC working with educators in virtual
worlds
Pam McKinney/Sheila Webber 2022

5.
What is Theory of Change?
● A theory-based, participatory approach to project evaluation developed by the Aspen
Institute (Connell & Kubisch 1998)
● Designed to create a link between funding and outcomes for stakeholders
● Can be used as an organisational level tool across multiple projects to standardise
evaluation practice and identify and share good practice.
● Theory-based evaluation explains how and why results are achieved through
understanding the links stakeholders construct between activities and outcomes
● Participatory evaluation avoids feelings that evaluation is “done to” projects, instead it
is a flexible tool applied throughout the project that involves multiple stakeholders.
● Longitudinal evaluation focuses on intermediate and longer term impacts, and can be
revisited long after a project has ceased.
Hart, Diercks-O’Brian & Powell, 2009
Pam McKinney/Sheila Webber 2022

6.
The role of the facilitator: creating a Theory of Change
with stakeholders
● Literature is positive about the role of an external facilitator to collaboratively
design an educational intervention, define a Theory of Change and evaluation
plan, and carry out evaluation activities
● In the CILASS project an educational developer led the ToC creation process with
project stakeholders (e.g. academic staff, students, professional services staff and
librarians)
● An audio-recorded planning session, featuring interviews with key stakeholders,
with a draft ToC created
● The educational developer then led the revision and finalisation of the ToC, and
the resulting evaluation plan.
Hart, Diercks-O’Brian & Powell, 2009
Pam McKinney/Sheila Webber 2022

7.
Constructing a Theory of Change
Drivers Resources/
enabling
factors
Activities Outcomes Anticipated
impact
What is the
current situation
that has led to
the project?
What support is
needed to do
the project
activities?
What activities
need to take
place to
achieve the
project
outcomes?
What are the
desirable and
feasible
outcomes for
the project?
What is the
longer term
impact of the
project?
There should be a consistent and credible narrative developed across the 5 columns, where
an outcome has specific activities and enabling factors associated with it.
Pam McKinney/Sheila Webber 2022

8.
Example Theory of Change from the CILASS programme
Pam McKinney/Sheila Webber 2022

9.
Role of reflection in Theory of Change
● Reflection is a key aspect of the Theory of Change design
● Based on “reflective practitioner” model (Kolb, 1984; Lewin, 1946; Schön,
1983).
● Recognition that change is a complex, socially-constructed process, and
reflection can help articulate this.
● Critical reflection from project stakeholders helps
identify relationships between actions and effects
● Reflection is a desirable aspect of the evaluation
plan
Hart, Diercks-O’Brian & Powell 2009
Photo
Nigel
Ford:
Pam
McKinney/Sheila
Webber
2021

10.
NOW - work
on your
theory of
change with a
partner
How could youuse
Theories of Change to
evaluate Information
Literacy initiatives?

11.
Next step - developing an evaluation plan
● Once a Theory of Change has been agreed by project stakeholders, an
evaluation plan is created.
● Each Enabler, Process and Outcome indicator should have an evaluation
process associated with it
● Multiple data collection methods can be used, qualitative and quantitative
● A significant role for reflection in evaluation, often taking the form of reflective
interviews with project stakeholders
● An evaluation matrix is created which articulates the evaluation method to be
used for each indicator
Pam McKinney/Sheila Webber 2022

12.
Evaluation matrix
ToC indicator Student focus
group
Librarian
questionnaire
Project leader
reflection
Enabler: Effective support structures are
developed for librarians to help them
design effective online IL tutorials
X
Process: A suite of online IL tutorials is
created based on ACRL IL framework
X
Outcome: students have confidence in
referencing practices and can accurately
cite sources in their work.
X X
Pam McKinney/Sheila Webber 2021

13.
Sheila Webber
Information School
University of Sheffield
s.webber@shef.ac.uk
Twitter: @sheilayoshikawa
http://information-literacy.blogspot.com/
http://www.slideshare.net/sheilawebber/
Dr Pamela McKinney
Information School
University of Sheffield
p.mckinney@shef.ac.uk
Twitter: @ischoolpam
https://www.slideshare.net/PamelaMcKinne
y/
Workshop outline and references at
https://tinyurl.com/toclilac22

14.
Examples for further reading

15.
Examples of Theory of Change in practice
(1) USA's Public Library Association (2018) - developed a ToC for Professional
Development
● Identified goals, outcomes, strategies & competencies required to achieve
outcomes
● Identified 6 key roles for LIS professionals to achieve outcomes
(2) The Youth Cafe - a "pan-African youth empowerment organisation" - uses ToC
and stresses participative element "We have incorporated the views of young people,
youth-led and youth-serving organizations, and experts so that our map reflects young
people’s view of how change occurs" (The Youth Cafe, 2021)
Pam McKinney/Sheila Webber 2021

16.
The
Youth
Cafe
(2021)

17.
Examples of Theory of Change in practice: 3
De Buck et al. (2018) describe the process of developing a ToC aimed at
changing handwashing and sanitation practices in low- and middle-income
countries
1. A draft ToC was based on "existing sources of information, including
theoretical models, frameworks and systematic reviews" (p.11)
2. Draft ToC circulated to stakeholders & workshop held to develop ToC, with
2nd draft presented to stakeholders for further response
3. "Improved" ToC submitted to Campbell Collaboration as part of the review
protocol for a systematic review, to get peer review
4. Systematic review carried out, final ToC again presented to stakeholders
Pam McKinney/Sheila Webber 2022

18.
Pam McKinney/Sheila Webber 2022
De Buck et al. (2018, p.15)