Presented at LILAC 2022

1. The future of feedback
Evaluation of information and digital literacy
teaching in higher education
Siobhan Dunlop (they/them)
University of York

2. Background to the project
Part of a Postgraduate Certificate in
Academic Practice.
Work in a team that delivers digital skills
training in University of York's Library,
Archives and Learning Services.
Need to think beyond our current Google
Form.

3. What was the project?
Evaluation of information and digital literacy teaching in higher education
Evaluative practice in higher education (Sanders, 2011) - not the evaluation of HE
Information and digital literacy
Embedded within modules or non-embedded and outside of academic courses
Synchronous teaching and asynchronous teaching interventions

4. Why do the review?
Examine current landscape of evaluating information and digital literacy teaching.
Build on previous literature reviews by Schilling and Applegate (2012) and Erlinger
(2018).
Aimed to draw conclusions about how to choose methods for individual evaluative
practice.

5. Methodology
Searched 4 databases
Screening criteria:
➔ Published post-2000
➔ Teaching intervention - not
measuring students' skills
➔ Definable methodology for
evaluation
Categorised full-text articles

6. Review stats
Teaching intervention type Count Percentag
e
embedded synchronous
31 20.39%
embedded asynchronous
20 13.16%
non-embedded synchronous
14 9.21%
non-embedded asynchronous
18 11.84%
embedded drop in
1 0.66%
non-specific/mixed
68 44.74%
Evaluation Count Percentage
single 68 44.74%
multiple 84 55.26%
Evaluation method Count
session-specific survey 67
end of module survey 15
focus group 9
single self-assessment of skills 2
pre- and post- self-assessment of skills 15
single skills test 10
pre- and post- skills test 36
assessment results/grades 17
analysis of student work 33
student reflective writing 13
student observation 3
interviews/1:1 meetings 6
electronic resource stats 9
in-class formative assessment/feedback 9
other 24
Frequency of teaching intervention types
Single or multiple evaluation methods
Frequency of evaluation methods

7. Key limitations of evaluation methodologies (1)
Experimental design
➔ Lack of control group
Impact of voluntary attendance
➔ Don't know why students choose to come
➔ Students may come to one or many sessions - hard to evaluate as a series
Collecting subjective views of students
➔ Using only surveys makes it hard to compare feedback with data on students'
learning

8. Key limitations of evaluation methodologies (2)
Survey issues
➔ Low response rates
➔ Unclear terminology in survey - e.g. what does "helpful" mean to each student?
Time and student recruitment
➔ Getting students to take part
➔ Time for focus groups and interviews
➔ Time for keeping skills/self-assessment questions up to date

9. Review conclusions
Consider "contextualised" evaluation (Nygaard and Belluigi, 2011).
Use mixed methods that complement each other.
Issues of longitudinal data and ongoing evaluation as technology and platforms change.
Need further research into evaluation as day-to-day practice, rather than for specific
research projects, where time and experimental design differ.

10. Impact on our work
Re-evaluation of survey questions.
Use of in-session formative assessment/self-assessment.
Collaborate with academics to get more meaningful evaluation of our embedded
teaching.

11. Obligatory pandemic reflection
Pivoted to Google Forms from a paper form - response rates
declined.
Attendees more likely to leave session early.
Should we evaluate online and face-to-face teaching in the
same ways, or different?

12. What is the future (of feedback)?
➔ Could we move away from the
survey? Do we want to?
➔ Does there need to be a wider
move in higher education away
from the survey?
➔ What is the purpose of feedback
going forward? Can that change?

13. References
Erlinger, A. (2018). Outcomes assessment in undergraduate information literacy instruction: a systematic
review. College and research libraries, 79(4), 442-479.
Nygaard, C. and Belluigi, D.Z. (2011). A proposed methodology for contextualised evaluation in higher
education. Assessment and Evaluation in Higher Education, 36(6), 657-671.
Sanders, M. (2011). Setting the scene: the four domains of evaluative practice in higher education. In
Sanders, M., Trowler, P. and Bamber, V. (Eds.), Reconceptualising evaluation in higher education: the
practice turn. Berkshire: McGraw Hill, pp. 1-17.
Schilling, K. and Applegate, R. (2012). Best methods for evaluating educational impact: a comparison of the
efficacy of commonly used measures of library instruction. Journal of the Medical Library Association,
100(4), 258-269.

14. Siobhan Dunlop
Teaching and Learning
Advisor
University of York
Email:
siobhan.dunlop@york.ac.uk