Plagiarism and AI tools: an example of linking information- and digital liter...
Dodd - The information literacy impact factor: how to measure value
1. The Information Literacy Impact
factor: How to Measure Value
Lorna Dodd
College Liaison Librarian
UCD Library | Dublin 4.
email: Lorna.dodd@ucd.ie
Tel: +353 (0)1 716 7074
www.ucd.ie/library
2. Outline
Why conduct a survey of module coordinators?
Review of the Literature
Rationale
How the survey was conducted
Survey results
Implications for the future
Conclusions
Lorna Dodd: LILAC 2013
5. Looking Back
Large University
No single strategy for the development of
Information Literacy
Heavy load of information skills delivery
Driven by individuals
Sometimes evaluated but rarely measured
No clear indication in meeting learning outcomes
(Dodd, L. & Kendlin, V., 2010)
Lorna Dodd: LILAC 2013
6. Looking Forward…
New Library Strategic Plan 2010-2014
“monitor, measure and evaluate the Library’s teaching and
learning strategy and activities.”
(UCD Library, 2010)
Lorna Dodd: LILAC 2013
7. Looking forward
Realignment of staffing structure
From approx 17 Liaison Librarians to 6 College
Liaison Librarians
Move from module to programme approach
Move from responsive to consultation
Measure impact on student learning
Lorna Dodd: LILAC 2013
9. Literature Review (1)
A lot available discussing why it is important to measure
impact
Much of what has been written focuses on Library use rather
than instruction.
(Stone 2011; Poll & Payne 2006; Schilling & Applegate 2012).
Limitations of traditional organisational approaches at
measuring success and meeting targets.
Most libraries focus on process and output indicators rather
than measuring impact.
Measuring impact on student learning requires systematic
evaluation of training
(Markless & Streatfield 2006; Schilling & Applegate 2012))
Lorna Dodd: LILAC 2013
10. Literature Review (2)
Several problems arise when measuring impact.
Data protection rules
Differing value of services for specific groups
Difficulty in measuring long-term impact
Time consuming nature of measuring impact of Library
instruction.
Important to use results of any study conducted to measure
the impact of Library services
Benchmarking activities
Improving services
Justifying resources used for services
Campaigning for increased funding.
Common/Successful ways to measure impact of Library
instruction
Surveys; Pre and post-tests; Self-assessments; Behavioural observations.
(Poll & Payne 2006)
Lorna Dodd: LILAC 2013
11. Literature Review (3)
2006 LIR/SCONUL measuring impact initiative
Developed impact process
Identified key performance indicators
Used to evaluate the effectiveness of interventions
supporting teaching and learning
Research provided information regarding emerging
issues throughout the process.
Results included:
Improved relationships with academic community
A raised profile for Library staff
Better understanding of how the Library can support
academic programmes.
(Markless & Streatfield 2006; Blagden, 2005)
Lorna Dodd: LILAC 2013
12. Literature Review (4)
2012 CONUL Information Literacy Survey
Gathered feedback on the impact of information literacy
instruction
Found library instruction had a positive impact on
students’ skill development
Information Literacy often assessed
Targeted respondents
Academics already fully engaged
First year students only
Small sample from each institution
(CONUL Advisory Committee on Information Literacy, 2013)
Lorna Dodd: LILAC 2013
13. Rationale
We needed to: Problems:
Make academics aware of new Measuring impact
approach retrospectively
Create a strategy to get Perceived impact is
academics to embrace new
subjective
approach
External factors influence
Identify what has been
successful to date development of information
skills
Collect feedback from those
not actively engaged with the More likely to opt for
Library options that are familiar
Identify what methods would
be useful going forward
Lorna Dodd: LILAC 2013
14. Survey
Seeking information on modules
Two parts:
1. Evaluating information skills delivery to date
2. Identifying what would be useful in the future
Survey Monkey
11 questions
Mainly quantitative
Multiple choice questions
Measurement/ranking questions
Lorna Dodd: LILAC 2013
15. Survey
Introduction:
Level of module – undergraduate/taught postgraduate
If there has been Library instruction
Part 1:
Kind of instruction
Impact of instruction on students’ abilities
Part 2:
Should all students have library instruction
When should library instruction happen?
Which services should be developed
Free Question
Lorna Dodd: LILAC 2013
16. Promotion
Approx. 700 module coordinators
Only possible to target academic staff
Not all module coordinators are academic staff
and not all academic staff are module
coordinators
Targeting email – approx 1,900
College Liaison Librarian contacts
Three weeks
Lorna Dodd: LILAC 2013
17. Survey Results
Type of instruction
Impact of instruction
When and who?
Preferred services for the future
Qualitative feedback
Lorna Dodd: LILAC 2013
40. Influence of instruction had on preference?
Instruction First Choice Second Choice Last Choice
had
Tour Subject Generic workshop Tour
Specific Tutorial
Workshops Online Video
Lecture Subject Generic Workshop Tour/Online
Specific Tutorial Video
Workshops
Workshop Subject Generic Workshop Tour
Specific Tutorial
Workshops
No Subject Subject Specific Tour
Instruction Specific Workshops
Workshops
Lorna Dodd: LILAC 2013
44. “One of the best approaches would be to create
materials that are reusable and for any student,
regardless of level. Just because a student moves up a
level doesn't mean they've acquired all of the learning
they should have”
“…really good online video materials, from complete
intro to specific and advanced, and available on
demand is the way to go”.
“Tutorials (tutors would be trained by Library
staff)”
Lorna Dodd: LILAC 2013
46. Comments Themes
20% (53) of those who completed the survey
made comments
In line with our new strategy
Online options
Constructive comments & suggestions on current
and future services
Complimenting services to date
Comments on the survey
Lorna Dodd: LILAC 2013
47. Comments
5 respondents made comments inline with our
new strategy
Programme Approach
Reusable Learning object and Online tools
6 respondents made comments about proposed
future instruction
Lorna Dodd: LILAC 2013
48. Comments
12 respondents made comments about current
services and changes
Problems with current service:
“…same 'introductory' talks by librarian using lecture slots of
various modules for at least three times (the third time he
skipped as he found it ridiculous)
“Students just skip the guest lecture.”
Lorna Dodd: LILAC 2013
49. Comments
Benefits of current service:
“…requires personal contact with a friendly face early on.
“The current reduction in library services is a real
retrograde step especially with the explosion in available
information sources”
Lorna Dodd: LILAC 2013
50. Comments
Over half (28) complimented current strategy
“we really love our librarians; thank you”
‘Campaigning’ for traditional service
Acknowledging individuals
Recognising value of information literacy to students
Lorna Dodd: LILAC 2013
51. “…it makes a noticeable difference
in the quality of assignment
students produce. The Library
Tutorial is typically rated as one of
the most important lectures they
have in that Stage 1 module.”
Lorna Dodd: LILAC 2013
53. Impact of Instruction
Definite benefit in Library instruction
Active learning environment is preferred
Stronger evidence in the discovery and use
of resources
Remains difficult to identify the ‘real’
impact
Lorna Dodd: LILAC 2013
54. Timing of Instruction
Most module coordinators feel all students
should have Library instruction
Instruction for students at every stage is desirable
Taught Postgraduates
Beginning and middle of Semester 1
Undergraduates
Stage 1
Stage 3
Lorna Dodd: LILAC 2013
55. Preferred Services
Active learning sessions tailored to subject needs most
preferred
Tours perceived as ineffective
Questions:
Asked to identify how effective instruction was at improving
students’ information literacy skills.
When they request instruction, do they usually consider this?
Or do they feel students need to know about the library rather
than what they need in order to transition from 2nd level, develop
researching skills etc.
Lorna Dodd: LILAC 2013
56. Questions
Did respondents choose workshop environment over
online version because they are more familiar?
If so, then why did the majority of those who had no
instruction choose workshop?
If delivering workshops on this scale is unsustainable,
what can we do ensure effectiveness?
What work do we need to do in terms of marketing?
Lorna Dodd: LILAC 2013
57. Conclusion
Shared understanding between Library & Academic in:
Benefit of Instruction
Taking a programme approach
Ensuring all students get an equal opportunity (core
modules)
Difference in:
Most appropriate method of delivery
Lorna Dodd: LILAC 2013
58. References
Blagden, P. (2005) ‘The LIRG/SCONUL Measuring Impact Initiative: Overview of
phase 1 impact projects’, Library & Information Research (LIR) vol. 29 (91)
Dodd, L. & Kendlin, V. (2010) ‘Damned if we do and Damned if we don’t: How to
address sustainability in the delivery of information literacy components in UCD’:
Librarians' Information Literacy Annual Conference (LILAC), March 29-31, Limerick,
Ireland.
Markless, S & Streatfield D. (2006) ‘Gathering and applying evidence of the impact of
UK university libraries on student learning and research: A facilitated action research
approach’, Journal of Information Management vol. 26 pp. 3-15
Poll, R & Payne, P. (2006) ‘Impact measures for libraries and information services’,
Library Hi Tech vol. 24 (4) pp. 547-562
Schilling, K. & Applegate, R. (2012) ‘Best methods for evaluating education impact: a
comparison of the efficacy of commonly used measures in library instruction’, Journal
of the Medical Library Association, vol. 100 (4) pp. 258-269
Stone, G. et al (2011) ‘Does Library use affect student attainment? A preliminary
report on the library impact data project’ Liber Quarterly vo. 21(1) pp. 5-22
UCD Library (2010) ‘UCD Library Strategic Plan 2010-2014’
http://www.ucd.ie/library/news_publicity/showcase_strategy /
Lorna Dodd: LILAC 2013
Spoke to colleagues in teaching and learning for advise on best tool to collect this information We currently don’t offer a wide range of online options. Academics more likely to say they would prefer the option they are familiar with
Tours good for using better & broader range resources of improved ability at finding information Significant proportion feel tours are not good for citing and plagiarism awareness Disagreement regarding students being better at everything
Similar to tours at good: Lectures good for better & broader range of resources & improved ability at finding information Less disagree that students more aware of citing & plagiarism Disagreement regarding students being better at everything
Significantly better in all aspects Most noticeable is that very few disagree with any Stage 2 significantly more positive
Feel Taught postgrads get more out of tour in all aspects Significant difference to UGs is that no disagreement in all aspects except plagiarism
Similar pattern to tours
Similar pattern Overall most effective at improved ability at finding information
Bespoke sessions Specifically tailored – much better at achieving all categories
Of 268 people who completed survey
Nearly everyone who had library instruction believes that ALL students should have it Whether they have had library instruction as slight impact on whether they think all students should have it
Of 268 people who completed survey
Subject specific workshop Even split between others
Much more even split Face-to-face with library professional still preferred
Similar pattern to undergraduates
Similar pattern to undergraduates
Not everyone rated from 1-6 so results a little unreliable Think that explains why subject specific workshops got more last choices than any other Same pattern for those who had tours, lectures, workshops, no instruction
Fear of loosing these services
Fear of loosing these services
Definite benefit of library instruction and should therefore continue in some way Active learning preferred – presents a challenge More emphasis on impact we can have on learning how to cite and avoid plagiarism Measuring impact still very subjective
Agreement that all students should have the opportunity If resources are limited need to focus on specific groups – stage 1 & 2
Does this have implications for the lower impact in students understanding how to cite and avoid plagairsim?