Webinar presented Jan. 14, 2013 for the Libraries Thriving series sponsored by Credo Reference.

1. This presentation was prepared for the LibrariesThriving webinar series sponsored by
Credo Reference, which took place on Jan. 14, 2013.
In the presentation, we explore
• the nature of evaluation in higher education, and how current trends are affecting
the delivery of information literacy instruction.
• the proposition that embedded librarianship offers an effective framework for
delivering information literacy instruction, and also lends itself – more than any
other option – to meaningful evaluation.
The presentation also reviews the definition of embedded librarianship really is, and
discusses some key factors that contribute to applying it successfully. Finally, other
valued roles beyond instruction are discussed.
1

2. Let’s focus on the connections between three elements: higher education,
information literacy instruction, and evaluation.
2

3. Let’s start with the intersection of evaluation and higher education.
Over the past few years, academic institutions have come under increasing scrutiny.
They are being called on to demonstrate their effectiveness, justify their value, and
adapt to radically new circumstances – and they haven’t always been up to the task.
In the abstract of her address to the Library Assessment Conference last Fall, Dr.
Judith Eaton, President of the Council for Higher Education Accreditation, had this to
say, “Higher education’s work in assessment … is not sufficiently responsive to today’s
environment in which greater transparency, reliable evidence of student achievement
and clear and readily accessible information about our performance have taken
center stage.”
3

4. And really we have seen only the beginning. Predictions abound that new computing
and communications technologies, along with new operational models, are poised to
cause a fundamental disruption in higher education. Mark Andreessen, creator of the
first graphical web browser and now a successful Silicon Valley venture capitalist, was
quoted in Wired magazine last May as saying that education was high on the list of
industries to be disrupted.
4

5. No doubt you’re familiar with examples of the disruption taking place – initiatives like
the Khan Academy, the EdX joint venture of Harvard, M.I.T., and now others, the
Stanford University experiments that led to Coursera and Udacity – et cetera, et
cetera. This list is going to continue to grow.
5

6. So it’s clear that university administrators have to sharpen the focus on questions
like: are we doing a good job? Are we worth what people are paying us? How do we
know, and how can we prove it? What changes will enable us to do even better?
They’re going to ask us and every other unit on campus too. They recognize that
expectations are rising continually, and the status quo is not an option.
6

7. Now let’s look at evaluation and information literacy instruction.
7

8. Academic libraries are no strangers to disruption. We have been living with disruption
of our traditional operational model for just exactly 20 years now. It was in 1993 that
Andreessen’s Mozilla graphical web browser, along with the opening of the World
Wide Web to all users, began to disrupt society’s information infrastructure in the
most profound way since Gutenberg. The resulting disruption is evident in this graph
of reference transactions reported by members of the Association of Research
libraries in the first decade of the 21st century. The numbers are down 45% in 10
years.
8

9. As Michael Stephens put it in Library Journal, People do not think of the library first
when they need information.
(“Stuck in the Past.” LJ Apr 15, 2011, p. 54)
9

10. We’ve responded to the disruption in a variety of ways: creating and integrating
digital collections; repurposing our buildings; and taking on an increased instructional
role are some of the major changes we have introduced. Information literacy
instruction has been an effective response by academic librarians to the disruption
we have been facing.
10

11. But we haven’t done so well when it comes to evaluation. The authors of ARL Spec Kit
318 stated in 2010 that they had found “shockingly little work that focuses on
investigating whether use of library resources and services correlate with measures of
success for library users.” Yet that’s exactly what our institutions are demanding of us,
and will continue to demand, as society and new competitors demand it of them.
11

12. So, still we are left with the questions – the same ones our parent institutions are
facing: and not very good ways to answer them.
12

13. Note that what we need here is not just more evaluation, but different evaluation. To
help us analyze the options for evaluating our work, I’d like to cite Philip Kotler, of
Northwestern University, Kellogg School of Management. Kotler is widely regarded as
the dean of academic thinkers in the field of Marketing. He identifies three levels of
evaluation, which he terms Output, Outcome, and Impact.
Adapting these three levels to today’s discussion, I’d like to offer these definitions.
13

14. First, Output. We might expand this to include Inputs and Activities as well. When we
measure our budgets, or the number of items in our collections, or the number of
items we buy, or catalog, we are measuring inputs, activities, and outputs. These
measures can tell us something about our value, indirectly, but their relationship to
value is tenuous. They don’t tell us if those items we acquire, catalog, and hold are
used; or if the use benefits anyone.
14

15. Then there are Outcomes. Outcomes tell us how people responded to our actions. In
terms of traditional library services, measures like the number of circulations, or
database accesses, or reference questions, or interlibrary loan transactions are a form
of Outcome measurement. They tell us that the students and faculty we have tried to
reach used our services to a greater or lesser degree. Surveys are generally a form of
Outcome measurement as well. In surveys, we often ask whether our audiences are
familiar with our services, whether they have used them, and what they think of the
quality and effectiveness of our work. These are Outcomes.
15

16. Finally there are the Impacts. Impacts tell the story of how our work has changed the
world. Sadly, in traditional library work we rarely are able to gauge the Impacts of our
work. We don’t know if a book that was checked out got read. We don’t know if
reading that book helped the student get an A on the paper, or an A for the course.
We don’t know if a great article retrieved from one of our databases – or even the
help of a reference librarian – changed the student’s major, or career choice.
Edwin Perry was an academic reference librarian for 43 years. As he wrote in his
article in the December 2011 issue of Searcher magazine, as a reference librarian
*quote+ “you rarely learn the end result of your work. Did your help really help to
produce the desired results? If you are lucky, you will experience a few situations
where you get the answer to that question.” *end quote+ … But a few situations aren’t
enough for the kind of evaluation demanded of us today.
But a few situations aren’t enough for the kind of evaluation demanded of us today.
What we need to do is to align our goals with the instructional goals of the courses
and programs of our institutions, and ultimately with their overarching goals for
student learning and achievement. That way, our work will be evaluated as part of
the overall institutional evaluation strategy. So, how do we do that?
16

17. To develop an answer, let’s start by analyzing information literacy instruction. Then
we’ll connect up IL instruction with evaluation.
Clearly, information literacy instruction is meeting a need. It’s needed because in
some ways that disruption of our society’s information infrastructure that we
mentioned earlier has made it harder, not easier, to get the right information at the
right time. So, learning about instructional design, developing instructional programs,
and delivering them have become hot skills for academic librarians.
There are several different ways that we can analyze the options for information
literacy instruction. I’m going to label three important dimensions as the Technology
dimension, the Mode dimension, and the Structural dimension.
17

18. Technology is probably the most obvious. We have an ever-proliferating array of
technologies to choose from when we are designing IL instruction. Here’s a selection
of the ever-multiplying array of tools at our disposal.
18

19. Compared to the growing array of technologies at our disposal, what I’m calling the
Mode dimension is pretty simple. We can either deliver instruction when everyone is
together at one time – synchronously – or we can design instructional programs that
allow students to work through the instruction whenever they want to –
asynchronously. Similarly, we have options for bringing everyone together in one
place, or conducting our instruction across a distance, with remote students: co-
located, or dispersed.
19

20. Finally there is what I’m calling the Structural dimension. What I have in mind here is
the administrative structure under which we deliver our IL instruction. There are
three main options. The first is ad hoc delivery. The most traditional form of ad hoc
instruction takes place when we turn a reference interview into a teachable moment,
and use the student-initiated interaction to conduct a little training exercise in
research methods. Beyond that, we can post tutorials on the web, and promote
them, and hope students use them. We can offer workshops and seminars. All of
these, because they are more or less informal and voluntary, I group together under
the label “Ad Hoc.” Next is the standalone, formal course. This can be either non-
credit or for-credit; mandatory or voluntary. But it’s a formal learning program that
students register for, that requires them to go through a set of experiences, and
includes some sort of formal evaluation. It’s “standalone” because it’s unilaterally
administered and led by the library organization. Then, finally, there’s the form of
instruction that’s commonly referred to as Embedded. With the Embedded structure,
IL instruction is incorporated into an academic program. Typically, this is at the level
of a specific subject course – so it’s sometimes called course-embedded. However, it
doesn’t have to be at the course level. Thus, there’s some overlap between the
standalone and embedded models. That is, you might have a separate information
literacy course, but one that is developed to meet the needs of a specific academic
program and is tailored to the needs of that program.
20

21. So, these are the three dimensions we can use to analyze approaches to IL
instruction: Technology, Mode of instruction, and administrative Structure. The
choices we make on each dimension can affect the success of our work. But the point
I want to emphasize is this: They’re substantially independent of each other.
In particular, it’s critically important to recognize that embedded librarianship isn’t a
technology, or a mode, and it isn’t dependent on any technology or mode of delivery.
You can be an embedded librarian for a distance course and do all your work in a
course management system, or you can stand up in a classroom and lead a class
session. Embedded librarianship is something entirely independent of technology or
mode.
21

22. Obviously, technology, mode of instruction, and structure are all going to affect how
you evaluate your instruction. But let’s focus on just the structural dimension, so we
can examine how the evaluation differs among the options of Ad Hoc, Standalone,
and Embedded. I’ll use a series of tables to illustrate the issues we face. Remember
from our earlier discussion that it’s impacts we are truly after. Outputs and outcomes
have their place, and I’m not dismissing them, but ultimately what we also need to
show is that our work has an impact on student success, in terms of course goals and
institutional goals for student achievement.
22

23. We’ve already discussed the point that traditional ad hoc instruction in the reference
interview doesn’t give us much insight into outcomes and impacts. The same is true
for ad hoc tours, orientations, and guides. Web-based tutorials do offer at least the
possibility of assessing student learning, through the use of self-administered quizzes.
However, almost all the ones I’ve seen are pretty simplistic. Usually you can just guess
the answer to a multiple-choice question – and if you guess wrong, you get to guess
again, so you “pass” the test. Plus, students taking ad hoc tutorials are often
anonymous, so there would be no systematic way to relate their use of the tutorials
to academic results. That’s the reason why I’ve grayed out the Impacts of web-based
tutorials in the table. It’s pretty hard to tell if we’ve “changed the world” – that is,
effected a lasting change in the students’ capabilities – from ad hoc instruction.
23

24. When it comes to standalone instruction, the situation is better. A formal course,
whether non-credit or for-credit, enables the librarian to set up some form of
evaluation of student learning. A for-credit, graded course might be the strongest
candidate for demonstrating an impact, because it affords the librarian the
opportunity to make a formal, recorded evaluation of student achievement. If the IL
course is truly standalone – not connected to the needs of an academic program, or
to overall institutional goals and mission – then we can correlate IL course
performance with overall academic performance.
24

25. With course-embedded instruction, we get essentially the same measures as with
Standalone Instruction. But we also get a few more. Since the IL instruction takes
place as part of the course, course assignments can be structured for students to
apply, and be evaluated on, Information Literacy skills. The immediate application of
the skills in context, and evaluation, provides a powerful learning and evaluation
opportunity that’s missing from the Standalone model.
Also, subject faculty gain direct, immediate insight and appreciation for the impact of
IL instruction. When it works well, they see the immediate improvement in student
performance. This generates a “virtuous cycle” in which they then become advocates
for embedded IL instruction, and reinforce it. There are examples of this in the
literature and I’ve encountered them in my own research.
25

26. And here’s just one vivid example – an observation by a Biology professor I
interviewed in the course of my research. She had worked with an embedded
librarian in the first-year course and then had the following perception when she saw
some of the same students again as sophomores.
“The information literacy skills developed in the Great Problems course *a first-year
course with an embedded librarian] stay with the students and enable them to
perform better *as sophomores+.”
This instructor has perceived a direct connection between a librarian’s effective IL
instruction and ongoing student academic performance. There are other examples
like this in my own research and other literature as well. It seems to me that that is
the kind of change we are trying to create in the world as a result of our work – and
the kind of impact that we need to show to our academic administrators and
institutional leaders.
I do want to emphasize that general correlations have their place. But, as I’ll observe
in another context in a few minutes, direct perception and support from subject
faculty are extremely powerful. You want them on your side!
26

27. Now let’s explore in more depth the nature of embedded librarianship, and how it
affords us opportunities to strengthen both our work and our ability to evaluate our
impacts.
27

28. So here’s my definition of embedded librarianship:
An embedded librarian is one who develops strong working relationships with
members of a team or community; develops shared objectives with the community
and responsibility for achieving them; and makes customized, highly-valued
contributions to the team, which play an important role in the success of the team
and the institution. Applied to IL instruction, this means that the embedded librarian
builds strong working relationships with instructors and students. The relationships
lead to shared understanding – the librarian’s understanding of the course and
academic requirements; the instructor’s understanding of the role of information
literacy in the course. From the basis of this understanding, the two can collaborate
effectively to provide a powerful learning opportunity for the students. For the
librarian’s part, this means customized, relevant professional work. It’s these
elements that lead to achieving the learning objectives of the course and the
outcomes that the institution wants to produce for the students.
28

29. To develop this further, here are five key factors that differentiate embedded
librarianship from traditional librarianship.
29

30. As I talk to academic librarians and read accounts of embedded librarianship, I see a
considerable number of so-called “embedded IL instruction” programs that don’t fully
achieve the promise of embedded librarianship. There’s a sense that something is
missing. So, now I’d like to address what works and what doesn’t. I’ll focus on two key
factors.
The first is the relationship between the librarian and the subject instructor, and the
second is the key role of leadership on the part of senior library adminstrators.
30

31. Sometimes just because there’s a link to a librarian, library resources, or library
tutorials, in the course management system, or the librarian is able to participate in
CMS-based activities, the instruction is considered “embedded”. Sometimes a one-
shot lecture is moved from the library to the classroom, and it’s considered
“embedded”. Sometimes librarians are designated (by the library administrator) as
embedded, but instructors fail to engage. I recently read an account of a librarian
who quote embedded unquote a video introduction into some courses in a course
management system. Only she didn’t tell the instructors ahead of time that she was
doing this. Guess what? Some of them weren’t too happy about it. That was not
embedded librarianship!
So what was lacking in that example, is lacking in a substantial number of IL
instructional initiatives that aren’t really embedded: the fundamental collaborative
relationship that is the prerequisite for embedded librarianship. Or, the relationship
may not have matured into a true collaboration.
Here’s a set of questions that can help you evaluate the strength of collaboration
between a librarian and a subject instructor.
31

32. If all the above sounds like a lot of work, it is. So, how are we going to get all of that
done? That’s where the second factor comes in. It is effective strategic leadership.
This probably involves the senior library administrator, though it may involve others
as well. In my research, I’ve seen counterproductive leadership behavior ranging from
passivity to outright undermining of embedded librarianship. On the other hand, I’ve
seen examples of active and effective strategic leadership. Here are four key elements
of this effective leadership.
32

33. I promised to close with some observations about other roles for librarians in higher
education. That might seem a bit disconnected from the emphasis on evaluation of
information literacy instruction in higher education, but it’s not. The connection is
that final point about evaluating our work by how we contribute to our parent
universities.
There’s a general formulation of the mission of institutions of higher learning, and it
incorporates three things: teaching, research, and service.
So far, all we’ve talked about in this presentation is the teaching mission. It’s
important to be sure, but what about research, and service? Increasingly, academic
librarians are finding ways to embed themselves in these activities.
33

34. Many academic libraries are getting involved in curating research data. Embedded
librarians are in a position to contribute a great deal to these initiatives. Because of
their relationships with faculty and researchers, they understand the data and can
contribute their understanding to its management.
Another contribution that spans teaching, research, and service is knowledge
management. A higher education trend we haven’t talked about today is the trend
toward interdisciplinarity. In all aspects of academic life, the integration of disciplines
is increasingly important, and librarians are well positioned to contribute.
Academic librarians are also able to leverage their embedded relationships to take a
leadership role in service.
34

35. These are just a couple of examples of the trend. Barbara Dewey, then of the
University of Tennessee, gave an enduring, comprehensive vision of the potential in
her 2004 article in “Resource Sharing & Information Networks.” She wrote:
“The embedded librarian, who is truly integrated into the academic, administrative,
athletic, cultural, research, teaching, and learning arenas of the university, provides
quality and depth to the total campus experience.”
That’s what we’re after. So, when you’ve established your embedded information
literacy instruction program, and you’ve demonstrated its success in terms of impacts
on the university mission – don’t think you’re done. As an academic librarian, you
have more opportunities awaiting you!
35

36. And now I’d like to close by thanking you for your attention. I’d also like to thank
Laura and the Credo Reference organization for sponsoring today’s session. If you’d
like to know more, here are three ways to follow up:
1. Feel free to email me. I love to hear from those of you who are doing this work, or
thinking about it. I’d love to hear from you.
2. Check out my blog at www.embeddedlibrarian.com . I only post when I have time
and something to say, so if you subscribe to it you won’t be inundated.
3. And finally, you might be interested in my book, published last summer by
Information Today. The second half of it is an in-depth guide to developing and
sustaining embedded librarianship, based on my research and experience in the field.
I hope it may be helpful to you.
36

37. Credits
37