Carl 2014 slides_gotime

Letting Users Lead:
Analyzing Search Queries & Relevancy
in USC’s Web-Scale Discovery Tool
California Association of Research Libraries, 2014
April 5, 2014
Beth Namei, University of Southern California
Christal Young, University of Southern California

Web Scale Discovery Services in a nutshell
image: http://www.colinburnett.com/wp-content/uploads/2014/01/livingunderrock.jpg

The hype - Have we chosen wisely?
image: http://jessicaknauss.blogspot.com/2012/09/the-grail-knight-as-inspiration.html

Why USC got Summon
#1: To provide better discoverability of our subscription
and purchased content (via a unified access point)
#2: Provide relevant results to our users (most
urgently to provide relevance ranked results for items in
our SIRSI OPAC)
#3: To provide a better user experience with the
library’s website

Our study
We re-executed a sample of Summon search
queries to see how successful our users were in
retrieving relevant results.

April 2010 - Pre-Summon homepage

July 2010 - Summon is launched as the default tab

July 2012 - Our current Summon-centric homepage
(Catalog tab is removed)

Motivation for our study
• There were a lot of anecdotal complaints.
• To get evidence about how successful Summon
was in leading users to relevant sources.
• To learn more about user search behavior with this
single search box

our
methodology
image: http://www.rsc.org/chemistryworld/issues/2007/december/thechemistrysetgeneration.asp

Transaction Log Analysis (TLA)
A transaction log is a history of actions executed in a system.
TLA involves looking at the data captured in a transaction log
to investigate interactions between users and a search tool.

Advantages of TLA
• Unobtrusive
• Large quantities of data that can reveal large-scale
patterns
• Low cost

Limitations of TLA
• Does not capture contextual information such as the
user’s emotions, motivations, intentions and needs.
• Does not capture demographic information.
• Does not capture user satisfaction with the
overall search experience.

Summon searches in Fall 2013: 1,243,250
# of unique searches: 184,076
Our sample: 384 searches
Margin of error: +/- 5%

Defining success (and failure)
image: http://abovethelaw.com/wp-content/uploads/2013/12/thumbs-up.jpg

Success = Relevant results
"We are used to hearing people talk about the 'simple
search box' as the goal” of Discovery services.
But a simple search box has only been one part of the
Google Formula. PageRank has been very important in
providing a good user experience, and Google has
progressively added functionality as it included more
resources in the search results" (Dempsey, 2012, 4).

Information overload
“Surveys of...users
reveal a consistent
theme: most are
overwhelmed and
confused by the
disorganized flood
of information”
(Riley 24).
image: https://www.flickr.com/photos/youreyestellies/7984526358

“Surveys of Internet users reveal a consistent theme:
most are overwhelmed and confused by the
disorganized flood of information. As a result,
librarians have an opportunity to carve their niche as
the de facto information navigators of the Digital
Age”
Riley, Margaret. "Riley's Guided Tour: Job Searching On The Net." Library
Journal 121 (1996): 24-27.

Information used to
be scarce….
Image: http://allaboutalpha.com/blog/wp-content/uploads/2011/04/iStock_000014587595XSmall.jpg

But today our attention
span and time are scarce
while information is
abundant and easily
accessible.
“There are so very many
search engines now and so
much information; I preferred
the ‘good old days’ when there
weren’t so many ways to
access information”
- USC Journalism faculty
member.
image: http://gcn.com/articles/2013/09/30/smarter-cities-cloud.aspx

Summon’s Relevance Ranking
Looks at:
• term proximity - how close query terms are to one
another.
• term frequency - how often do the query terms appear
in the record
• field weighting - where is the query term found?
Finding query terms in some fields are more important
than others.

Also considers:
• Some content types are boosted over others
o Books and journal articles over newspaper articles and book reviews.
o The Journal over the articles in that journal
• Publication Date - Generally, items with newer publication dates are
favored over older items.
• Citation Counts - Items cited more are boosted
• Local Collections - Content from institutional catalog(s) and repositories
are boosted.
• Content Size - Longer works aren’t necessarily more relevant even though
search terms might appear in them more often

A “Success Engine”
Relevancy Enhancements: “to ensure users don’t miss out on the most
relevant content.”
This includes automatically searching for synonyms, term-stemming and smart
searching of stop-words depending on their importance to the search phrase.
image: http://blog.dappersnappers.com/wp-content/uploads/2012/02/baby-laptop-computer.jpg

Measuring Relevancy in our Study
We used “systems-oriented relevance” to rate the
success of search queries.
We rated the relevancy of Summon’s results without
knowing the context of the original query. We looked at
how well the topic of the search was represented in the
topics of the results retrieved in Summon, Google and
Google Scholar.
- Maglaughlin & Sonnenwald 2002, 328-9.

Inter-rater reliability
mage: - http://www.artofmanliness.com/2008/08/21/manly-feats-of-strength/

Types of searches
Known item searches
Had to be specific enough for
us to recognize a definitive
match. If the search got
numerous matches and was
general, we would likely not
categorize it as a known
item.
Examples:
• marketing alterity moore
• 978-0078026676
• An Empirical Analysis of Cigarette
Addiction
• "happy days are here again" garland
streisand
Keyword/topic/exploratory
searches
Encompassed broad, general or
ambiguous searches. Named
persons were put into this category
as well.
Examples:
• Catherine the Great,
• vulnerability and happiness
• PTSD and Substance use in african
american women
• supersitions

Our Relevancy Rubric
Known Item Searches
Relevant:
1st item in the
list of results
Partially
Relevant:
2nd-10th item in
the list of results
Not relevant:
Not listed in the
first 10 results or
no results
retrieved.
(could mean we don’t
own the item or there
is a user input error)

Our Relevancy Rubric
Keyword Searches
Relevant: ALL search
terms appear in item's title or
record.
All search terms appear to
have a relationship to one
another, they are not just
randomly placed throughout
the title/record.
First 5 items look like a
"perfect"/solid match or
clearly seem to be ABOUT
the topic as identified
through the search terms.
Partially Relevant
Not all search terms are
visible in the title or record of
the items.
At least 3 of the first 5 results
appear to be somewhat
related to the topic as
entered, even if broadly or
tangentially.
Would a user easily recognize
a connection between the
results and the topic as it was
entered?
Not Relevant: At least
four of the first 5 items appear
to be false hits.
Even if one or more of the
search terms (or synonyms of
those terms) appear in the
title, abstract or record,
results appear to be only
about a portion of the search
terms entered and not about
all of them combined.

Summon vs. Google vs. Google Scholar
Known Item
Searches:
For Google & Google
Scholar, the relevancy
was determined by a
match, did not have to
be a full-text match.
Links to Amazon,
Google Books,
WorldCat, imdb.com,
and Youtube were
matches.
Image: http://www.geekwire.com/2013/ibm-takes-watson-cloud/

Top 100 most frequent queries in Fall 2013
Consisted of 23,813 individual searches

Long Tail of unique searches
87% (160,263) of total searches
make up the long tail
Top 100 unique
searches make up
13% of total queries

Image: http://seanarchy.files.wordpress.com/2013/12/educated-poor.jpg

Summon vs. Google vs. Google Scholar
Which do you think did better?
http://bit.ly/summon-faceoff

Successful Searches:
54% of all our sample searches
(204) retrieved relevant results.
Summon’s Overall Relevancy Report Card = F
After the curve = C
Moderately Successful +
73% (273) retrieved partially
relevant - relevant results
image: http://bleedingedge.pynchonwiki.com/wiki/images/b/b8/Dunce-cap.jpg

85% (318) of the sample
searches retrieved
relevant results
Google’s Relevancy Report Card = B
After the curve: A
Moderately Successful
+ Successful Searches:
95% (356) of the searches
retrieved partially relevant
- relevant results.

54% of the sample
searches (203) retrieved
relevant results.
Google Scholar’s Relevancy Report Card = F
After the curve: C
Moderately Successful +
75% (281) retrieved
partially relevant -
relevant results.

Relevancy of all Keyword Searches
n=236

Relevancy of all known item searches
n=139

Failed searches in Summon
32% (45) of all known item searches did not locate the
item being searched for
• 33% (15) of these failed searches are for items USC
does not own
Of the items we do own:
• 57% (17) did not show up due to user error
• 40% (12) did not show up due to a Summon problem
(bad metadata, poor relevancy, not indexed in Summon)
• 3% (1) had irregular characters and found no results

Relevancy of “academic” known item searches that USC
owns
Summon
improved 13%
Google Scholar
improved 15%
Google improved 1%
n=109

Revised Relevancy Report Cards
Google Scholar = F
57% (192) retrieved
relevant results
(up from 53%)
After Curve = C+
79% (272) retrieved
relevant results
(up from 73%)
Summon = F
59% (202) retrieved
relevant results
(up from 53%)
After Curve = C+
79% (271) retrieved
relevant results
(up from 73%)
Google = B
84% (291) retrieved
relevant results
(down from 85%)
After Curve = A
95% (328) retrieved
relevant results
(no change)

User errors
18% (66) of the
searches in our
sample had a
user input error
Image: http://human-error.sarkisozlerik.com/human-error/a-lifetime-by-design.html

“Did you mean?”
Showed up 24 times
• 83% (20) triggered by user input errors.
• 42% (10) of the time the “Did you mean” links took
users to relevant results.

Google automatically redirects searches with errors
Summon lets the user
decide whether to follow
a “corrected” path

Median: 3
Mean: 3.182
Mode: 2
# of words used in Keyword Searches

Impact of # of search terms entered on relevancy

Type of Keyword Searches Executed
n=236

Impact of search type on relevancy

Duplicates
image: http://amarkedman.com/wp-content/uploads/2011/07/Matrix-Clones.jpg
Known item searches:
15 searches had 2 or
more duplicates (11%)
Keyword searches:
51 searches had 2 or
more duplicates (22%)

Linking to full-text
"Linking users to full text as
quickly as possible after
discovery results are available
is a paramount concern"
(NISO ODI Report, 2013, 7).
There were only 3 bad links
(out of 55 known item article
searches)
image: https://flic.kr/p/mqjHgR

What we learned:
Summon has some work to do to improve the
relevancy of its results.
But, it is doing better in other areas….

Summon added
as default search
tab (July 2010)
Catalog search
tab removed
from homepage
(July 2012)

Winning! (sort of…)
image: http://beautelicious.com/2011/09/charlie-sheen-warner-bros-close-settling-wrongful-termination-suit/

Final Report Card:
Relevancy = C+
Intuitive starting place =
Fast =
Bringing users back to the library =
Maximizing usage of collections =

Moving forward - Following our users’ lead
image: http://blog.cityspoon.com/2012/02/08/gathering-followers/

Leadership Strategy #1: Learn and use your
library’s discovery tool
• 3 million searches in 2013!
• Will give you insight into what
users are experiencing, both
good and bad.
• Discovery tools are taking up
prime real estate on many of
our websites.
image: http://www.creativity4us.com/wp-content/uploads/2012/02/blinders-crop.jpg

Leadership Strategy #2: Change what and how
we teach
”Librarians must reconsider training students to use
advanced search features or Boolean logic if students
purposefully choose not to use them or fail to use them
correctly. Rather than teaching students more effective
search syntax, more attention should be placed on
developing critical thinking and evaluative skills"
(Holman, 2011, 24).

Leadership Strategy #3: Be a squeaky wheel
• Many users will get frustrated and abandon the library
without out ever letting us know about problems
• We cannot depend on other people to report problems

Leadership Strategy #4: Imporvise and fall in
front of students
• Show students how to troubleshoot a failed search.
• Show searches with typos or how to revise a search
that gets no results)

"If we think like users (instead of as librarians) it is easy
to understand the frustration. Our tools must seem
broken or outdated to them….Are we in the business of
promoting library databases or the business of helping
users accomplish their tasks?” (Matthews, 2013)
Leadership
Strategy #5:

“The trouble with Summon is that students don’t need to be
taught how to use it, but librarians do”
-Matt Borg, Sheffield Hallam University, 2012

English Faculty Member:
“I want an easy way to find a book with
author and title, and an easy way to move
from that to journal articles if that’s what
I want….”
Religion Faculty Member: “If I need to
search something I don’t go to any USC
search engine, which is totally a waste of
time. I go to Google where I can get
things ten times as fast.”
image: http://www.morvimmer.com/blog/free-download-staples-easy-button/

Leadership strategy #6: Empathize with users
AND colleagues
• Try not to criticize or judge
• Invite skeptical librarians into your classes to watch you
teach w/the discovery service
• Showing vs telling - talking can only get you so far

Common complaints
• “I think it's a cheat. Too many students don't learn basic searching skills that
would make any search better - like planning before typing. They just throw in
anything and take what comes up first” - 3/1/14 Survey of USC Instruction
Librarians
• “It is so imprecise" (Buck & Steffy, 2013, 76).
• Perpetuates "the homogenization of information" - when "everything looks and
feels the same" (Bawden & Robinson, 2008, 181).
• “pandering to the 'principle of least effort'” (Richardson 2013; Meadow &
Meadow 163-4).
• They “impinge upon the development of critical research skills” (Wiles &
Hofmann, 2013, 156).
• “these systems...reinforce unreflective research habits” (Asher, 2013, 6)

Complaining is not a (constructive) strategy
“When you invent something new, if customers come to
the party, its disruptive to the old way….The internet is
disrupting every media industry...people complain
about that but complaining is not a strategy. Amazon
is not happening to bookselling, the future is
happening to bookselling.”
-Jeff Bezos, 60 Minutes, [8:55]. 12/1/2013

Leadership Strategy #7: Solicit feedback AND
then listen
• Look for positive AND negative feedback
• Re-frame negatives as positives or as opportunities for
dialogue

Leadership Strategy #8: Turn negativity to
your advantage
“Engage and transform the
most negative person in your
library system into a productive
team member.” By “converting
the most negative person has a
huge impact on the rest of the
staff” (Cuillier, 2011, 439).
Image: http://www.salon.com/2013/03/12/why_is_francis_underwood_a_democrat/

Leadership Strategy #9: Gather evidence
• Test your assumptions
• Test your colleagues’ assumptions
• Study user behavior, formally and informally
• Assess the tool and then assess it again

Leadership Strategy # 10: Redefining ourselves

References
Asher, Andrew D., Lynda M. Duke, and Suzanne Wilson. “Paths of Discovery: Comparing the
Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and
Conventional Library Resources.” College & Research Libraries, 74.5 (2013): 464-488.
Bawden, D., and L. Robinson. “The Dark Side of Information: Overload, Anxiety and Other
Paradoxes and Pathologies.” Journal of Information Science 35 2 (November 21, 2008):
180-91. doi:10.1177/0165551508095781.
Buck,, Stefanie, and Christina Steffy. “Promising Practices in Instruction of Discovery Tools.”
Communications in Information Literacy, 7.1 (2013).
Cuillier, Cheryl. “Choosing Our Futures … Still!” Journal of Library Administration 52.5 (July 2012):
436–51. doi:10.1080/01930826.2012.700806.
Dempsey, Lorcan. “Thirteen Ways of Looking at Libraries, Discovery, and the Catalog: Scale,
Workflow, Attention.” Educause, December 10, 2012.

Holman, Lucy. “Millennial Students’ Mental Models of Search: Implications for Academic Librarians
and Database Developers.” The Journal of Academic Librarianship 37.1 (January 2011):
19–27. doi:10.1016/j.acalib.2010.10.003.
Maglaughlin, K. L., and D. H. Sonnenwald. “User Perspectives on Relevance Criteria: A
Comparison among Relevant, Partially Relevant, and Not-Relevant Judgments.” Journal of
the American Society for Information Science and Technology, 2002.
Matthews, Brian. “Database vs. Database vs. Web-Scale Discovery Service: Further Thoughts on
Search Failure (or: More Clicks than Necessary?) (or: Info-Pushers vs. Pedagogical
Partners).” Chronicle of Higher Education. Ubiquitous Librarian, August 21, 2013.
Meadow, Kelly, and James Meadow. “Search Query Quality and Web-Scale Discovery: A
Qualitative and Quantitative Analysis.” College & Undergraduate Libraries 19.2-4 (2012):
163–75. doi:10.1080/10691316.2012.693434.
NISO ODI Working Group. National Information Standards Organization ODI Survey Report:
Reflections and Perspectives on Discovery Services, January 2013.

Pan, Bing, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. “In
Google We Trust: Users’ Decisions on Rank, Position, and Relevance.” Journal of
Computer-Mediated Communication 12.3 (April 2007): 801–23.
doi:10.1111/j.1083-6101.2007.00351.x.
Richardson, Hillary A. H. “Revelations From the Literature: How Web-Scale Discovery Has Already
Changed Us.” Information Today, May 2013.
Rose-Wiles, Lisa M., and Melissa A. Hofmann. “Still Desperately Seeking Citations: Undergraduate
Research in the Age of Web-Scale Discovery.” Journal of Library Administration 53.2–3
(February 2013): 147–66. doi:10.1080/01930826.2013.853493.

Carl 2014 slides_gotime

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Carl 2014 slides_gotime

Similaire à Carl 2014 slides_gotime (20)

Dernier

Dernier (20)

Carl 2014 slides_gotime