Presented at QQML 2013: Qualitative and Quantitative Methods in Libraries International Conference. Rome, Italy.

1. Developing a Mixed Qualitative and
Quantitative Research Design to
Inform Library Policy Decision-
making
David P. Kennedy
RAND Corporation
Marie R. Kennedy
Loyola Marymount University
QQML 2013: Qualitative and Quantitative
Methods in Libraries International Conference

2. Outline
• Background
• Overview of Cultural Domain Analysis
• Results of use of two methods that combine
qualitative and quantitative approaches
– Free Listing
– Pile Sorting

4. Systematic Data Collection

5. Cultural Domain Analysis
• 2 Key Methods
–Free Listing
–Pile Sorting

6. What is a Cultural Domain?
• Cognitive Domain:
– How people think (cognition) about categories
(domains)
– Typically named
• examples: illnesses, vegetables, countries
– Categories contain items
– Items have semantic relationships with each other
• Examples: X is part of Y, X is similar to Y, X causes Y, etc.
• Some may be categories themselves
– For example
– Books -> reference books -> dictionaries -> types of dictionaries ->
and so on
• When it is Cultural?
– Shared

7. Cultural Domain Analysis at
William H. Hannon Library

8. Cultural Domain Analysis at
William H. Hannon Library
• The WHH Library is the library at
Loyola Marymount University (LMU)
• Private, Catholic
• In Los Angeles, California (USA)
• Primary attention given to undergraduates
• Strong student employee program in the library

9. Goal: Efficiently and effectively determine
items in a domain and understand how
they are related to each other.
• Step One: Free Lists
Evaluating a Cultural Domain

10. Free Listing
• Goal: Native categories and terminology
• Basic idea:
– “Tell me all the <category name> you can think of”
– Typically loosely timed, no questions allowed
– “Grand tour” question
• Contrasts with survey open-ended question
– Open-ended is typically about the respondent:
• What do you like about this product? What ice-cream flavors do
you like? What illnesses have you had?
– Free list is about the domain:
• What ice-cream flavors are there? What illnesses exist? What
are all of the fruits and vegetables?

11. Free Listing:
Cultural Domain of Library Usage
• Systematic: Everyone Given the Same
Question:
• “Think about all of the things that people on
LMU campus do when they use library
services. Off the top of your head, list all of
the ways that people on LMU campus use the
library.” NOTE: Not just what they do!
• Exploratory: How they answer is up
to them

12. Free List Analysis
• Different ways to analyze free list data
– Sort in descending order of frequency mentioned
– Tally average position in lists
– Combine frequency and position to create
“salience” measure
• Software: Anthropac (Visual Anthropac 1.0)
• Goal: Identify core set of items
– Explore patterns in what people say

13. FREE LIST RESULTS

14. Free List Results
• 4 Interviewers
– Library Staff
– 3 Female, 1 male
• 21 Respondents
– Student Workers
– 16 Freshman/Sophomores
– 13 Had worked in library less than 1 year
– 18 Female
• 324 responses
– Collapsed into 62 Unique responses

15. Free List Results: > 50%
Item Freq. % Rank Avg. Salience
Use printers 19 86.4 6.6 0.5
Use computers 17 77.3 6.8 0.5
Use group study rooms 15 68.2 5.9 0.5
Borrow books 15 68.2 5.3 0.5
Study 14 63.6 4.1 0.5
Attend classes 13 59.1 12.1 0.2
Use copier 11 50.0 11.4 0.2
Get research help 11 50.0 8.0 0.3
Spend time in a quiet place 11 50.0 7.1 0.3
Do research 11 50.0 3.3 0.4

16. Free lists: Scree Plot, % mentioned
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
• No sharp scree fall
• No strong core set of Items
• Short Tail
• Relatively few
idiosyncratic answers
• Top electronic resource
13.6%

17. Goal: Efficiently and effectively determine
items in a domain and understand how
they are related to each other.
• Step Two: Pile Sorting items identified in
free list interviews
Evaluating a Cultural Domain

18. Pile Sorting Example
• Fruits and Vegetables

19. Free List Text to Cards
Apple
Pear
1
3Corn
Banana
2
4

20. Cards to Piles
Apple
Banana
Breadfruit
Kumquat
Currant
Pile 1: “Fruit I eat
at lunch”
Pile 2: “Fruit I
don’t like”
Pile 3: “Fruit I’m not sure
what they are”
Orange
Coconut

21. Pile Sort Analysis
• Different ways to analyze pile sort data
– Produce Aggregate Proximity Matrix
• How often did each item end up in the same pile
across all pile sorters
– Multivariate Analysis Techniques
• Multidimensional Scaling (MDS)
• Cluster Analysis
• Consensus Analysis
– Compare results of quantitative analysis to
qualitative descriptions of piles

22. Pile Sort Analysis
• Software: Anthropac (Visual Anthropac
1.0)
• Goal: Identify Patterns in How People
Group Items
• 35 items from free list – top frequency
– 5 additional items not named: electronic
resources
• Better understand how electronic resources fit into
cultural domain
• Provide insight into how to better market electronic
resources
• Increase their salience

23. PILE SORT RESULTS

24. Pile Sort Results
• 4 Interviewers
– Library Staff
– 3 Female, 1 male
• 21 Respondents
– Student Workers
– 12 Freshman/Sophomores
– 13 Had worked in library less than 1 year
– 18 Female
• 102 piles total across respondents
– Nearly 5 piles on average
– Range 2 to 8 piles

25. Use computers
Use printers
Borrow books
Use group study rooms
Attend classes
Study
Get research help
Use copier
Do research
Spend time in a quiet place
Go to the Café
Use scanners
Read
Work on group projects
Conduct archival research
Attend events
Access the Internet
Do homework
Employment
Borrow DVDs / movies
Watch movies
Borrow technical equipment
Meeting place
Access course reserves
Use presentation
equipment/room
Use media rooms
Use fax machine
Read newspapers
Get technical help
Ask questions
Look at art/exhibits
See the view
Use databases
Read periodicals/magazines
Hang out
ProQuest
EBSCO
Google Scholar
Wikipedia LibGuides
Research
Borrow things and get help
Social and enjoyment
related activities
Stress: .133
High Consensus : Strong
evidence of one culture
MDS and Cluster Analysis

26. Reactions from Data Collectors
• “I was also surprised by students who did not
immediately identify functions in their own areas
of work as things that people do when they come
to the library! (For example, ref desk students
who listed research help only at the very end of
their list, if at all.)”
• “Not a single student separated the ideas of using
the library to explore and gratify personal or
intellectual curiosities or independently
broadening their knowledge of their own major
areas of study as separate from the overall idea
of research.”

27. Conclusions and Recommendations
• Three primary sub-domains
– Research
– Getting help / borrow things
– Socialize
• Electronic resources are not highly salient
elements of the cultural domain of library activities
– Loyola Marymount Campus
– Student workers
• Market resources as aspects of other main
domains
– Peer educators

28. Thank You!
davidk@rand.org
marie.kennedy@lmu.edu