social pharmacy d-pharm 1st year by Pragati K. Mahajan
Research methodology 101
1. Texas Library Association Annual Conference, 2004
San Antonio, Texas
Research Methodology 101
Yes! You CAN Do a
Research Study in
Your Library
A presentation by
Kate Manuel, Instruction Coordinator and
Susan E. Beck, Collection Development Coordinator New Mexico
State University Library
2. What question do you have about
the research methodologies in
these studies?
1. A study assessing student learning
outcomes in 2 broad categories
(concepts, techniques) by examining
student research journals in 1 section of
an elective information literacy course
in fall semester.
3. What question do you have about
the research methodologies in
these studies?
2. An experimental study that
proposes a fund allocation formula for
academic library collections based on
the following:
average of overall book price + average of overall
serial prices * degree level (10 for undergraduate to
30 for doctorate) / the number of students enrolled
in degree program as majors + the total number of
faculty in the department * three * total number of
students in program.
(OAB + OAS) * D/(Sn +(Fn*3))*Sn
N.B. Not a standard formula
4. What question do you have about
the research methodologies in
these studies?
3. A newspaper article you read just the
other day stated that in a recently
published study done at a major U.S.
university, researchers found that
domestic violence affects 1 in every 4
women.
5. What question do you have about
the research methodologies in
these studies?
4. A 2004 article on a library use and
services satisfaction study that used
as its measurement tool a survey
given to every nth person entering the
library building on 40 randomly
selected days throughout the school
year.
6. What question do you have about
the research methodologies in
these studies?
5. An outcomes assessment research
project of a 5 year old IL program in
which all incoming freshmen must
participate. Total student population on
campus is divided between 32%
freshmen to senior (or 4 year) and
68% transfer students.
7. What question do you have about
the research methodologies in
these studies?
6. Over a one year period, researchers
studied the occurrence of turn-aways in
a virtual reference service and noted
that the significantly high occurrence of
turn-aways indicates increased need for
virtual reference service.
8. What question do you have about
the research methodologies in
these studies?
7. A survey of faculty found that the
majority of those interview interacted
most with librarians at the reference
desk. The researchers concluded that
most faculty view librarians in a servile
role.
9. First things first
► Basics
► Topic ideas
► Typical methodologies
► Common pitfalls
► Getting started and putting it all
together
► Questions/discussion
10. Basic steps of a research
project
►Find a topicWhat, When
►Formulate questionsWhat, Why
►Define populationWho, When
►Select design & measurementHow
►Gather evidenceHow
►Interpret evidenceWhy
►Tell about what you did and found out
12. Topic ideas
►Online chat reference
● Types of questions
Subject? Type?
# of turnaways*
● Difference in discourse
In-person vs. chat
● Partnership studies
Similar libraries with same software
13. Topic Ideas
►E-book usage
►Usability studies of
● Online tutorial(s)
● ‘My Library” portals
►Analysis of library web sites or
library instruction sites or
pathfinders by best practices
►Student learning outcomes in LI
programs
14. Types of methodologies
►QuaLitative Measures
● Descriptive
● Numbers not the primary focus
● Interpretive, ethnographic,
naturalistic
►QuaNtitative Measures
● N for numbers
● Statistical
● Quantifiable
15. QuaLitative measures
►Content Analysis
● Analyzed course syllabi of
library use through discipline
and level (Rambler)
● Studied online tutorials,
applying best practices
recommendations (Tancheva)
16. QuaLitative Measures
►Discourse Analysis
● Analyzed student responses in writing
and discussions to a short film &
compared findings to parallel study
with LIS grad Ss (Vandergrift)
►Focus Groups
● Discussed how participants
experience & use the library (Von
Seggern & Young)
● Studied why students use the
Internet and how much time they use
it (Wilson)
17. QuaLitative Measures
►Interviews
● Studied 25 HS students’ web use for
research assignments (Lorenzen)
● Looked at what type of information first
year students need and how they go about
acquiring it (Seamans)
►Observation (obtrusive)
● Observed students as they conducted
online research & noted their activities
(Dunn)
►Observation (Unobtrusive)
● Retrieval of discarded cheat sheets to
analyze academic misconduct (Pullen et. al.)
18. QuaLitative Measures
►Think Aloud Protocols
● Studied how users navigate a library
web site (Cockrell & Jayne)
►Usability testing
● Examined students’ mental models
of online tutorials (Veldof & Beavers)
20. F
QuaNtitative measures
►Comparison studies
● Experimental and control groups
● Instructional methodologies (Colaric;
Cudiner & Harmon)
● Program assessment using before/after
analysis of research papers(Emmons &
Martin)
21. G
QuaNtitative measures
►Pre & Post Tests (Van Scoyoc)
►Measures & Scales
Bostick’s Library Anxiety Scale (Onwuegbuzie
& Jiao; Van Scoyoc)
Procrastination Assessment Scale
(Onwuegbuzie & Jiao)
23. Ready Made Data Sets
►National Survey of Student
Engagement (Whitmire)
►College Student Experiences
Questionnaire (Kuh and Gonyea)
►The Web
● Internet Archive (Ryan, Field &
Olfman)
● Electronic journals (Dellavalle)
►Library server logs
25. Research Problem #1
A study assessing student learning
outcomes in 2 broad categories
(concepts, techniques) by examining
student research journals in 1 section
of an elective information literacy
course in fall semester.
26. Research Problem #4
A 2004 article on a library use and
services satisfaction study that used
as its measurement tool a survey
given to every nth person entering
the library building on 40 randomly
selected days throughout the school
year.
27. Research Problem #5
An outcomes assessment research
project of a 5 year old IL program in
which all incoming freshmen must
participate. Total student population on
campus is divided between 32%
freshmen to senior (or 4 year) and
68% transfer students.
29. Research Problem #2
An experimental study that proposes
a fund allocation formula for academic
library collections based on the
following:
average of overall book price + average of
overall serial prices * degree level (10 for
undergraduate to 30 for doctorate) / the number of
students enrolled in degree program as majors +
the total number of faculty in the department * three
* total number of students in program.
(OAB + OAS) * D/(Sn +(Fn*3))*Sn
N.B. Not a standard formula
30. Research Problem #3
A newspaper article you read just the
other day stated that in a recently
published study done at a major U.S.
university, researchers found that
domestic violence affects 1 in every 4
women.
31. Research Problem #4
A 2004 article on a library use and
services satisfaction study that used
as its measurement tool a survey
given to every nth person entering the
library building on 40 randomly
selected days throughout the school
year.
32. Research Problem #5
Over a one year period, researchers
studied the occurrence of turn-aways
in a virtual reference service and noted
that the significantly high occurrence of
turn-aways indicates increased need
for virtual reference service.
34. Research Problem #1
A study assessing student learning
outcomes in 2 broad categories
(concepts, techniques) by examining
student research journals in 1 section
of an elective information literacy
course in fall semester.
35. Research Problem #7
A survey of faculty found that the
majority of those interviewed interacted
most with librarians at the reference
desk. The researchers concluded that
most faculty view librarians in a
servile role.
36. Keep In Mind That
►No study is perfect
►“All data is dirty is some way or
another; research is what you do
with that dirty data” (Manuel)
►Measurement involves making
choices
37. Be Critical About Numbers
(Best 2001)
►“Every statistic is a way of summarizing
complex information into relatively
simple numbers.” (Best)
►How did the researchers arrive at these
numbers?
►Who produced the numbers and what is
their bias?
►How can key terms be defined & in how
many different ways?
38. Be Critical About Numbers
►How was the choice for the
measurement made?
►What type of sample was gathered
& how does that affect result?
►Is the statistical result interpreted
correctly?
►If comparisons are made, are they
appropriate?
►Are there competing statistics?
39. Getting Started
► Read to learn; read to analyze
● About research methodology
● Studies on similar topics
● Interesting studies
● Non-library studies
40. Getting Started
►Finding a topic needn’t be
traumatic
● Work projects Research studies
P&T overhaul
Library GO Bond Proposal Project
Library workshop trends
User repair strategies
41. Getting Started
►Data collection involves agreement
& consent
►Forge partnerships
►At some point you will need to
leave the comfort zone of reading
and literature gathering and …
Welcome to Research Methodology 101. Today we would like to cover some basics concepts related to library research and also touch on a few practical matters such as putting research into practice into your library, your job. I believe that we all can agree—no matter what type of library we work in or what type of position we hold within that library—that accountability and assessment are two governing themes of the day. We are being asked to assess and be accountable for—our services, our programs—especially our instruction programs—our collections. At the same time, ACRL IS has recently revised the Research Agenda for Library Instruction and Information Literacy. This research agenda can serve to help guide instruction librarians in assessing what they’re doing and how well they’re doing it, and how students are and are not learning. In any sort of assessment, one does research. And research need not be a scary thing. A research project can be a lot of fun.
Before we get started, I’d like to briefly go over your handout. After the title page with the funny little man who has lost all hairs but one to a detestable research project, is a short activity which we’ll dive into in a moment. Following that is an annotated bibliography of many research studies. We’ve noted the gist of each study, type of instrument used, groups studied, setting, etc. to help you along. We will refer to many of these studies throughout the course of this presentation. At the end of the handout is an index of specific research methodologies, such as pre and post test, usability studies, etc. Keep in mind that this bibliogrpahy is not exhaustive by any means. It’s biased towards library instruction and more toward qualitative than quantitative studies—although both are included. As a way of setting the tone for today’s session, please turn to the inside cover of your handout and look at the 7 descriptions of research methodologies in these studies. I’ll quickly show each of these on the screen, reading them aloud. Then what I would like you to do for the next several minutes is note any questions you have about these methologies, any problems or flaws that you note. Feel free to talk amongst yourselves. Compare notes. READ SLIDE (Note to self: self-selection, generalizability, sampling)
READ SLIDE (Note to self: Poorly defined (formula); Problems in measurement in that the average book and serial prices can hurt some areas and that it makes assumptions about the scholarly communication formats that may not be true. Also, if this is an elderly formula—as many of our funding formulas are then it’s a good time to take a look at whether this formula still fits and if not, why. Beware of unchanging measures)
READ SLIDE (Note to self: Definition of “affect” of “domestic violence”!!! Perhaps this ratio is based on Guessing??? How did the researchers conduct this study? Survey? Sample? Representative? How were questions formed? How about this Q: “Have you been affected by domestic violence in your lifetime?” A: “Oh yes, I watch the Lifetime channel for women and I have seen a lot of movies on battered women. I remember that one with Farrah Fawcett—she’s a Corpus Christi native, ya know, where she burned her husband in his bed? And then she was tried for murder? But I think she got off because he beat her so bad?)
READ SLIDE (Note to self: Definition of group? A lot of false negatives here. Folks that ought to be included but aren’t because they don’t physically enter the library. Perhaps this campus has a large distance ed program or that gate count stats are declining whereas external web activity has skyrocketed. Perhaps the library has a doc/item delivery program so heavy users of the library’s services rarely, if ever, come in the building? Random sample? How were the days selected? Hours? Sometimes Random is given a lot more attention than it deserves. Skewness? Could end up missing heavy use days of year.)
READ SLIDE (handout is a bit different, doesn’t include the terms “freshmen to senior”) (Note to self: Whoa! Sample is not consistent with the aims of the program. And perhaps the program itself needs some work. Maybe this study needs to address it)
READ SLIDE (Note to self: Definition problems. Significantly high compared to what? Is one significant? How about in person turnaways? Phone turnaways? E-mail? Software problem? Networking problems? Benchmarks please? Hasty conclusions)
READ SLIDE (Note to self: When did the reference desk ever mean servility? This is a good example of the problem of Transformation where researchers and/or those interpreting the data make a quantum leap in conclusions. Similar to that children’s rhyme: Drip is a drop, drop is nature, nature is beautiful, thanks for the compliment. Or the six degrees of separation motif.
Note to self use timer for activity. 2?3? minutes after reading the slide. Good. I can see that you all have some questions about these methodologies and that now you are all warmed up for this session. Let me give you our session overview. Right now we’re going to map out some research basics and then quickly list a few topic ideas that could work into a research project within your library. After that we’ll talk a bit about the technical stuff—the research methodologies themselves. We’ll give you definitions and examples of studies that have used these methodologies. Most, if not all, of these are mentioned in the bibliography. Next. we will discuss common pitfalls found in any research study. Then we want to talk about how you can get started on your research agenda, how you can put it all together in your library setting.
Hey there, does this list of steps sound familiar with all you instruction folk out there? Or any of you who have conducted a class session on research basics? Yes! It’s the same basic plan. It was when you were writing your first research paper in high school and college and it’s the same plan now. Finding the right topic can seem like a daunting task but we’ll show you some ways to make that step easier. After that you need to figure out just what your research focus really is, and that’s often done in the form of a question. Next, or even simultaneously, you should define your population of study. Students? Faculty? Users in your library? Which users? On to the next step of deciding your research design as well as deciding on your research instrument. You might ask yourself, “Am I going to conduct a survey? Via the web? E-mail? In person? Mail in? Will I interview people? Will I use a published measurement or scale? Will I do a pre and post test study?” Next you need to put your research plan into action by gathering your data set. Maybe you are collecting transaction logs from your web site or from your catalog or maybe you are doing classroom research so you are collecting data from your students over many semesters to do a learning outcomes assessment study. Next, you need to interpret what you have found. This step takes a little time and more than a lot of thought. Finally, you should write up your findings. Think of it as telling a story about what you did and what you found out. Simple? No? Fun? Sometimes~ Long term rewards? Priceless!
Since grade school many of us are familiar with the 5 W’s as a way to frame topics, situations, studies. So here’s another way to look at a research project or to figure out the boundaries of your study. What exactly are you studying? Defining your problem or your focus is extremely important. For instance, I want to study how faculty value library instruction. I want to look at their attitudes towards not only library instruction but also towards librarians and libraries. But who exactly am I going to study? All faculty? Maybe only faculty who really value library instruction by their using our LI program over several semesters. What time period are you looking at? Most importantly, Why—what’s your point? What could someone generalize from your findings? But what does it mean to the larger community? What can we all learn from this? Finally, how are you going to put this project, this study in place?
Let’s take a look at a variety of research topics that could easily turn into a research project. Online chat reference or virtual reference is a good example. Maybe you are interested in looking at the nature of questions you’re getting via your new service and you want to examine these by subject or by type or perhaps by transaction length. Maybe you are interested in the number of turnaways to the service because it’s busy or not staffed. Perhaps you want to analyze the differences and similarities in discourse between in-person reference and chat or virtual reference. And when I say discourse I mean the structure and the nature of the conversation or the communication exchange between the user and the librarian. Let’s say that one or more libraries similar to yours are all using the same virtual reference package—question point, LSSI, etc. and you want to construct a study that encompasses all those libraries VRef services.
Here’s some more opportunities for studies. Let’s say you have e-book collections from NetLibrary or Ebrary and wish to look at usage of those collections. Or you have a rough draft of one or more online tutorials or a My Library portal that you would like to test out. Let’s say that you want to apply best practices characteristics to a set of library instruction sites or library web sites or online pathfinders. Perhaps you are interested in measuring student learning outcomes in your library instruction program. A lot of these types of projecta occur normally in libraries. It’s part of what we do. We implement and maintain services and then we want to see how those services are doing. So we measure them. So you see, almost everything you are doing in lib in job could become the basis for a study. It’s worth emphasizing the synergy and efficiency to make your work project into a research project.
When we talk about methodologies within the context of library research, we would like to divide these measures into two different areas: Qualitative and Quantitative. Briefly Qualitative measures is more descriptive in nature, examining in detail people and things. While numbers are not the primary focus of a qualitative study, you will find statistics within these studies. Essentially qualitative research is more interpretive and ethnographic. An ethnography describes and inteprets characteristics or behaviors of a culture or a group. On the other hand, quantitative measures are numerically based. They include statistics and statistical measures. Quantitative measures compare, assess and quantify. These types of studies are those that frequently use those mathematical symbols and have their own language with words and phrases such as analysis of variance and statistically significant and central tendency and regression and correlation and chi square. Typically quantitative analyses or studies look at the big picture whereas qualitative studies examine groups or things in detail. Think of one as the telescope and the other as the microscope.
To give you some grounding and some encouragement in launching your own research project, let’s look as some examples of both types of studies. We’ll start with qualitative studies. According to Thomas Schwandt ”Content analysis is a generic term for many different types of textual analysis & it involves first coding then comparing, constrasting and organizing a body of data to test hypotheses”. Linda Rambler conducted a content analysis of syllabi at Penn State to determine first whether courses included library use and if so, which type of library use. Another example of content analysis is found in Kornelia Tancheva’s study of online tutorials based on recommendations of “best practices”. By the way, Kornelia’s study builds on Nancy Dewald’s study of online tutorials. So you see you can take someone else’s study and focus yours in a different direction, adding to the body of knowledge about that area.
Another frequently used qualitative measure is discourse analysis. This is a method to analyze the process of communication. And that communication can be from a variety of sources, be it between people, found within a text, broadcast on the radio, or television, or within a film. It is concerned with the “how” of communication.Kay Vandergrift’s study in School Library Media Quaterly compares junior high school students responses to a short film, The Shooting Gallery, that is a parable on totalitarianism, with repsonses of graduate students to the same film. It’s a discourse analysis in that Vandergrift is looking at the words and phases and imagery students from both groups use to describe & respond to the film. On the other hand, the focus group methodology comes to us from the field Marketing and is used to quickly gather ideas, views and attitudes from a group of people. It is a method noted for synergy in stimulating discussion and bouncing ideas off of many at once. Marilyn Von Seggern and Nancy Young used this method to examine how their users viewed the rapidly changing world of information and what impact those changes had on their searching for information. In another study, Rebecca Wilson used focus groups of a total of 75 students and looked at why students use the Internet and how much time they spend on the Internet.
Here’s a couple of other examples of qualitative measures. Interviews are typically conducted as a way to discover attitudes, beliefs and values. Typically interview sessions are informal where the interviewer asks open-ended questions to allow for a high degree of flexibility and responsiveness from the interviewee. Michael Lorenzen interviewed students to figure out how they used the internet in doing their research assignments and to what extent they used the Internet—more, less or the same as other information sources. Observation has been used for a very long time as a qualitative methodology. Obtrusive studies are when the participants know that they are being observed. Unobtrusive studies are conducted without the participants knowledge that they are being observed. Kathleen Dunn used obtrusive observation—among many other types of methodologies-- in her assessment study of information literacy skills within the Cal State University System. Robert Pullen and his team took a novel approach to an unobtrusive study by picking up wadded cheat sheets from the floor or retrieving them from waste paper baskets to study how cheat sheets were used.
Think Aloud testing came about in the early 80s as a way to understand the thought processes that go into writing by asking the writers to vocalize their thoughts about what they are doing and why they are doing it as they compose. Think aloud tests have been used in library research and have involved asking a person to complete a task (like searching a database, or navigating a web page) at the same time as they vocalize their thoughts about what they are doing and their decision making processes. It’s a good way to do an early test of a prototype and uncover any misconceptions or confusions but it’s also a bit of an unnatural act for many participants. Also, the act of vocalization slows down the thought processes. Barbara Cockerell and Elaine Jayne used the TA method when they did a usability test of their library web site. Usability tests, on the other hand, had more practical origins. These very popular tests came from product engineering as a way to see if a product does what it’s supposed to do, whether folks can figure out how to use something. Jerilyn Veldof and Karen Beavers conducted a usabilty study to figure out what mental models students use in navigating online library tutorials.
Now let’s switch our focus to Quantitative studies. Think N for numbers, here. Essentially, these studies, these methodologies take the big or bigger picture approach. They compare things or people, showing the differences and similarities, they count things and they survey things.
Comparison studies are often experimental in nature, using one group with no or tradition treatment and the other with the experimental treatment. Comparison studies often look at the outcomes of different types of teaching methods. Susan Colaric examined three different methods (teaching by example, instruction with conceptual models and instruction without illustrations) to figure other how students learned specific searching concepts. Mark Emmons and Wanda Martin compared student portfolios from before and after the implementation of new course-integrated library instruction component.
Often when making these comparisons, researchers measure peoples knowledge or attitudes before and after a “treatment. Sometimes using established measures and scales as a pre and post testing instrument. For example, Anna Van Scoyoc, used Bostick’s Library Anxiety Scale as a pre and post tests to determine whether computer based or traditional library instruction reduced library anxiety.
Judit Bar-Ilian examined the number and nature of other web pages linking to Cybermetrics home page; whereas Robert Dellavalle and other analyzed the frequency, format and activity of internet reference from three science journals.
Keep in mind that you don’t always have to gather your own data; you can use pre-existing sets. The web provides an excellent spot for data mining. Ryan, Field and Olfman used the Internet Archive to study how state government web pages evolved over time. You might want to mine your own server logs. Nancy Turner and I looked at our catalog transaction logs to analyzed the types of repair strategies users employed when searching.
You should try to avoid some typical problems that befall researchers. One of these is found in population. First of all is your sample representative of what you are trying to study? How did you arrive at your sample? Did you not exclude those that need to be included or did you include those that shouldn’t be included. Let’s look at some of those research problems we looked at in the beginning of this session.
Note that this is an elective course so students have self-selected. They not only want to be in the class, they are probably pretty good at the topic already. Also this study is looking at just one section of the course in one semester.
I am sure you all caught the problem of surveying only those who enter the building as opposed to survey those that use the library virtually.The random sampling of days could well skew the study. Perhaps busiest days are missed entirely. Random and representative are not always compatible.
Here it’s fairly obvious that the majority of those being assessed have not received the treatment.
It’s important to thoroughly define what you are measuring and how you are measuring it otherwise you may run into some problems.
READ SLIDE (Note to self: Poorly defined (formula); Problems in measurement in that the average book and serial prices can hurt some areas and that it makes assumptions about the scholarly communication formats that may not be true. Also, if this is an elderly formula—as many of our funding formulas are then it’s a good time to take a look at whether this formula still fits and if not, why. Beware of unchanging measures)
READ SLIDE (Note to self: Definition of “affect” of “domestic violence”!!! Perhaps this ratio is based on Guessing??? How did the researchers conduct this study? Survey? Sample? Representative? How were questions formed? How about this Q: “Have you been affected by domestic violence in your lifetime?” A: “Oh yes, I watch the Lifetime channel for women and I have seen a lot of movies on battered women. I remember that one with Farrah Fawcett—she’s a Corpus Christi native, ya know, where she burned her husband in his bed? And then she was tried for murder? But I think she got off because he beat her so bad?)
READ SLIDE (Note to self: Definition of group? A lot of false negatives here. Folks that ought to be included but aren’t because they don’t physically enter the library. Perhaps this campus has a large distance ed program or that gate count stats are declining whereas external web activity has skyrocketed. Perhaps the library has a doc/item delivery program so heavy users of the library’s services rarely, if ever, come in the building? Random sample? How were the days selected? Hours? Sometimes Random is given a lot more attention than it deserves. Skewness? Could end up missing heavy use days of year.)
READ SLIDE (Note to self: Definition problems. Significantly high compared to what? Is one significant? How about in person turnaways? Phone turnaways? E-mail? Software problem? Networking problems? Benchmarks please? Hasty conclusions)
Before we get started, I’d like to briefly go over your handout. After the title page with the funny little man who has lost all hairs but one to a detestable research project, is a short activity which we’ll dive into in a moment. Following that is an annotated bibliography of many research studies. We’ve noted the gist of each study, type of instrument used, groups studied, setting, etc. to help you along. We will refer to many of these studies throughout the course of this presentation. At the end of the handout is an index of specific research methodologies, such as pre and post test, usability studies, etc. Keep in mind that this bibliogrpahy is not exhaustive by any means. It’s biased towards library instruction and more toward qualitative than quantitative studies—although both are included. As a way of setting the tone for today’s session, please turn to the inside cover of your handout and look at the 7 descriptions of research methodologies in these studies. I’ll quickly show each of these on the screen, reading them aloud. Then what I would like you to do for the next several minutes is note any questions you have about these methologies, any problems or flaws that you note. Feel free to talk amongst yourselves. Compare notes. READ SLIDE (Note to self: self-selection, generalizability, sampling)
READ SLIDE (Note to self: When did the reference desk ever mean servility? This is a good example of the problem of Transformation where researchers and/or those interpreting the data make a quantum leap in conclusions. Similar to that children’s rhyme: Drip is a drop, drop is nature, nature is beautiful, thanks for the compliment. Or the six degrees of separation motif.