This document summarizes the notes from an empirical research methods class. It discusses why the course exists, including the need for empirical evidence in information science research involving people. It also covers topics like the scientific method, different types of empirical research methods, and the purposes and products of empirical research. Examples are provided to illustrate key points.

1. IS 4800 Empirical Research Methods
for Information Science
Class Notes January 11, 2012
Instructor: Prof. Carole Hafner, 446 WVH
hafner@ccs.neu.edu Tel: 617-373-5116
Course Web site: www.ccs.neu.edu/course/is4800sp12/

2. Topics for Today
I. Why does this course exist ?
II. Evidence-based research in IT
III. Scientific method
IV. What is an empirical research method ?
V. Purposes and Products of empirical
research
VI. Examples, Assignment 1

3. I. Why does this course exist?
■ Background facts
■ Empirical research means gaining new knowledge
through systematic observation and analysis of
data
■ Empirical methods usually taught only to graduate
student
■ Usually taught only in behavioral/social sciences
■ In the past, little use of empirical research in CS
(notable exception is HCI/UI design). This is
slowly changing.

4. Why does this course exist?
■ The need for an empirical approach to research
■ IS is concerned with users and clients as well as
the computer and the software
■ Research involving people requires empirical
studies, since logical proofs are not possible
■ Many decisions today are based on “anecdotal
evidence” or opinions
Example: proponents of software design and development
methods (functional programming, object-oriented
programming, formal methods, design patterns, eXtreme
Programming) claim their method improves productivity
and/or quality based on their opinion or anecdotal evidence.

5. Why does this course exist ?
■ Why teach empirical research to undergraduates
who are usually not aiming for careers as
researchers?
1. For careers in system admin/IT management
2. For careers in system design/development
3. For careers in IT planning/policy (“governance”)
4. All educated people should be able to analyze and
evaluate claims based on empirical evidence.

6. Examples: sexting, e-readers
■ Articles in WSJ August 25 2010
■ “it’s difficult to pinpoint how common the practice
(sexting) is. Various surveys have estimated . . .
from roughly 4% to 25%.”
■ “ a study of 1,200 e-reader owners . . found that
40% said they now read more than they did with
print books.”
■ Companies with IT product lines, legislators, etc. need their
questions answered with evidence, not just opinions.
Empirical research skills will be increasingly valuable.

7. II. Evidence-based research in IT:
the MIS researcher’s perspective
■ Research on (organizational) impact
■ What is the impact of computerized systems on productivity,
group communication, power relationships and
organizational practices ?
■ Does the system make the organization more effective in
carrying out its mission ?
■ Do the economic benefits of computerized systems exceed
their costs?
■ Research on IT management practices
■ What approaches work best for IT governance ?
■ How can risks be identified and minimized ?

8. Evidence-based research in IT:
the computing researcher’s perspective
■ Research on system lifecycle (requirements
analysis, design, development)
■ How can system designers translate people’s needs and
preferences into requirements ?
■ What are the advantages and disadvantages of proposed
software/system development methods ?
Both MIS and computing perspective
■ What problems should be expected when systems are
deployed in their target environment
■ What are the best approaches (technical and managerial) to
protect the security of systems

9. Evidence-based research in IT:
How the Internet fits in
■ We can study who uses the Internet for what purposes
■ We can study how the Internet affects us as
individuals and as a society
■ We can use the Internet as a tool in our research (e.g.,
for surveys or interviewing)
■ We can study how Internet-based research is different
from other types (e.g., different ethical issues arise).

10. III. The Scientific Method
 Methods of inquiry
 Method of authority
 Method of logic (deductive reasoning from
known facts)
 Scientific Method (inductive reasoning)
“The Name of the Rose” by Umberto Eco: best selling
novel about the transition from authority to reason in
1300’s. (movie starring Sean Connery)

11. Scientific Explanation
 Empirical
 Rational
 Testable
 Tentative
 Parsimonious
 Rigorously evaluated (vs. “anecdotal”)

12. Steps of the Scientific Method
 The scientific method consists of four steps
 Noticing a phenomenon
 Formulating testable explanations (hypotheses)
 Observing and/or experimenting
 Refining and retesting explanations
 The scientific method can be tedious and time
consuming
 But it is the most widely accepted methodology for
acquiring knowledge about the world.

13.  Noticing a Phenomenon
 After noticing a phenomenon, you identify
VARIABLES or THEMES that appear important in
explaining what happens
 Formulating Tentative Explanations
 Initial observations allow you to develop a
HYPOTHESIS, or tentative statement, about the
relationships among the variables identified

14.  Observing and Experimenting
 You DESIGN and carry out systematic
OBSERVATIONS of the behavior of interest,
 These observations are directed at testing your hypothesis
 Refining and Retesting Explanations
 Supported hypotheses are often REFINED and
subjected to further exploration
 Disconfirmed hypotheses may be reworked and RETESTED

15. The importance of skepticism
 Ask whether the observed data can be
explained another way?
 Seek data that disconfirms a theory
 View all theories as tentative (accepted until
disproved)
How do I know whether this is true??
How could it be proved false??

16. Challenges for Scientific Method in
Behavioral/Social Research
■ Finding hypotheses that are both testable and
that answer your question
■ The feasibility of collecting data to test your
hypothesis
■ The ethics of collecting data to test your
hypothesis
Ex: social impact of parental neglect:
experiments carried out on chimpanzees

17. IV. What is a “research method”?
A strategy of inquiry, which moves from underlying
philosophical assumptions to a research design
Philosophical schools regarding research (“epistemology”):
positivist – research should seek objective truth
interpretive – research should take human experience
into account, truth can be subjective
critical – research should be aimed at making the world better
Philosophical
assumptions
Research
strategy/method
Research
design

18. Categories of Empirical Data
•Artifacts
•Laboratory experiments
•Natural observations
•Questionnaires
•Interviews
•What kinds of questions can be answered?
•What sources of data are utilized?
•How is the data generated/collected?
•How is the data analyzed?
•How do the results of analysis result in answers?
Attributes of research methods

19. Overview of the research methods
■Quantitative
■correlational studies
■surveys
■experiments
■quasi-experiments
■Qualitative
■ethnography
■case studies
■grounded theory
■Mixed

20. Classification schemes for research methods
1. Epistemological – what kind of claims can be
made
■ Descriptive/Interpretive
■ Correlational/Relational
■ Causal
2. Methodological: how we go about it (process)
3. Level of generality
■ Scientific research aims to discover general principles or
test a general theory
■ Evaluation research aims to support decision making or test
a hypothesis in a specific context

21. Deploying Empirical Research Methods
Idea
Observation Prior Knowledge
Research hypothesis or question
Research Method
What to observe and
appropriate measures
Subject Population/
Sampling technique
Research Design
Step-by-step research plan
How the data will be
analyzed

22. Steps of Empirical Research
1. Select a topic or phenomenon to study
2. Perform background investigation/literature review
3. Identify specific research questions or hypotheses
4. Select a research method
5. Develop a detailed research plan/proposal
a. For identifying or recruiting subjects
b. For collecting data
Define variables or phenomena to be observed and a
coding scheme
Determine how the data will be obtained
c. For analyzing data
6. Execute the plan
7. Document the research you performed (ongoing)

23. V. Purposes and Products
■ Reasons for doing empirical research
■ Disconfirm or support a theory
■ Solve a design or technical problem
■ Evaluate a product or solution
■ Inform practice (develop “best practices”)
■ Understand why people do what they do
■ Predict the future
■ Contribute to a better world
■ Advance personal goals

24. V. Purposes and Products
■ Products of Empirical Research
■ Evidence of a relationship
■ Disconfirm a theory (or some aspect of it)
■ Comparison of existing products or techniques
■ A new or improved product or technique
■ In-depth study of a particular outcome
■ Explanation of unanticipated outcomes
■ A new theory
■ A new improved research tool or technique
■ A critical analysis (pros and cons)