Dr. Lani discusses all aspects of the dissertation methodology, including: selecting a survey instrument, population, reliability, validity, data analysis plan, and IRB/URR considerations.

1. Methodology and IRB
By Dr. James Lani

2. Methodology and IRB/URR
Dr. James Lani
Take Away Message
Research design is a blueprint with
several components:
• Research method: Quant/Qual/Mixed
• Operational constructs: How
constructs are measured
• Sampling strategy and procedure
What is Research
Research follows the scientific method.
What’s the scientific method?
• Formulation of testable questions
or hypotheses
• It’s organized knowledge: logical
(theory) and evidence based
(observable)
• Precise constructs
• Can be disproven (falsifiable)
• Parsimonious (simplest
explanation)

3. Methodology: The Cookbook Metaphor
Cooking Researching
Making a stew… Examining research questions…
Ingredients: 2 lbs beef, I clove garlic…
Preparation: Cube beef, mince garlic…
Cooking Instructions: Bake at 350 for one
hour
Celebrate: You’ve made a replicable) stew!
Ingredients: 20 participants, 15 item
questionnaire, semi-structured interview.
Preparation: Administer questionnaire before
and after lecture; semi-structure interview with
participants for 20 minutes using a tape
recorder.
Data analysis plan: Conduct dependent sample
t-test; transcribe interviews then thematize
participants responses.
Celebrate: You’ve conducted a (replicable)
research study!
Where the recipe can be replicated

4. Methodology Essential Ingredients
Restate
research
question and
hypotheses
Overview
Research
Design
Participants
Materials/
Instruments
Data
Collection
Procedures
Sample Size
Data
Limitations
Analysis Plan

5. Methods Quantitative Qualitative Mixed
You can count it Not quantitative Both
Goal
Tend to answer “What
questions” (What is
relationship or
differences) or “When
questions” (when is
theory supported…after
intervention
Tend to answer “Why
questions” (understand
why people feel that way)
or “How questions”
(explore how they see
things)
Both
Research methods/
Strategies of data
collection
Experimental (random
assignment) and Non-experimental
methods
(no control group),
Observations (time
participant)…
Semi-structured
interviews, Archive data,
Observations (write down
positive and negative
feeling words)… Both
Operationalize
variables
Define constructs/
instruments
Define constructs/
instruments Both
Sampling strategy Discuss sampling process Discuss sampling process
Both

6. Theoretical (or Conceptual) Framework
Theory is a systematic explanation of behavior
of phenomena.
• Theory guides analyses
• List existing theories and how your
research questions relate to those
theories
• E.g., Theory of mind: the ability to understand that
others have their own beliefs, desires, intentions.
Empathy. Tested by Faux Pas Task (ability to
recognize a faux pas). Research question: Does
alcohol abuse impact empathy?

7. Population
The population is the group you want to
generalize to.
• Describe characteristics of population
• Why is population relevant to problem
(look at other peer reviewed study’s
justification)
• Distinguish the population from the sample

8. Sampling Frame and Sample
Universe:
theoretical
population to
generalize to
Population: largest
target population
from universe you
have access to or
“Sampling Frame”
Original Sample
Final Sample
Attrition

9. Survey
Item
Old (one’s own SE) Change in item New (perceptions
of others’ SE)
1
I feel that I have a
number of good qualities.
“I have” to “she has” I feel that she has a
number of good
qualities.
2
I feel I do not have much
to be proud of.
“I do not” to “she
does not”
I feel she does not
have much to be
proud of.
3
On the whole, I am
satisfied with myself.
“I am” to “she is” and
“myself” to “herself”
On the whole, she is
satisfied with herself.
…
If you need to amend instrument, use a change
matrix; do not create your own instrument!

10. Materials: Informed Consent
• State purpose of project
• State procedure and how long it will take
• State voluntary nature of participation
• State risks (if any)
• Have them sign or state that by filing our
survey they are agreeing to participate.

11. Constructs vs. Variables
Empirical Theoretical
Construct A:
Social
Environment
Construct B:
Personality
Variable A:
Birth-Order
Variable B:
Introversion/
extroversion

12. Constructs vs. Variables
Constructs are the invisible abstract things we’re measuring (e.g.,
personality), while variables are the way we’re assessing
(measure/operationalize) those invisible things.
Constructs and variables need to be precise (is personality
measured by introversion scale or by conscientiousness scale?)
e.g., Intelligence is a construct, while the number of words
remembered is a way of assessing intelligence.
e.g., Personality is a construct, while the scores on an
introversion/extroversion test is a way to assess an aspect of
personality.

13. Variables
Operationalize Constructs, Make Distinction between
IV’s/DV’s, and Describe Level of Measurement
Example:
Does Empathy differ by group (alcohol abuse vs. no alcohol abuse)?
Empathy is my dependent variable and Group is my independent
variable.
Empathy (my construct) is measured by scores on the Faux Pas task.
Alcohol abuse (my construct) is measured by 5 or more drinks in one
day.
Empathy is a ratio-level variable measured with scores ranging from 0-
25, while my Group variable is a nominal-level (categorical-level) variable
because participants re in one of two groups (alcohol abuse group or
not in alcohol abuse group).

14. Reliability and Validity
Valid but not
Reliable
Reliable but
not Valid
Reliable and
Valid

15. Validity
• Internal Validity: IV causes a change in the DV (not time or
other covariates, etc.
• External validity: can be generalized to the population
• Construct validity: does the scale measure the theoretical
construct
• Translational validity:
• Face validity (items are reasonable)
• Content validity (items match the domains of interest)
• Criterion related validity: measures behave as theory predicts
• Convergent validity: how close the variable aligns with the
construct (use EFA)
• Concurrent validity: construct relates to established
instruments
• Predictive validity: measure can predict an outcome
(GPA→Income)

16. Reliability
Internal consistency: Cronbach alpha. Average
inter-item correlation
Inter-rater: if interval, correlate; if dichotomous,
kappa
Test-retest: administer same test at two times
Split half: divide instrument into 2 parts and
calculated totals, then correlate totals.

17. Brief Review: Units of Analyses
Quantitative (e.g., Age)
Nominal-level (Latin for name). Gender (M/F), Grouped (Old =
65+, middle age = 36-64, young = 35 or younger). Assign any
number of groups (old = 1, middle = 2, young = 3).
Ordinal-level is ranked (Latin for showing order). GPA (A-F), or
age (group 1 = age 1–15, group 2 = age 16-25, group 3 = age 25-54,
group 4 = age 65+)
Interval/Ratio-level (also named scale or continuous; Latin for
[equal] space [between numbers]). What is your age today in
years? ____ (a number from 1-105)

18. Types of Methodology Models
Theories explain phenomena,
Models represent Phenomena.
• SEM and Path models
• Regression models (linear, logistic, ordinal)
• ANOVA models (repeated-measures)
• Time-series
• Heirarchical Linear Models (HLM)
• Correlational Models

19. Relationship Among Variables
Employee
Tenure
Leadership
Style
Employee
Satisfaction
Customer
Satisfaction
Mediator
Moderator

20. Data Collection Method --
Overview
• Describe the procedures used to administer
the materials to the participants
• Remember to be as detailed as necessary
so someone can literally replicate your
study

21. Data Collection
Qualitative
• Procedure for accessing participants
• Selection of data collected
• Number and duration of interviews
• How and when data is collected
• How data is recorded (e.g., hand notes,
audiotaped)
• Role of researcher-relationship to researcher
Quantitative
• Procedure to administer measures

22. Pilot Test (only if you develop
instrument)
• Detect potential issues in the instrument
• Allows you to get feedback and to finalize
your survey/interview items
• Makes sure participants understand survey
items
• Assess typical responses to survey items or
interview questions (were participants
comfortable, long-winded, defensive, etc.
• Can you access data?

23. Data Analysis Plan
• Quantitative: Describe the analysis plan used
to test each hypothesis, the assumptions of
the statistical analyses, and a justification of
the appropriateness of the analysis for each
research question.
• Qualitative: Describe how the data will be
analyzed (or thematized)
• Phenomenological
• Case Study
• Grounded theory

24. Sample Size
Quantitative
Varies by type of
statistical analysis
1. Research questions
in statistical language
2. Level of
measurement of
variables
3. Determine
statistical analysis
4. Conduct power
analysis
Qualitative
Varies by which
qualitative approach
is taken
• Phenomenological
• Case Study
• Grounded theory

25. Assumptions, Limitation, Delimitations
Assumptions: Discuss things out of your control
about the population and design, then justify
assumptions (e.g., participants will answer
honestly.)
Limitations: Are out of your control and
describe weaknesses in design, threats to
validity (e.g., generalizability).
Delimitations: Are in your control and relate to
choices you will make to narrow the scope of
the study (e.g., variables, research questions).

26. Ethical Considerations
• Describe informed consent procedures
• State whether your study will be anonymous
or confidential with respect to the participants
• Describe considerations for children or
vulnerable participants

27. See yourself graduate in 2014!
1-1 Personalized Dissertation Consulting
877-437-8600
Info@StatisticsSolutions.com

28. Thank you for
your participation
and attention!
Join us for our next webinar on
Wednesday, December 17th at 8:30pm
ET.
Results and Discussion Chapters,
Editing, and Defense

29. Join us for our 1-day
Dissertation
Workshop
Dallas/Fortworth, Texas
December 6th 9am-5pm
$199/student $89
lodging/night