2. STRUCTURE OF THE CHAPTER
• Defining validity
• Validity in quantitative research
• Validity in qualitative research
• Types of validity
• Triangulation
• Validity in mixed methods research
• Ensuring validity
• Reliability
• Reliability in quantitative research
• Reliability in qualitative research
3. STRUCTURE OF THE CHAPTER
• Validity and reliability in interviews
• Validity and reliability in experiments
• Validity and reliability in questionnaires
• Validity and reliability in observations
• Validity and reliability in tests
• Validity and reliability in life histories
4. BASES OF VALIDITY IN
QUANTITATIVE RESEARCH
BASES OF VALIDITY IN
QUALITATIVE RESEARCH
Controllability Natural
Isolation, control, manipulation of
Variables
Thick description
Replicability Uniqueness
Predictability
Emergence, unpredictability
Generalizability Uniqueness
Context-freedom Context-boundedness
Fragmentation and atomization Holism
Randomization of samples Purposive sample/no sampling
Neutrality Value-ladenness of observations
Objectivity Confirmability
Observability Observable and non-observable
meanings/ intentions
Inference Description, inference, explanation
‘Etic’ research ‘Emic’ research
Observations Meanings
5. BASES OF RELIABILITY IN
QUANTITATIVE RESEARCH
BASES OF RELIABILITY IN
QUALITATIVE RESEARCH
Reliability Dependability
Demonstrability Trustworthiness
Stability and replicability Stability and replicability
Parallel forms Parallel forms
Context-freedom Context-specificity
Objectivity Authenticity and confirmability
Coverage of domain Comprehensiveness of situation
Verification of data and analysis Honesty and candour
Answering research questions Depth of response
Meaningfulness to the research Meaningfulness to respondents
Parsimony Richness
Internal consistency Credibility
Generalizability Transferability
Inter-rater reliability & triangulation Inter-rater reliability and triangulation
Accuracy and precision Accuracy and comprehensiveness
Neutrality Multiple interests represented
Consistency Consistency
Alternative forms (equivalence)
Split-half and inter-item correlation
6. VALIDITY IN QUANTITATIVE AND
QUALITATIVE RESEARCH
• Validity in quantitative research often
concerns: objectivity, generalizability,
replicability, predictability, controllability,
nomothetic statements.
• Validity in qualitative research often
concerns: honesty, richness, authenticity,
depth, scope, subjectivity, strength of feeling,
catching uniqueness, idiographic statements.
11. ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH
• Prolonged engagement in the field
• Persistent observation
• Triangulation
• Leaving an audit trail
• Respondent validation
• Weighting the evidence (giving priority)
• Checking for representativeness
• Checking for researcher effects
• Making contrast/comparisons
• Theoretical sampling
• Checking the meaning of outliers
• Using extreme cases
12. ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH
• Ruling out spurious relations
• Replicating a finding
• Referential adequacy
• Following up surprises
• Structural relationships
• Peer debriefing
• Rich and thick description
• Looking for possible sources of invalidity
• Assessing rival explanations
• Negative case analysis
• Confirmatory data analysis
• Effect sizes
13. THREATS TO VALIDITY IN
QUANTITATIVE RESEARCH
• History
• Maturation
• Statistical regression
• Testing
• Instrumentation
• Selection Bias
• Experimental mortality
• Instrument reactivity
• Selection-maturation interaction
• Type I and Type II errors
14. VALIDITY PROBLEMS IN
CROSS-CULTURAL RESEARCH
• Failure to operationalize elements of cultures
• Whose construction of ‘culture’ to adopt: ‘emic’/‘etic’
• False attribution of causality to cultural factors rather than
non-cultural factors
• Directions of causality
• Ecological fallacy
• Sampling and instrumentation
• Convergent and discriminant validity
• Response bias and preparation of participants
• Language problems
• Problems of equivalence (conceptual, psychological,
meaning, instrument, understanding, significance, relevance,
measurement, linguistic)
15. THREATS TO EXTERNAL VALIDITY
IN QUANTITATIVE RESEARCH
• Failure to describe independent variables explicitly
• Lack of representativeness of available and target
populations
• Hawthorne effect
• Inadequate operationalizing of dependent variables
• Sensitization/reactivity to experimental/research conditions
• Interaction effects of extraneous factors and experimental/
research treatments
• Invalidity or unreliability of instruments
• Ecological validity
•
16. THE HAWTHORNE EFFECT
Between 1927 and 1932 researchers carried out
experiments at the Western Electric Company’s
Hawthorne plant.
• Purposes: To examine the effects of changes of
working conditions on output of workers
• Sample: Six women, chosen as average workers
• Method: Women worked in a test room. Output
measured under different conditions (e.g. no change
→ change to method of payment → introduce two
rest periods → introduce six rest periods →
changes in lighting conditions, early clocking-off,
five-day working week → return to initial conditions
• Duration: 15 weeks
17. THE HAWTHORNE EFFECT
• Results: Output rose steadily during test
period and after the test period.
• Conclusion: Output did not seem to depend
on test conditions. Increased output seemed
to be due to the fact that the people had been
involved in the experiment itself, i.e. the act of
research had affected the results. The
results were a research of the research itself.
• Implications: The act of being involved in
research itself affects the results.
18. THREATS TO EXTERNAL VALIDITY
IN QUALITATIVE RESEARCH
• Selection effects
• Setting effects
• History effects
• Construct effects
19. ENSURING VALIDITY AT THE
DESIGN STAGE
• Choose an appropriate time scale;
• Ensure adequate resources for the research
• Select appropriate methodology
• Select appropriate instruments
• Use an appropriate sample
• Ensure reliability
• Select appropriate foci
• Avoid having biased researcher(s)
20. ENSURING VALIDITY AT THE
DATA COLLECTION STAGE
• Reduce the Hawthorne effect
• Minimize reactivity
• Avoid drop-out rates amongst respondents
• Take steps to avoid non-return of questionnaires
• Avoid too long or too short an interval between pre-tests
and post-tests
• Ensure inter-rater reliability
• Match control and experimental groups
• Ensure standardized procedures for gathering data
• Build on the motivations of respondents
• Tailor instruments to situational factors
• Address researcher characteristics
21. ENSURING VALIDITY AT THE
DATA ANALYSIS STAGE
• Use respondent validation;
• Avoid subjective interpretation of data
• Reduce the halo effect
• Use appropriate statistical treatments
• Recognize extraneous factors which may affect data
• Avoid poor coding of qualitative data
• Avoid making inferences/generalizations beyond the data
• Avoid equating correlations and causes
• Avoid selective use of data
• Avoid unfair aggregation of data
• Avoid degrading the data;
• Avoid Type I and/or Type II errors
22. ENSURING VALIDITY AT THE
DATA REPORTING STAGE
• Avoid using data selectively and unrepresentatively
• Indicate the context and parameters of the
research
• Present the data without misrepresenting the
message
• Make claims which are sustainable by the data
• Avoid inaccurate or wrong reporting of data
• Ensure that the research questions are answered
• Release research results neither too soon nor too
late
23. RELIABILITY IN QUANTITATIVE
AND QUALITATIVE RESEARCH
• Reliability in quantitative research:
– consistency (stability), accuracy,
predictability, equivalence, replicability,
concurrence, descriptive and causal
potential.
• Reliability in qualitative research:
– accuracy, fairness, dependability,
comprehensiveness, respondent validation,
‘checkability’, empathy, uniqueness,
explanatory and descriptive potential,
confirmability.
24. • Reliability as stability:
– Consistency over time and samples;
• Reliability as equivalence:
– Equivalent forms of same instrument;
– Inter-rater reliability;
• Reliability as internal consistency:
– Split half reliability (e.g. for test items)
TYPES OF RELIABILITY IN
QUANTITATIVE RESEARCH
28. RELIABILITY AND REPLICATION
IN QUALITATIVE RESEARCH
Repeat:
• The status position of the researcher
• The choice of informants/respondents
• The social situations and conditions
• The analytic constructs used
• The methods of data collection and analysis
Address:
• Stability of observations
• Parallel forms
• Inter-rater reliability
• Respondent validation
29. IMPROVING RELIABILITY
• Minimise external sources of variation;
• Standardise conditions under which
measurement occurs;
• Improve researcher consistency;
• Broaden the sample of measurement
questions by:
a) adding similar questions to the
instrument;
b) increasing the number of researchers
(triangulation);
c) increasing the number of occasions in
an observational study.
• Exclude extreme responses (outliers).
30. RELIABILITY AND VALIDITY AT
ALL STAGES
• Design and methodology
• Sampling
• Instrumentation
• Timing
• Data collection
• Data analysis
• Data reporting
