This presentation is prepared as part of the Course assignment of “Public Health Research” for the Master's Degree of Public Health (MPH), Pokhara University and can be used as reference materials. The content and facts included in the presentation are as of information available till February 2023 and no conflict of interest is associated with the presentation. The presentation is prepared by Sagar Parajuli & Raju Prasad Sah.

1. Validity and Reliability Threats, Construction
and use of appropriate research tools to
reduce threats
Raju Prasad Sah
Sagar Parajuli
MPH Second Semester 2023
School of Health & Allied Sciences
Pokhara University

2. 2/24/2023
2
Presentation Outlines
 Introduction to Validity and
Reliability in research
 Basics about Threats
 Threats to Internal Validity
 Threats to External Validity
 Reliability Threats
 Construction and use of
appropriate research tools to
minimize threats

3. 2/24/2023
3
Validity in Research
 Validity is the extent to which test or method measures what is
supposed and intended to measure.. A test is considered valid if it truly
measures what it is supposed to measure without any biases or
inaccuracies.
 The validity of a research study refers to how well results among
study participants represent true findings among similar individuals
outside the study.
 Validity in research refers to the extent to which a study accurately
measures what it claims to measure.

4. 2/24/2023
4
Validity in Research
Types of Validity
 Face validity: Relevance of a measurement appear obvious
 Content validity: Measurement of all variable components
 Construct validity: Development of construct based-on established theory
 Criterion validity: Measurement and in comparison with reference
 Internal validity: Extent to which a study accurately measures what it
claims to measure.
 External validity: Extent to which we can generalize the findings of a
study to other measures, settings or groups

5. 2/24/2023
5
Reliability in Research
 Extent or Degree to which a measure gives same results on repeated
observations, referred as repeatability or reproducibility
 Reliability refers to the consistency of the measure. High reliability
indicates that measurement system produces similar results under the
same conditions.
 Types of Reliability
a) Internal reliability: extent to which a measure is consistent
within itself
b) External Reliability: extent to which measure is consistent when
assessed over time and on different individuals, Test-retest and
Inter-rater method

6. 2/24/2023
6
Validity and Reliability in Research

7. 2/24/2023
7
Validity and Reliability in Research

8. 2/24/2023
8
Basic about Threats in Research
 Factors that cause or are source of an errors and bias in research
 Affects validity and reliability of research
 Threats in Research mainly can be categorized as
a) Threats to Validity
b) Threats to Reliability
 Appropriate research design, use of scientifically proven tools and
techniques, use of appropriate research method and approaches,
sound understanding of research on researcher are key for minimizing
threats in research.

9. 2/24/2023
Threats to Internal Validity
9
Threats to
Internal
Validity
Threats associated
with participants
Selection Attrition History Maturation
Threats associated
with Measurement
Regression
Instru-
mentation
Testing
effect

10. 2/24/2023
Threats to Internal Validity
• Selection; difference in participant
characteristics
• Confounding Variables
• Experimental Mortality; loss to
follow up due to illness, relocation,
absent
• Instrumentation; instrument
decay, data collector
characteristics and bias
• History; previous or concurrent
event influencing outcome or
result
1
0

11. 2/24/2023
Threats to Internal Validity
• Location; place of data collection or intervention conducted
• Maturation; change in study participant that might influence result
like ageing
• Testing; pretest influencing post-test
• Attitude of Subjects; subject opinion and participation, Hawthorne
effect
• Regression
• Diffusion; resentful demoralization
• Implementation; difference in execution/methods/procedure
11

12. 2/24/2023
Threats to External Validity
External validity is the extent to which
we can generalize the findings of a study
to other measures, settings or groups.
Referred as generalizability and
includes Population and ecological
validity.
 Sampling bias: Sample not
representative of population
 History
 Observer bias
 Hawthorne effect
12

13. 2/24/2023
Threats to Other Validity
Threats to External Validity
 Testing effect; Pre and post test affecting outcomes
 Situation effect
Threats to other Validity
• Threats to construct validity- faulty construct in research designing
• Threats to statistical conclusion validity due to Type I and II errors
13

14. 2/24/2023
Threats to Validity in Qualitative Research
 Researcher bias refers to any kind of negative influence of researcher’s
knowledge, or assumptions, of the study, including the influence of his or
her assumptions of the design, analysis or, even, sampling strategy.
 Reactivity refers to a possible influence of the researcher on the studied
situation and people.
 Participant bias refers to situation where participants do not provide
honest responses for any reason, which may include them perceiving a
given topic as a threat, or them being willing to ‘please’ the researcher
with responses they believe are desirable.
14

15. 2/24/2023
Threats to Reliability
Reliability: repeatability, reproducibility referred sometimes as
dependability in qualitative researches
Threats to reliability: Factors that cause or are source of an errors in
research
 Researcher (Observer) error: human error
o Inter-observer variation
o Intra-observer variation
 Environmental changes
 Participant changes; change in participant during research process
15

16. Construction and use of appropriate
research tools to reduce threats

17. 2/24/2023
Key to minimizing threats in research
 Ensuring an appropriate methodology as per characteristics of study
 Standardization of the conditions under which the research study is
carried out will help minimize threats from history and instrumentation.
 Obtain as much information as possible about the participants in the
research study minimizes threats from mortality and selection.
 Obtain as much information as possible about the procedural details of
the research study, for example, where and when the study occurs,
minimizing threats from history and instrumentation.
17
Minimizing Reliability & Validity Threats in Research

18. 2/24/2023
Key to minimizing threats in research
 Choose an appropriate research design
 Replications counter almost all threats by enhancing generalizability to
other settings, populations and conditions.
• Field experiments; pre-testing counter testing and situation effects by
using natural contexts.
• Probability Sampling counters selection bias
• Recalibration or reprocessing also counters selection bias using
algorithms to correct weighting of factors (e.g., age) within study samples.
18
Minimizing Reliability & Validity Threats in Research

19. 2/24/2023 19
Minimizing Reliability & Validity Threats in Research
Key to minimizing threats in Qualitative research

20. 2/24/2023
Key to minimizing reliability threats in research
 Researcher Reliability: ensuring researcher, data collector and
enumerator well trained and aware of an purpose of the study, check
coding to ensure codes are entered accurately
 Variations in observations: shifting time and place of an observations
 Test-retest method
 Split-half method: employed to ensure reliability by soliciting a number of
answers from a participant to the same question asked in a variety of
ways.
20
Minimizing Reliability & Validity Threats in Research

21. 2/24/2023
Construction of appropriate tool to minimize Threats
Key consideration for choosing an appropriate tool
 Study design and the objectives of study
 Data collection technique
 Research settings
 Nature and characteristics of study participants
 Ethical considerations
 Design validity and test validity
21

22. 2/24/2023
Construction of appropriate tool to minimize Threats
Process for Tool Development
22
Step I
Planning &
Identification of Tool
• Setting objectives of
study
• Listing variables under
study
• Designing tool based
on technique
Step II
Tool Development
• Transforming variables
under study into tool
like questionnaires,
checklists, guideline
• Assessing tools with
objectives of study
Step III
Tool Validation
• Pre-testing (Field
Experiment)
• Assessment of Internal
reliability and validity
(Cronbach’s alpha-split
half method, test-
retest, parallel method)

23. 2/24/2023
Appropriate tool to minimize Threats
Choosing an appropriate tool
23
Threats to Validity and
Reliability
Appropriate Tool to minimize Threats
Selection of study participants-
Selection bias
Inclusion & exclusion criteria defining study
participants, eligibility checklists
FGD participant Guidelines, IDI, KII Guidelines
Attrition Follow-up timeline provided to study participants
Maturation Maturation factor ‘of possible significant impact’
addressing in Questionnaires

24. 2/24/2023
Appropriate tool to minimize Threats
Choosing an appropriate tool
24
Threats to Validity
and Reliability
Appropriate Tool to minimize Threats
Cofounding variables Questionaries addressing confounding variables,
Guidelines for Interview & group discussion
Regression & statistical
bias
Use of statistical tool based on data, addressing outlier,
skewness
Instrumentation Test-retest method, Test record & report

25. 2/24/2023
Construction of appropriate tool to minimize Threats
Key consideration for choosing an appropriate tool
25
Threats to Validity and
Reliability
Appropriate Tool to minimize Threats
Researcher bias Checklists ensuring researcher competencies (Knowledge,
researching procedure)
Measurement bias
Reliability Threat
Management
Use of appropriate instruments for measurement, Calibration
record
Test-retest reliability
Parallel reliability- Observation checklist, Interview Guideline
Inter-rater reliability
Inter-consistency- Split half method

26. 2/24/2023
References
• Babbie, E. R. (2010) “The Practice of Social Research” Cengage Learning
• Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth.
• Barlow, D. H., & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.).
New York: Pergamon.
• Cooper, J. O., Heron, T. E., & Heward, W. L. (1987). Applied behavior analysis. New York: Macmillan Publishing Co.
• Johnson, S. M., & Bolstad, O. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions
for field research. In L. A. Hamerlynck, L. C. Handy, & E. J. Mash (Eds.), Behavior change: Methodology, concepts and
practice (pp. 7-67). Champaign, IL: Research Press.
• Kazdin, A. E. (1977). Artefact, bias, and complexity of assessment: the ABCs of reliability. Journal of Applied Behavior
Analysis, 10, 141-150.
• Kerlinger, F. N. (1964). Foundations of behavioral research: Educational and psychological inquiry. New York: Holt,
Rinehart and Winston Inc.
• Johnston, J. M., & Pennypacker, H. S. (1993). Strategies and tactics of behavioral research (2nd ed.). Hillsdale, NJ:
Lawrence Erlbaum Associates.
• Cohen, L., Manion, L., Morrison, K, & Morrison, R.B. (2007) “Research methods in education” Routledge
• Wilson, J. (2010) “Essentials of Business Research: A Guide to Doing Your Research Project” SAGE Publications
26

27. 2/24/2023 27
Thank You!
Note: It is important to understand that threats to research reliability and validity can
never be totally eliminated, however researchers need to strive to minimize this threat as
much as possible.