1. Validity and validation in qualitative
and quantitative research:
A battle of truth?
Rolf B. Fasting (rolf.fasting@hioa.no)
Professor of Special Education Needs
Oslo and Akershus University College of Applied sciences
3. Validity: epistemological considerations
— Positivism - the “objective” truth
— Post-positivism - interpretivism
— Epistemological relativism:
„From a moderate constructivist position, realism
assumes that knowledge of both nature and other
minds depend on a reiterated sequence of
perceptions, cognitions, and inferences, all of which
may be questioned, rejected, and revised.’
(Jensen, 2002: 268 f.)
3
4. Validity: origin and emergence
Statistical Methods for Research Workers (Ronald Fisher
1925)
The Reliability and Validity of Tests: derivation and
interpretation of fundamental formulae concerned with
reliability and validity of tests and illustrative problems
(Louis Leon Thurstone 1931).
“Technical standards for evaluating tests and the
contents of test manuals” (APA 1950).
“Validity will be evaluated in terms of two major criteria.
First, and as a basic minimum, is what can be called internal
validity: did in fact the experimental stimulus make some
significant difference in this specific instance? The second
criterion is that of external validity, representativeness, or
generalizability: to what populations, settings, and variables
can this effect be generalized.”
(Campbell, 1957, p.297, emphasis in the original)
4
5. Diverging perspectives…
The concept of validity used in quantitative
research is rooted in positivism and
consequently the concept used in qualitative
research needs a different content and a
different basis
(Altheide & Johnson 1994)
5
7. Validity: a battle of truth, or a need of appropriate
strategies to face the threats of trustworthiness?
7
8. Validity and validation…
Validity: the approximate truth of an inference;
Shadish, Cook & Campbell 2002)
[D]ata in themselves cannot be valid or invalid;
what is at issue are the inferences drawn from
them
Hammersley & Atkinson (1983: 191)
Validity refers to the inferences drawn, not to the
processes in use
8
10. Construct validity…
validity of inferences:
- from theoretical framework or interest to indicators
(from what we have seen to what we “call” what we have seen)
“[T]o what extent are the constructs of theoretical interest
successfully operationalized in the research”
(Judd, Smith & Kidder 1991: 29)
In qualitative research indicators generally are first
observed, and constructs are «constructed» through the
process of analysis.
How well does the concept represent the indicators?
10
13. Validating construct validity
Random measurement errors
— To what extent would another observer (with the same
theoretical position, using the same tools, and at the
same time) observe the same things and make the
same interpretations?
— To what extent will we observe the same things if we
observe at another point of time?
13
14. Assessment of Writing Proficiency, grade 5 and 8:
16 schools; 320 scripts; 82 raters. (Oct. 2011)
Pair of raters – pair A Pair of raters – pair B Pair of raters – pair x
Assessed domains: Assessed domains: Assessed domains:
Communication Communication Communication
Content Content Content
Text structure Text structure Text structure
Language Language Language
Spelling Spelling Spelling
Punctuation Punctuation Punctuation
36 scripts 36 scripts 36 scripts
+ 4 scripts from pair B + 4 scripts from pair A + 4 scripts from pair A
+ 4 scripts from pair x + 4 scripts from pair x + 4 scripts from pair B
14
15. Internal validity….
validity of inferences:
- from an observed covariation to a causal interpretation
(to the interpretation that something is influenced by
another thing)
—no basis for causal interpretation until alternative
causal interpretations are eliminated or at least
shown to be unlikely
Is there any causal relationship between the
included variables?
15
16. Validating internal validity
Quantitative designs:
— Molar causation: the overall relation between a “treatment
package” and its effects
-- --
Qualitative designs:
— Molecular causation: which parts of a “treatment package”
are more or less responsible for which parts of the effects
through which mediational processes.
— Causal [molecular] descriptions: “[R]eferring to the actual
causal mechanisms and processes that are involved in
particular events and situations” (Maxwell, 2004: 9).
16
17. Statistical [conclusion] validity…
validity of inferences
- about covariation between variables or phenomena
(trivial or worthy of a substantial interpretation?)
Is a tendency substantial enough to be worthy of an
interpretation?
17
18. Validating statistical [conclusion] validity
In quantitative studies:
— if a tendency should be considered substantial enough
to be worthy of an interpretation (p. value)
— is there any reasonable evidence from which to infer
that the presumed cause and effect covary?
— If so; how strongly (effect size - f.x. Choen’s d)?
In qualitative studies
— Is the children’s change of behaviour in the classroom
from time A to time B a real change, or just a trivial
fluctuation?
18
20. External validity
validity of inferences
- inferences from the context of the study to a wider context or
to other contexts
- Generalization - Transferability
A study has good external validity to the extent it makes
possible non-statistical generalizations to or over relevant
individuals, situations and time with reasonable certainty
(Lund, 2002: 121)
20
21. External validity: Non-statistical generalization
Non-statistical generalization:
Generalisation over situations; individuals and
groups: mainly depending on similarities and
differences between situations and persons actually
studied, and the situations and persons we draw our
inferences about
Generalisation over time: mainly a question about
nearness in time, and what important changes may
have taken place in the meantime
(Lund, 2002: 121)
-- -- --
“Thorough” or “Thick” descriptions (Geertz 1973; Ryle 1968)
…qualitative tools when judgments are made about
transferring results to other situations
Cronbach 1975)
21
22. External validity
Cronbach’s (1975) advice concerning generalizations:
1. Consider the results (knowledge claims) to be
context-bound
2. Consider generalizations as work hypotheses
rather than conclusions
3. Study the same phenomenon in other contexts to
see whether we get the same results there
4. Pay attention to exceptions as well as to cases
confirming «the rule», as exceptions may indicate
context-specific conditions
22
23. Conclusion:
The relevance of the various aspects of validity
depends on what kind of inferences are drawn,
not on what kind of data used as basis for the
inferences
(rolf.fasting@hioa.no)
23
24. Further reading:
— Altheide, D.L. & Johnson, J.M. (1994). Criteria for assessing interpretive
validity in qualitative research. In N.K. Denzin & Y.S. Lincoln (eds.):
Handbook of Qualitative Research. Thousand Oaks, CA: SAGE
Publications.
— Cook, T.D. & Campbell, D.T. (1979). Quasi-Experimentation. Design and
Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin Company.
— Crocker, L. & Algina, J. (1986). Introduction to Classical and Modern
Test Theory. Fort Worth, TX: Harcourt Brace Jovanovich College
Publishers.
— Denzin, N.K. & Lincoln, Y.S. (2003). Paradigmatic controversies,
contradictions, and emerging confluences. In N.K. Denzin & Y.S. Lincoln
(eds.): The Landscape of Qualitative Research. Theories and Issues.
Thousand Oaks, CA: SAGE Publications.
— Fisher, R. A. (1935). The Design of Experiments. Endiburgh, Oliver & Boyd.
— Geertz, C. (1973). Thick Description: Toward an Interpretive Theory of
Culture. In The Interpretation of Cultures: Selected Essays. Clifford Geertz.
pp 3–30. New York: Basic Books.
— Guba, E.G. & Lincoln, Y.S. (1994). Competing paradigms in qualitative
research. In N.K. Denzin & Y.S. Lincoln (eds.): Handbook of Qualitative
24 Research. Thousand Oaks, CA: SAGE Publications.
25. — Hammersley, M. & Atkinson: (1983). Ethnography: Principles in Practice.
London: Tavistock Publications.
— Jensen, K.B. (2002). The complementarity of qualitative and quantitative
methodologies in media and communication research. In K.B. Jensen (ed.): A
Handbook of Media and Communication Research. Qualitative and
Quantitative Methodologies. London: Routledge.
— Judd, C.M., Smith, E.R. & Kidder, L.H. (1991). Research Methods in Social
Relations. Fort Worth, TX: Holt, Rinehart and Winston, Inc.
— Kvale, S. (1989). To validate is to question. In S. Kvale (ed.): Issues of
Validity in Qualitative Research. Lund: Studentlitteratur.
— Lund, T. (2005a). The qualitative-quantitative distinction: some comments.
Scandinavian Journal of Educational Research, 49(2), 115–132.
— Maxwell, J.A. (2004). Causal explanation, qualitative research and scientific
inquiry in education. Educational Researcher, 33(2), 3–11.
— Messick, S. (1995). Validity of psychological assessment. American
Psychologist, 50(9), 741–749.
— Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002). Experimental and Quasi-
experimental Designs for Generalized Causal Inference. Boston, MA:
Houghton Mifflin Company.
25
31. Aims
—ontological and epistemological fundation of
validity
—validity theory and correspondence theory
—validation validity.
31
32. Aims
—Validity: ontological and epistemological
consederations
—validity theory and correspondence theory
—validity and validation.
32
33. Validity - ontologial and epistemological
considerations
Validity within quantitative methodology:
… a post-positivist critical realist perspective.
Validity within qualitative methodolog:
—constructionist position (Guba and Lincoln 1994)
—critical realism (Maxwell 1994; Miles & Huberman 1994)
—subtle realism (Hammersley 1992)
—pragmatic realism (Alvesson & Sköldberg 1994)
33
34. Ontological and epistemological positions of
critical realism (Klaus Bruhn Jensen 2002; 268)
Ontological realism: Rejecting skepticist and
nominalist positions which variously have held
that no knowledge of the empirical world is
possible, or that reality is nothing but the sum of
our descriptions of it, realism reverses the burden
of proof, in a sense. A realist would argue that we
must assume the existence of reality as a limit
condition or regulatory ideal in order to account
for the sorts of natural and cultural phenomena
one encounters in science as well as in everyday
life.
34
35. The moderate posisjon
«Weak relativists believe that both the ontological
world and the worlds of ideology, interests,
values, hopes, and wishes play a role in the
construction of scientific knowledge.»
Shadish, Cook and Campbell (2002: 36)
35
36. Cook and Campbell’s validity system
(Shadish, Cook & Campbell, 2002).
Construct validity:
validity of inferences
from indicators to constructs
(from what we have seen to what we call what we have seen)
Statistical validity:
validity of inferences
about covariation between variables
(trivial or worthy of a substantial interpretation?)
Internal validity:
validity of inferences
from an observed covariation to a causal interpretation
(to the interpretation that something is influenced by another thing)
External validity:
validity of inferences
from the context of the study to a wider context or to other
contexts
36
