2. Experimental Research Designs have
Two Purposes:
• …to provide
answers to
research
research
questions
• ...to control variance
(differences)
E-Content- Module 12-Research Methods and Statistics.
2
SXCE,Patna
3. The main function of the
experimental research
design is to control variance.
• Principle: maximize
systematic variance,
systematic variance,
control extraneous
systematic variance,
and minimize error
variance.
In other words control variance.
E-Content- Module 12-Research Methods and Statistics.
3
SXCE,Patna
4. Therefore the researcher attempts to:
• maximize the variance of the variable (s) of the research
hypothesis (i.e., maximize the difference in the dependent
variable [outcome] caused by maximizing the differences in the
independent variable [treatment]).
•
control the variance of extraneous or "unwanted "
•
control the variance of extraneous or "unwanted " variables
that may have an effect on the experimental outcomes , but
which he/she is not interested (limit factors other than the
treatment (IV) that could be causing differences in the
outcome (DV) .
• minimize the error or random variance (i.e., avoid unreliable
measurement instruments which have high errors of
measurement ).
E-Content- Module 12-Research Methods and Statistics.
4
SXCE,Patna
5. Maximization of Experimental
Variance
• experimental variance
– the variance due to the
manipulated (i.e., treatment)
or attribute (i.e., gender)
variables (IV)
variables (IV)
research precept:
– design, plan and conduct
research so that experimental
conditions are as different as
possible on the independent
variable.
E-Content- Module 12-Research Methods and Statistics.
5
SXCE,Patna
6. Control of Extraneous
Variables (EV)
• eliminate the variable (i.e., if you are worried
about gender, only include one gender in the study).
• randomization (i.e., if you randomly assign subjects to
groups, the extraneous variable should be equally
distributed among the groups)
• build it into the design – make it a moderator variable (i.e.,
• build it into the design – make it a moderator variable (i.e., if you
are worried about gender, build it into the analysis [2- way
ANOVA]—we’ll learn about this later)
• match subjects (i.e., match the characteristics of subjects and
put one of each matched pair in each group)
• statistically equate groups (i.e., use ANCOVA [Analysis of
Covariance] to analyze the data with the extraneous variable
used as a covariate—we’ll learn about this later)
E-Content- Module 12-Research Methods and Statistics.
6
SXCE,Patna
7. Minimizing Error Variance has
Two Principle Aspects:
• reduction of errors of
measurement through
controlled conditions (i.e.,
standardize testing
procedures)
• increase in the reliability of
measures (i.e., revise test
instruments or find more
reliable ones)
E-Content- Module 12-Research Methods and Statistics.
7
SXCE,Patna
8. Experimenta
l Designs
Should be
Developed to
Ensure
Ensure
Internal and
External
Validity of
the Study
E-Content- Module 12-Research Methods and Statistics.
8
SXCE,Patna
9. Internal Validity:
• Are the results of the
study (DV) caused by
the factors included in
the study (IV) or are
they caused by other
factors (EV) which
were not part of the
study?
There are 16 common
threats to internal validity.
E-Content- Module 12-Research Methods and Statistics.
9
SXCE,Patna
10. Subject
Characteristics
Threats to Internal Validity
(Selection Bias/Differential Selection ) -- The groups
may have been different from the start. If you were
testing instructional strategies to improve reading
and one group enjoyed reading more than the other
group , they may improve more in their reading
because they enjoy it, rather than the instructional
strategy you used.
Characteristics
E-Content- Module 12-Research Methods and Statistics.
10
SXCE,Patna
11. (Mortality) -- All of the high or low scoring
subject may have dropped out or were
missing from one of the groups. If we
collected posttest data on a day when the
Loss of Subjects
Threats to Internal Validity
collected posttest data on a day when the
honor society was on field trip at the
treatment school, the mean for the
treatment group would probably be much
lower than it really should have been.
E-Content- Module 12-Research Methods and Statistics.
11
SXCE,Patna
12. Perhaps one group was at a
disadvantage because of their
location. The city may have
Location
Threats to Internal Validity
been demolishing a building
next to one of the schools in
our study and there are
constant distractions which
interfere with our treatment.
E-Content- Module 12-Research Methods and Statistics.
12
SXCE,Patna
13. The testing instruments may not be
scores similarly. Perhaps the person
grading the posttest is fatigued and
pays less attention to the last set of
papers reviewed. It may be that those
papers are from one of our groups and
Threats to Internal Validity
papers are from one of our groups and
will received different scores than the
earlier group's papers
Instrumentation
Instrument Decay
E-Content- Module 12-Research Methods and Statistics.
13
SXCE,Patna
14. The subjects of one group may react differently to the
data collector than the other group. A male
interviewing males and females about their attitudes
toward a type of math instruction may not receive the
same responses from females as a female
interviewing females would.
Threats to Internal Validity
Data Collector
Characteristics
E-Content- Module 12-Research Methods and Statistics.
14
SXCE,Patna
15. The person collecting data my favors one group,
or some characteristic some subject possess,
over another. A principal who favors strict
classroom management may rate students'
attention under different teaching conditions with
a bias toward one of the teaching conditions.
Threats to Internal Validity
Data
Collector
Bias
E-Content- Module 12-Research Methods and Statistics.
15
SXCE,Patna
16. The act of taking a pretest or posttest may influence the
results of the experiment. Suppose we were conducting
a unit to increase student sensitivity to prejudice. As a
pretest we have the control and treatment groups watch
Threats to Internal Validity
Testing
pretest we have the control and treatment groups watch
Shindler's List and write a reaction essay. The pretest
may have actually increased both groups' sensitivity
and we find that our treatment groups didn't score any
higher on a posttest given later than the control group
did. If we hadn't given the pretest, we might have seen
differences in the groups at the end of the study.
E-Content- Module 12-Research Methods and Statistics.
16
SXCE,Patna
17. Threats to Internal Validity
Something may happen at one site during our study that
influences the results. Perhaps a classmate dies in a car
accident at the control site for a study teaching children bike
safety. The control group may actually demonstrate more
concern about bike safety than the treatment group.
History
E-Content- Module 12-Research Methods and Statistics.
17
SXCE,Patna
18. There may be natural
changes in the subjects
that can account for the
changes found in a study.
A critical thinking unit may
Threats to Internal Validity
A critical thinking unit may
appear more effective if it
taught during a time when
children are developing
abstract reasoning.
Maturation
E-Content- Module 12-Research Methods and Statistics.
18
SXCE,Patna
19. The subjects may respond differently just because they are
being studied. The name comes from a classic study in
Threats to Internal Validity
Hawthorne Effect
which researchers were studying the effect of lighting on
worker productivity. As the intensity of the factory lights
increased, so did the worker productivity. One researcher
suggested that they reverse the treatment and lower the
lights. The productivity of the workers continued to increase.
It appears that being observed by the researchers was
increasing productivity, not the intensity of the lights.
E-Content- Module 12-Research Methods and Statistics.
19
SXCE,Patna
20. One group may view that it is in competition with the other
group and may work harder than they would under normal
circumstances. This generally is applied to the control group
"taking on" the treatment group. The terms refers to the
classic story of John Henry laying railroad track.
Threats to Internal Validity
John
Henry
Effect
E-Content- Module 12-Research Methods and Statistics.
20
SXCE,Patna
21. The control group may become discouraged
because it is not receiving the special attention
that is given to the treatment group. They may
perform lower than usual because of this.
Threats to Internal Validity
Resentful
Demoralization of
the Control Group
E-Content- Module 12-Research Methods and Statistics.
21
SXCE,Patna
22. (Statistical Regression) -- A class that scores
particularly low can be expected to score slightly
higher just by chance. Likewise, a class that
scores particularly high, will have a tendency to
Threats to Internal Validity
Regression
scores particularly high, will have a tendency to
score slightly lower by chance. The change in
these scores may have nothing to do with the
treatment.
E-Content- Module 12-Research Methods and Statistics.
22
SXCE,Patna
23. The treatment may not be implemented as
intended. A study where teachers are
asked to use student modeling techniques
may not show positive results, not because
modeling techniques don't work, but
Threats to Internal Validity
modeling techniques don't work, but
because the teacher didn't implement them
or didn't implement them as they were
designed.
Implementation
E-Content- Module 12-Research Methods and Statistics.
23
SXCE,Patna
24. Threats to Internal Validity
Someone may feel sorry for the control
group because they are not receiving much
attention and give them special treatment.
For example, a researcher could be studying
the effect of laptop computers on students'
attitudes toward math. The teacher feels
sorry for the class that doesn't have
Compensatory
Equalization of Treatment
sorry for the class that doesn't have
computers and sponsors a popcorn party
during math class. The control group begins
to develop a more positive attitude about
mathematics.
E-Content- Module 12-Research Methods and Statistics.
24
SXCE,Patna
25. Experimental Treatment
Diffusion
Threats to Internal Validity
Sometimes the control group
actually implements the
treatment. If two different
treatment. If two different
techniques are being tested in
two different third grades in the
same building, the teachers
may share what they are
doing. Unconsciously, the
control may use of the
techniques she or he learned
from the treatment teacher.
E-Content- Module 12-Research Methods and Statistics.
25
SXCE,Patna
26. Once the researchers are confident that
the outcome (dependent variable) of the
experiment they are designing is the
result of their treatment
(independent variable)
[internal validity],
[internal validity],
they determine for which
people or situations
the results of
their study apply
[external validity].
E-Content- Module 12-Research Methods and Statistics.
26
SXCE,Patna
27. External
Validity:
• Are the results of the study generalizable to
other populations and settings?
External validity comes in two forms: population and ecological.
E-Content- Module 12-Research Methods and Statistics.
27
SXCE,Patna
28. 1. ...the extent to which one can generalize from the study
sample to a defined population--
If the sample is drawn from an accessible population, rather
Threats to External Validity
(Population)
Population Validity is the extent to which the results of a study
can be generalized from the specific sample that was studied to a
larger group of subjects. It involves...
If the sample is drawn from an accessible population, rather
than the target population, generalizing the research results
from the accessible population to the target population is
risky.
2. ...the extent to which personological variables interact with
treatment effects--
If the study is an experiment, it may be possible that different
results might be found with students at different grades (a
personological variable).
E-Content- Module 12-Research Methods and Statistics.
28
SXCE,Patna
29. Ecological Validity is the
extent to which the results of an experiment can be
generalized from the set of environmental conditions
created by the researcher to other environmental
conditions (settings and conditions).
Threats to External Validity
(Ecological)
conditions (settings and conditions).
There are 10 common
threats to external
validity.
E-Content- Module 12-Research Methods and Statistics.
29
SXCE,Patna
30. Threats to External Validity
(Ecological) Explicit
description of
the
experimental
treatment
(not sufficiently described for others to
replicate) If the researcher fails to
adequately describe how he or she
conducted a study, it is difficult to
determine whether the results are
applicable to other settings.
treatment
E-Content- Module 12-Research Methods and Statistics.
30
SXCE,Patna
31. (catalyst effect)
Threats to External Validity
(Ecological)
Multiple-treatment
interference
(catalyst effect)
If a researcher were to apply
several treatments, it is difficult to
determine how well each of the
treatments would work individually.
It might be that only the combination
of the treatments is effective.
E-Content- Module 12-Research Methods and Statistics.
31
SXCE,Patna
32. (attention causes differences)
Subjects perform differently because they
know they are being studied. "...External
Threats to External Validity
(Ecological)
Hawthorne effect
know they are being studied. "...External
validity of the experiment is jeopardized
because the findings might not generalize
to a situation in which researchers or
others who were involved in the research
are not present" (Gall, Borg, & Gall, 1996,
p. 475)
E-Content- Module 12-Research Methods and Statistics.
32
SXCE,Patna
33. Threats to External Validity
(Ecological)
Novelty and
disruption effect
(anything different makes a difference)
A treatment may work because it is novel and the subjects
respond to the uniqueness, rather than the actual
treatment. The opposite may also occur, the treatment
may not work because it is unique, but given time for the
subjects to adjust to it, it might have worked.
E-Content- Module 12-Research Methods and Statistics.
33
SXCE,Patna
34. (it only works with this experimenter)
Threats to External Validity
(Ecological)
Experimenter effect
(it only works with this experimenter)
The treatment might have worked
because of the person implementing it.
Given a different person, the treatment
might not work at all.
E-Content- Module 12-Research Methods and Statistics.
34
SXCE,Patna
35. (pretest sets the stage)
A treatment might only work if a
pretest is given. Because they
Threats to External Validity
(Ecological)
Pretest sensitization
pretest is given. Because they
have taken a pretest, the subjects
may be more sensitive to the
treatment. Had they not taken a
pretest, the treatment would not
have worked.
E-Content- Module 12-Research Methods and Statistics.
35
SXCE,Patna
36. (posttest helps treatment "fall into place")
The posttest can become a learning
Threats to External Validity
(Ecological)
Posttest sensitization
The posttest can become a learning
experience. "For example, the posttest might
cause certain ideas presented during the
treatment to 'fall into place' " (p. 477). If the
subjects had not taken a posttest, the
treatment would not have worked.
E-Content- Module 12-Research Methods and Statistics.
36
SXCE,Patna
37. Interaction of
history and
treatment effect
Threats to External Validity
(Ecological)
treatment effect
(...to everything there is a time...)
Not only should researchers be cautious about
generalizing to other population, caution should be
taken to generalize to a different time period. As
time passes, the conditions under which treatments
work change.
E-Content- Module 12-Research Methods and Statistics.
37
SXCE,Patna
38. Threats to External Validity
(Ecological)
Measurement of
the dependent
variable
(maybe only works with M/C tests)
A treatment may only be evident with
certain types of measurements. A teaching
method may produce superior results
when its effectiveness is tested with an
essay test, but show no differences when
the effectiveness is measured with a
multiple choice test.
variable
E-Content- Module 12-Research Methods and Statistics.
38
SXCE,Patna
39. Interaction of time
of measurement
and treatment
Threats to External Validity
(Ecological)
and treatment
effect
(it takes a while for the treatment to kick in)
It may be that the treatment effect does not occur until several
weeks after the end of the treatment. In this situation, a
posttest at the end of the treatment would show no impact,
but a posttest a month later might show an impact.
E-Content- Module 12-Research Methods and Statistics.
39
SXCE,Patna
40. First, and foremost, an experiment
must have internal validity. If the
researchers cannot be certain that the
results of the experiment are
dependent on the treatment, it does
not matter to which people or
situations they wish to generalize
(apply) their findings. The importance
(apply) their findings. The importance
of external validity is reliant on
having internal validity in much the same way that the validity
of a measurement instrument is reliant on the instrument
being reliable.
However, the more tightly experimenters design their study,
the more they limit the populations and settings to whom they
can generalize their findings.
E-Content- Module 12-Research Methods and Statistics.
40
SXCE,Patna
42. Suppose a researcher wants to
study the effect of a reading
program on reading achievement.
She might implement the reading program with a
group of students at the beginning of the school year
and measure their achievement at the end of
the year.
X
the year.
O
This simple design is known as a one-shot case
study design.
E-Content- Module 12-Research Methods and Statistics.
42
SXCE,Patna
43. Unfortunately, the students’ end of year reading scores
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
We also do not know whether the students’ reading skills
actually changed from the start to end of the school year.
We could improve on this design by giving a pretest at the
start of the study.
X O
start of the study.
O
This is known as a one-group pretest-posttest design.
E-Content- Module 12-Research Methods and Statistics.
43
SXCE,Patna
44. Unfortunately, the students’ end of year reading scores still
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
Our researcher may wish to have a comparison group.
X O
O
O
O
This is a static-group pretest-posttest design.
E-Content- Module 12-Research Methods and Statistics.
44
SXCE,Patna
45. If our researcher believes that the pretest has an impact
on the results of the study, she might not include it.
X O
O
O
O
This is a static-group comparison design.
E-Content- Module 12-Research Methods and Statistics.
45
SXCE,Patna
46. Because our researcher did not pretest, she might wish
to randomly assign subjects to treatment and control
group.
Random assignment of subject to groups should spread
the variety of extraneous characteristics that subjects
possess equally across both groups.
X O
O
O
O
R
R
This is a randomized posttest-only, control group design.
E-Content- Module 12-Research Methods and Statistics.
46
SXCE,Patna
47. Of course, our researcher could also include a pretest
with her random assignment.
X O
O
O
O
This is a randomized pretest-posttest control group design.
R
R
E-Content- Module 12-Research Methods and Statistics.
47
SXCE,Patna
48. Occasionally researchers combine the randomized
pretest-posttest control group design with the
randomized posttest-only, control group design.
X O
O
O
O
R
R
This is a randomized Solomon four-group design.
X O
O
R
R
O
O
R
E-Content- Module 12-Research Methods and Statistics.
48
SXCE,Patna
49. With the randomized Solomon four-group design, all
groups are randomly assigned
X O
O
O
O
R
R
and given the posttest.
Two of the groups are given pretests.
One of the pretest
groups is assigned to
treatmentand one of
the non-pretest groups
X O
O
O
O
R
R
R the non-pretest groups
is assigned to
treatment.
E-Content- Module 12-Research Methods and Statistics.
49
SXCE,Patna
50. Each of the designs described in this section has
advantages and disadvantages that influence the
studies internal and external validity.
E-Content- Module 12-Research Methods and Statistics.
50
SXCE,Patna