3. CHOOSING AN APPROPRIATE
STUDY DESIGN
The study design is the plan adopted by the
researcher in the conduct of a study. It is important
that the researcher selects the appropriate study
design to minimize errors and avoid reaching wrong
conclusions. This chapter defines, describes and
illustrates the most commonly used experimental and
non-experimental research designs.
11. WHAT IS A RESEARCH
DESIGN?
A research design is the “blue print” of the study. It guides
the collection, measurement and analysis of data (Cooper and
Schindler, 2001). It is a plan or course of action which the
research follows in order to answer the research question/s or
solve the research problem (Sanchez, et. al., 1966). The
design becomes the basis for determining what data will be
collected, and how they will be analyzed and interpreted.
12. A good research requires a good
design. The use of an
appropriate design minimizes
the occurrence of error in the
conduct of the study and in the
conclusions drawn from the
study.
13. Before the research is
implemented, the researcher must
already be able to determine the
research design he/she intends to
use. Will he/she use an
experimental design or a non-
experimental design?
14. A wrong choice of a design puts at risk the
validity and the reliability of the study. When
this happens, it is quite difficult to find the real
answer to a research question, because there
could be some rival hypotheses that can
explain the occurrence of a problem. The
selection of an appropriate study design
can minimize possible errors by maximizing
reliability and validity of the data.
15. RELIABILITY
Reliability refers to the consistency, stability
and dependability of data. A reliable measuring
device is one which, if used for the second
time will yield the same results as it did the first
time. If the results are substantially different,
the measurement is unreliable.
16. VALIDITY
Validity refers to the extent to which a measurement
does what it is supposed to do, which is to measure
what it intends to measure. Valid data are not
only reliable, but also true and sound. A researcher
must select a research design that will yield a true
and accurate information and avoid factors that can
invalidate study results.
17. VALIDITY THREATS
There are many threats to validity.
The most common of them are
history, selection, testing,
instrumentation, maturation, and
mortality.
18. VALIDITY THREATS
History. Sometimes there events in the life of
a research project, but which are not part of
the project, that can increase or decrease
the expected project outcomes. These
events are not expected, they just happen
and they produce effects that can invalidate
study results.
19. VALIDITY THREATS
For example, in a study of the “Effect of Anti-
Smoking Campaign on Cigarette Consumption
Among Young Adults in City A,” an intensive
information campaign against smoking was
launched in order to discourage smoking among
young adults. Anti-smoking messages were
disseminated on radio, television, and newspapers
daily for one month.
20. VALIDITY THREATS
In the course of the campaign, a cigarette company also launched
a product promotion, offering a free trip to Europe for the
costumer and dealer who could collect and submit the most
number of empty packs of the cigarette brand being promoted. A
month after the launching of the anti-smoking campaign
consumption in the study area. The researcher might conclude
that the campaign was a failure. The conclusion here would
be invalid because of the high possibility that the cigarette
promotion (the historical event) may have contributed to the
increase in cigarette consumption.
21. VALIDITY THREATS
Selection. In an experimental study, a threat to validity
occurs when the elements or subjects for the
experimental group is very difficult from those
selected for the control group. For instance, if at the
beginning the experiment, the experimental group
already has an advantage over the control group in
terms of the focus of the study, this difference will
definitely affect the results of the study.
22. VALIDITY THREATS
For example, in an experiment conducted to
determine if using games and puzzles as instructional
aids can improve performances of college freshmen
in Basic Math, the teacher used games and puzzles
in the experimental group, but did not use them in
he control group. After the experiment, it was found
that the experimental group got significantly better
grades in the subject than the control group.
23. VALIDITY THREATS
It was discovered, however, that most of the students
in the experimental group had very good grades in
high school math, while most of those in the control
group had average grades only. Attributing the better
performance of the experimental group to use the
games ad puzzles can be questioned. To avoid this
validity threat, the experimental and the control group
should have similar characteristics at the beginning.
24. VALIDITY THREATS
Testing. Whenever a pretest is given, it may take the
examinees “test wise,” and this can therefore affect the post-
test results. Research subjects who have been given a pretest
may remember some of the test items/questions for which they
may search answers and get these correct when they take
the post-test. Better performance in the post-test might be due
to the effect of the pretest and not necessarily to the
intervention or treatment.
25. VALIDITY THREATS
Instrumentation. When a research interment, such
as questionnaire or a measuring device, like
weighing scale or a thermometer is changed during
the study period or between the pretest and
the post-test, the change could result in an effect
that is independent of the intervention and yet, may
be attributed to it.
26. For example, in a survey study, an
instrumentation effect may be caused by an
interviewer who after conducting the pre-test
interview becomes more experienced in
interviewing. The interview/s experience will
enable him/her to generate better and/or more
complete information during the post test than
what was collected during pretest.
27. In a biomedical study, the use of an
unreliable device, like a scale
that badly needs calibration, a
contaminated syringe, or a very old
litmus paper may also threaten the
validity of test results.
28. Maturation. People and things change over
time. In other words they become more
mature, and this change can threaten the
validity of conclusions. Research subjects can
get tired, hungry, or bored during the duration
of the project. If the effect of a project is
measured with a test, their tiredness or
boredom can result in scores lower than
their “true” scores..
29. On the other hand, the subjects
may become more experienced,
more knowledgeable as they grow
older and as a result they may get
higher scores than than they did in
the pretest. In this regard the
changes can not be attributed to the
intervention.
30. Mortality. In studies that take a long time to finish,
say, one year or more, like cohort studies, where
the subjects (the same people) are followed up
over time, some cases may drop out, thus
resulting in a loss of cases. Some cases may have
transferred residence and are difficult to locate
during the follow-up interview. Cases which cannot
be contacted cannot be followed-up. This loss,
called mortality, may distort findings and
conclusions, if substantial and if it has introduced a
bias to the sample.
31. The loss could result in a big
difference between he pre-test and
the post-test results. This chafe may
be wrongly attributed to the
intervention, thus, threaten the
validity of the conclusions.
32. Commonly Used Research Designs (Campbell and
Stanley, 1968, Parel, et al., 1985, Fisher, et al.,
1994)
The choice of a research design depends on the
objectives of the study. There are many types of
research designs that can be used in basic and
experimental research. Described here are doe of
the most frequently used designs. They are
classified into: non- or pre-experimental designs,
true experimental designs, and quasi-experimental
designs.
33. Non/Pre-Experimental
Designs
Non-experimental designs are appropriate for collecting
descriptive information about a population or subjects of a
study. They are appropriate for descriptive studies, like
profile studies, exploratory studies, and for doing small case
studies. They are also ideal for diagnostic studies intended to
determine the effect or impact of a certain intervention or
treatment. Three non-experimental designs are described
below. They are the post-test only or after only design, the
pretest-posttest design, and the static group comparison.
34. Posttest Only Design or After-
Only Survey
Time
X O
(Observation/Testing/Survey)
35. The design is also called as one shot
survey because the data are collected
only once (O). This design is used when
the study objective is to describe a
situation/condition of a study population
as it exists, or to determine/describe the
characteristics of a
population/respondents. There is no
baseline data.
36. This design is cheap and easy to
conduct, but results canon the
conclusive in terms of causality or
effect of an intervention. It is not,
however, recommended for evaluation
studies that intend to measure the
effect of a program intervention, like
training.
38. This design is used when the study wants to know
the change in characteristics (e.g. knowledge,
attitude, practices) of the study population
(students, nurses, managers, clients, etc) in a
given area. A survey observation, or testing is
conducted before an intervention is introduced (O1)
. After a period of time the survey, observation or
testing is repeated (O2) and the results of pretest
(before) and the post-test (after) are compared to
determine change/s.
39. For example, if a researcher wants to know if an
information campaign against drug/substance
abuse in a certain city has reduced drug use in the
area after the campaign, a survey before and after
the campaign can be conducted. No “control” area
(area where no campaign is conducted), however,
is surveyed. With the absence of a control area,
this design cannot be considered an experimental
design. Any reduction in drug use overtime, cannot
be solely/conclusively attributed to the intervention
(campaign).
41. Control Group
In the static group design, there are two groups involved, an experimental
group and a control group. The experimental group receives or is exposed
to the intervention/treatment (X). This is followed by a measurement (O1),
the result of which is compared to the result of the
measurement/observation from a control group (O2) that did not
receive the intervention. The random process, however, was not used in
the assignment of subjects to the experimental and control groups
(indicated by a broken line). The problem with this design is the validity
threat of selection and mortality. It is possible that the two groups differ
greatly on the basis of the main variables of the study (selection) or some
subjects in the experimental group may drop out and be lost to follow-up
or second observation/testing (mortality).
42. True Experimental Designs
In true experimental designs, subjects are randomly assigned
to the experimental group and the control group to achieve
pre-intervention/treatment equality of the two groups, validity
threats are avoided. Before a researcher decides on an
alternative design, the feasibility of using true experimental
designs must first be considered. The two most frequently
used true experimental designs are the pretest-posttest
control group design and the post-test control group design.
44. In the pretest-postest control group design, the experimental
group is exposed to or covered by an intervention or
treatment (X), for example, training or a new strategy, while
the control group is left alone or given another kind of
treatment. Before the intervention/treatment is introduced to
the experimental group (O1) and control group (O2) using the
same device/instrument. The pre-intervention
survey/observation/test serves as pretest and the data
collected serve as baseline data. After the introduction of the
intervention in the experimental group or area, an evaluation
survey/observation/testing is conducted in both experimental
group/area (O3) and the control group/area (O4), using the
same instrument used in both during the pretest. The results
serve as the post-test/endline data.
45. The baseline (pretest) and end line (post-
test) data are compared. If the change in
the “impact/effect indicator/s” or dependent
variable/s is significantly better in the
experimental area/group than the change in
the control area/group, then the intervention
is considered effective. If not, then the
intervention is said to have had no effect.
46. Posttest Only Control Group
Design
RA
Experimental group
Control group
X O1
O2
Posttest
47. The Posttest Only Control Group design is
also used to determine the effects of an
intervention or treatment introduced to a
group of subjects (people/objects). As in
the pretest-posttest control group design,
at least two groups or areas (e.g. women
groups, communities, provinces) with
virtually the same characteristics are
chosen and randomly assigned (RA) to the
control and experimental group.
48. The experimental group or area is exposed to or
covered by an intervention/treatment, while the
control group is left alone. No pretest/pre-
intervention study is conducted. The experimental
and the control groups are assumed to have
similar characteristics at the start of the
study. After the introduction of an intervention in
the experiment group or area, an evaluation
survey/observation/testing is conducted in both
experimental and the control groups or areas,
using the same “fair” instrument.
49. The data gathered from the experimental
and control groups are compared. If the
experimental group or area
shows significantly better results than the
control area/group with respect to
the “impact/effect indicator/s” or
dependent variable/s, the intervention or
treatment is considered effective. If not,
then, the intervention is not effective.
50. Quasi-experimental Designs
In filed studies, it is very difficult to meet the
random assignment criterion of a true experimental
design. In this situation, a quasi-experimental design is
recommended. Quasi-experimental designs are nearly
the same as the true experimental designs, except that
the former do not have restrictions of random
assignment. The two most commonly used quasi-
experimental designs are the non-equivalent control
group design and the time series design.
52. In field research, it is possible to compare an
experimental group with a similar, but no necessarily
equivalent group. The two groups need only to
have “collective similarity,” which means that they
should have more or less the same characteristics in
terms of aspects which are relevant to the study. For
example, if one wants to determine the impact of an
educational campaign on school attendance of
children, the experimental and the control areas
should have more or less the same socioeconomic
characteristics, because these factors may also
affect school attendance.
53. As in the pretest-posttest control group design,
the intervention or treatment is introduced to the
experimental group, but withheld from the
control group. Before the introduction of the
intervention, a survey/observation/testing is
conducted in both the experimental
group (O1) and the control group (O2). After the
introduction of the intervention to the
experimental group, another observation/testing
(post-test) is conducted to both
groups (O3 and O4).
54. The pretest can be used to determine whether the two
groups have truly “collective similarity” at the start of
the experiment. The results of the two post-
tests (O3 and O4) will also be compared. The
intervention is effective if the change in the
impact/effect indicators in the experimental
group (O3 minus O1) is significantly higher/better than
the change in the impact/effect indicators in the control
group (O4 minus O3). If not, then the
intervention/treatment cannot be considered effective.
This design is a good one for evaluating training
programs, and other community interventions.
56. The time series design is similar to the non-experimental
pretest-posttest design except that, it has repeated
observations/measurements before and after the
intervention (X). Before the introduction of the
intervention/treatment, a measurement/observation with
respect to the impact/effect indicators will conducted
several times at a regular interval, say, every 30
days; (O1 , O2, O3), and then after the intervention, another
series of measurement/observation will be
conducted (O4 , O5, O6), also at the same time interval as
the first. The same measuring instrument/device should be
used at all times.
57. The result or pattern of the observations or testing
in the first series of measurements will be
compared with that in the series of measurements
after the intervention. If the post-intervention result
or pattern is better than that of the pre-intervention
series, then the intervention can be considered
effective. However, if the pre-intervention and post
intervention results or patterns are the same, or the
post intervention result is not significantly better
than that of the pre-intervention. then the
intervention cannot be considered effective.
58. For example, one wants to evaluate the effect of a
feeding program which is intended to improve the
nutritional status of pre-school children in a
barangay. Before the introduction of the feeding
program, the children (program beneficiaries) will
be weighed (measured) several times at regular
interval. say, every 30 days; (O1 , O2, O3) and then
after the feeding program another series of
weighing (measurement) will be
conducted (O4 , O5, O6) also at the same time
interval as the first series (every 30 days).
59. In order for the feeding program to be considered
effective in improving the nutritional status of the
children, the children’s weights should improve
after the feeding program. Since the children are
also growing, increase in weight may also be
observed during the series of pre-intervention
changes must be significantly better than the pre-
intervention changes. If norm the feeding program
could not be considered as having effectively
improved the nutritional status of the children.
