© ANGEL ROSE LEPAÑA

REPORT BY:

ANGEL ROSE LEPAÑA

KLARISSE CIARMIN FELIPE

DINO OCAMPO II

CHOOSING AN APPROPRIATE

STUDY DESIGN

The study design is the plan adopted by the

researcher in the conduct of a study. It is important

that the researcher selects the appropriate study

design to minimize errors and avoid reaching wrong

conclusions. This chapter defines, describes and

illustrates the most commonly used experimental and

non-experimental research designs.

LEARNING OBJECTIVES

After studying this

chapter, the learners

shall be able to:

1. define what

study design is,

2. explain the meaning

of reliability and

validity in research,

3. identify and describe

the different kinds of

validity threats and how

to handle each,

4. describe the different

kinds of research design

and distinguish the

experimental from the non-

experimental designs,

5. determine the

appropriate research

design/s for specific

types of research

problems, and

6. select an

appropriate design

for their own

research problem.

WHAT IS A RESEARCH DESIGN?

A research design is the “blue print” of the study. It guides

the collection, measurement and analysis of data (Cooper and

Schindler, 2001). It is a plan or course of action which the

research follows in order to answer the research question/s or

solve the research problem (Sanchez, et. al., 1966). The

design becomes the basis for determining what data will be

collected, and how they will be analyzed and interpreted.

A good research requires a good

design. The use of an

appropriate design minimizes

the occurrence of error in the

conduct of the study and in the

conclusions drawn from the

study.

Before the research is

implemented, the researcher must

already be able to determine the

research design he/she intends to

use. Will he/she use an

experimental design or a non-

experimental design?

A wrong choice of a design puts at risk the

validity and the reliability of the study. When

this happens, it is quite difficult to find the real

answer to a research question, because there

could be some rival hypotheses that can

explain the occurrence of a problem. The

selection of an appropriate study design

can minimize possible errors by maximizing

reliability and validity of the data.

RELIABILITYReliability refers to the consistency, stability

and dependability of data. A reliable measuring

device is one which, if used for the second

time will yield the same results as it did the first

time. If the results are substantially different,

the measurement is unreliable.

VALIDITYValidity refers to the extent to which a measurement

does what it is supposed to do, which is to measure

what it intends to measure. Valid data are not

only reliable, but also true and sound. A researcher

must select a research design that will yield a true

and accurate information and avoid factors that can

invalidate study results.

VALIDITY THREATS

There are many threats to validity.

The most common of them are

history, selection, testing,

instrumentation, maturation, and

mortality.

VALIDITY THREATS

History. Sometimes there events in the life of

a research project, but which are not part of

the project, that can increase or decrease

the expected project outcomes. These

events are not expected, they just happen

and they produce effects that can invalidate

study results.

VALIDITY THREATS

For example, in a study of the “Effect of Anti-

Smoking Campaign on Cigarette Consumption

Among Young Adults in City A,” an intensive

information campaign against smoking was

launched in order to discourage smoking among

young adults. Anti-smoking messages were

disseminated on radio, television, and newspapers

daily for one month.

VALIDITY THREATSIn the course of the campaign, a cigarette company also launched

a product promotion, offering a free trip to Europe for the

costumer and dealer who could collect and submit the most

number of empty packs of the cigarette brand being promoted. A

month after the launching of the anti-smoking campaign

consumption in the study area. The researcher might conclude

that the campaign was a failure. The conclusion here would

be invalid because of the high possibility that the cigarette

promotion (the historical event) may have contributed to the

increase in cigarette consumption.

VALIDITY THREATS

Selection. In an experimental study, a threat to validity

occurs when the elements or subjects for the

experimental group is very difficult from those

selected for the control group. For instance, if at the

beginning the experiment, the experimental group

already has an advantage over the control group in

terms of the focus of the study, this difference will

definitely affect the results of the study.

VALIDITY THREATS

For example, in an experiment conducted to

determine if using games and puzzles as instructional

aids can improve performances of college freshmen

in Basic Math, the teacher used games and puzzles

in the experimental group, but did not use them in

he control group. After the experiment, it was found

that the experimental group got significantly better

grades in the subject than the control group.

VALIDITY THREATS

It was discovered, however, that most of the students

in the experimental group had very good grades in

high school math, while most of those in the control

group had average grades only. Attributing the better

performance of the experimental group to use the

games ad puzzles can be questioned. To avoid this

validity threat, the experimental and the control group

should have similar characteristics at the beginning.

VALIDITY THREATS

Testing. Whenever a pretest is given, it may take the

examinees “test wise,” and this can therefore affect the post-

test results. Research subjects who have been given a pretest

may remember some of the test items/questions for which they

may search answers and get these correct when they take

the post-test. Better performance in the post-test might be due

to the effect of the pretest and not necessarily to the

intervention or treatment.

VALIDITY THREATS

Instrumentation. When a research interment, such

as questionnaire or a measuring device, like

weighing scale or a thermometer is changed during

the study period or between the pretest and

the post-test, the change could result in an effect

that is independent of the intervention and yet, may

be attributed to it.

For example, in a survey study, an

instrumentation effect may be caused by an

interviewer who after conducting the pre-test

interview becomes more experienced in

interviewing. The interview/s experience will

enable him/her to generate better and/or more

complete information during the post test than

what was collected during pretest.

In a biomedical study, the use of an

unreliable device, like a scale

that badly needs calibration, a

contaminated syringe, or a very old

litmus paper may also threaten the

validity of test results.

Maturation. People and things change over

time. In other words they become more

mature, and this change can threaten the

validity of conclusions. Research subjects can

get tired, hungry, or bored during the duration

of the project. If the effect of a project is

measured with a test, their tiredness or

boredom can result in scores lower than

their “true” scores..

On the other hand, the subjects

may become more experienced,

more knowledgeable as they grow

older and as a result they may get

higher scores than than they did in

the pretest. In this regard the

changes can not be attributed to the

intervention.

Mortality. In studies that take a long time to finish,

say, one year or more, like cohort studies, where

the subjects (the same people) are followed up

over time, some cases may drop out, thus

resulting in a loss of cases. Some cases may have

transferred residence and are difficult to locate

during the follow-up interview. Cases which cannot

be contacted cannot be followed-up. This loss,

called mortality, may distort findings and

conclusions, if substantial and if it has introduced a

bias to the sample.

The loss could result in a big

difference between he pre-test and

the post-test results. This chafe may

be wrongly attributed to the

intervention, thus, threaten the

validity of the conclusions.

Commonly Used Research Designs (Campbell and

Stanley, 1968, Parel, et al., 1985, Fisher, et al.,

1994)

The choice of a research design depends on the

objectives of the study. There are many types of

research designs that can be used in basic and

experimental research. Described here are doe of

the most frequently used designs. They are

classified into: non- or pre-experimental designs,

true experimental designs, and quasi-experimental

designs.

Non/Pre-Experimental

Designs

Non-experimental designs are appropriate for collecting

descriptive information about a population or subjects of a

study. They are appropriate for descriptive studies, like

profile studies, exploratory studies, and for doing small case

studies. They are also ideal for diagnostic studies intended to

determine the effect or impact of a certain intervention or

treatment. Three non-experimental designs are described

below. They are the post-test only or after only design, the

pretest-posttest design, and the static group comparison.

Posttest Only Design or After-

Only Survey

Time

X O

(Observation/Testing/Survey)

The design is also called as one shot

survey because the data are collected

only once (O). This design is used when

the study objective is to describe a

situation/condition of a study population

as it exists, or to determine/describe the

characteristics of a

population/respondents. There is no

baseline data.

This design is cheap and easy to

conduct, but results canon the

conclusive in terms of causality or

effect of an intervention. It is not,

however, recommended for evaluation

studies that intend to measure the

effect of a program intervention, like

training.

Pretest-Posttest Design or

Before-After Survey

Time (Intervention)

Observation/Survey 1

(Before X)Observation/Survey 2

(After X)

O1 O2

This design is used when the study wants to know

the change in characteristics (e.g. knowledge,

attitude, practices) of the study population

(students, nurses, managers, clients, etc) in a

given area. A survey observation, or testing is

conducted before an intervention is introduced (O1)

. After a period of time the survey, observation or

testing is repeated (O2) and the results of pretest

(before) and the post-test (after) are compared to

determine change/s.

For example, if a researcher wants to know if an

information campaign against drug/substance

abuse in a certain city has reduced drug use in the

area after the campaign, a survey before and after

the campaign can be conducted. No “control” area

(area where no campaign is conducted), however,

is surveyed. With the absence of a control area,

this design cannot be considered an experimental

design. Any reduction in drug use overtime, cannot

be solely/conclusively attributed to the intervention

(campaign).

Static Group Comparison

Time

XO1

O2Experimental Group

Control GroupIn the static group design, there are two groups involved, an experimental

group and a control group. The experimental group receives or is exposed

to the intervention/treatment (X). This is followed by a measurement (O1),

the result of which is compared to the result of the

measurement/observation from a control group (O2) that did not

receive the intervention. The random process, however, was not used in

the assignment of subjects to the experimental and control groups

(indicated by a broken line). The problem with this design is the validity

threat of selection and mortality. It is possible that the two groups differ

greatly on the basis of the main variables of the study (selection) or some

subjects in the experimental group may drop out and be lost to follow-up

or second observation/testing (mortality).

True Experimental Designs

In true experimental designs, subjects are randomly assigned

to the experimental group and the control group to achieve

pre-intervention/treatment equality of the two groups, validity

threats are avoided. Before a researcher decides on an

alternative design, the feasibility of using true experimental

designs must first be considered. The two most frequently

used true experimental designs are the pretest-posttest

control group design and the post-test control group design.

Pretest-posttest Control

Group Design

RA

Experimental group

Control group

O1

O2

O3

O4

Pretest Posttest

X

In the pretest-postest control group design, the experimental

group is exposed to or covered by an intervention or

treatment (X), for example, training or a new strategy, while

the control group is left alone or given another kind of

treatment. Before the intervention/treatment is introduced to

the experimental group (O1) and control group (O2) using the

same device/instrument. The pre-intervention

survey/observation/test serves as pretest and the data

collected serve as baseline data. After the introduction of the

intervention in the experimental group or area, an evaluation

survey/observation/testing is conducted in both experimental

group/area (O3) and the control group/area (O4), using the

same instrument used in both during the pretest. The results

serve as the post-test/endline data.

The baseline (pretest) and end line (post-

test) data are compared. If the change in

the “impact/effect indicator/s” or dependent

variable/s is significantly better in the

experimental area/group than the change in

the control area/group, then the intervention

is considered effective. If not, then the

intervention is said to have had no effect.

Posttest Only Control Group

Design

RA

Experimental group

Control group

X O1

O2

Posttest

The Posttest Only Control Group design is

also used to determine the effects of an

intervention or treatment introduced to a

group of subjects (people/objects). As in

the pretest-posttest control group design,

at least two groups or areas (e.g. women

groups, communities, provinces) with

virtually the same characteristics are

chosen and randomly assigned (RA) to the

control and experimental group.

The experimental group or area is exposed to or

covered by an intervention/treatment, while the

control group is left alone. No pretest/pre-

intervention study is conducted. The experimental

and the control groups are assumed to have

similar characteristics at the start of the

study. After the introduction of an intervention in

the experiment group or area, an evaluation

survey/observation/testing is conducted in both

experimental and the control groups or areas,

using the same “fair” instrument.

The data gathered from the experimental

and control groups are compared. If the

experimental group or area

shows significantly better results than the

control area/group with respect to

the “impact/effect indicator/s” or

dependent variable/s, the intervention or

treatment is considered effective. If not,

then, the intervention is not effective.

Quasi-experimental Designs

In filed studies, it is very difficult to meet the

random assignment criterion of a true experimental

design. In this situation, a quasi-experimental design is

recommended. Quasi-experimental designs are nearly

the same as the true experimental designs, except that

the former do not have restrictions of random

assignment. The two most commonly used quasi-

experimental designs are the non-equivalent control

group design and the time series design.

Non-equivalent Control

Group Design

Experimental group

Control group

O1

O2 O4

Posttest

X

O3

Pretest

Time

In field research, it is possible to compare an

experimental group with a similar, but no necessarily

equivalent group. The two groups need only to

have “collective similarity,” which means that they

should have more or less the same characteristics in

terms of aspects which are relevant to the study. For

example, if one wants to determine the impact of an

educational campaign on school attendance of

children, the experimental and the control areas

should have more or less the same socioeconomic

characteristics, because these factors may also

affect school attendance.

As in the pretest-posttest control group design,

the intervention or treatment is introduced to the

experimental group, but withheld from the

control group. Before the introduction of the

intervention, a survey/observation/testing is

conducted in both the experimental

group (O1) and the control group (O2). After the

introduction of the intervention to the

experimental group, another observation/testing

(post-test) is conducted to both

groups (O3 and O4).

The pretest can be used to determine whether the two

groups have truly “collective similarity” at the start of

the experiment. The results of the two post-

tests (O3 and O4) will also be compared. The

intervention is effective if the change in the

impact/effect indicators in the experimental

group (O3 minus O1) is significantly higher/better than

the change in the impact/effect indicators in the control

group (O4 minus O3). If not, then the

intervention/treatment cannot be considered effective.

This design is a good one for evaluating training

programs, and other community interventions.

Time Series Design

Time

XO1 O2 O3 O4 O5 O6

The time series design is similar to the non-experimental

pretest-posttest design except that, it has repeated

observations/measurements before and after the

intervention (X). Before the introduction of the

intervention/treatment, a measurement/observation with

respect to the impact/effect indicators will conducted

several times at a regular interval, say, every 30

days; (O1 , O2, O3), and then after the intervention, another

series of measurement/observation will be

conducted (O4 , O5, O6), also at the same time interval as

the first. The same measuring instrument/device should be

used at all times.

The result or pattern of the observations or testing

in the first series of measurements will be

compared with that in the series of measurements

after the intervention. If the post-intervention result

or pattern is better than that of the pre-intervention

series, then the intervention can be considered

effective. However, if the pre-intervention and post

intervention results or patterns are the same, or the

post intervention result is not significantly better

than that of the pre-intervention. then the

intervention cannot be considered effective.

For example, one wants to evaluate the effect of a

feeding program which is intended to improve the

nutritional status of pre-school children in a

barangay. Before the introduction of the feeding

program, the children (program beneficiaries) will

be weighed (measured) several times at regular

interval. say, every 30 days; (O1 , O2, O3) and then

after the feeding program another series of

weighing (measurement) will be

conducted (O4 , O5, O6) also at the same time

interval as the first series (every 30 days).

In order for the feeding program to be considered

effective in improving the nutritional status of the

children, the children’s weights should improve

after the feeding program. Since the children are

also growing, increase in weight may also be

observed during the series of pre-intervention

changes must be significantly better than the pre-

intervention changes. If norm the feeding program

could not be considered as having effectively

improved the nutritional status of the children.