the basic steps of research

The basic steps of research...The basic steps of research...

Scientific and disciplined inquiry is an orderly process, involving:

• recognition and identification of a topic to be studied (“problemproblem”)

• description and execution of procedures to collection information (“methodmethod”)

• objective data analysisanalysis

• statement of findings (“resultsresults”)

Research methods...Research methods...

QuantitativeQuantitative……collects and analyzes numerical datadata

obtained from formal instrumentsinstruments

Quantitative methods...Quantitative methods...

descriptive research (“survey research”)descriptive research (“survey research”) correlational researchcorrelational research experimental researchexperimental research

experimental researchexperimental research

……the researcher selects participants and the researcher selects participants and divides them into two or more groups having divides them into two or more groups having similar characteristics and, then, applies the similar characteristics and, then, applies the treatment(s) to the groups and measures the treatment(s) to the groups and measures the effects upon the groupseffects upon the groups

Conducting an experimental study…Conducting an experimental study…

select the problemselect the problem

select participants and instrumentselect participants and instrument

selection and execution of a research planselection and execution of a research plan

data analysis and formulation of conclusionsdata analysis and formulation of conclusions

Types of experimental comparison…Types of experimental comparison…

1.1. comparison of two different approaches ( comparison of two different approaches (A A versus versus BB))

2.2. comparison of an existing approach to a new comparison of an existing approach to a new approach (approach (AA and and ~~ A A))

3.3. comparison of differing amounts of a single comparison of differing amounts of a single approach (approach (AA and and a a or or aa and and AA))

where:where:

AA – experimental (“treatment”) group – experimental (“treatment”) group

BB – control (“no treatment,” “nonmanipulated”) – control (“no treatment,” “nonmanipulated”) groupgroup

Variable...Variable...

…a concept (e.g., intelligence, height, aptitude) that can assume any one of a range of values

Research variables...Research variables...

IndependentIndependent……an activity of characteristic believed to

make a difference with respect to some behavior

…(syn.) experimental variable, active variable, cause, treatment

DependentDependent…

…the change or difference occurring as a result of the independent variable

…(syn.) criterion variable, assigned variable, effect, outcome, posttest

ConfoundingConfounding…

…the fact that the effects of the independent variable may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable

…(syn.) criterion variable, assigned variable, effect, outcome, posttest

Validity of Experimental DesignValidity of Experimental Design

Internal validity—The accuracy in concluding Internal validity—The accuracy in concluding that the outcome of an experiment is due to the that the outcome of an experiment is due to the independent variable independent variable

External validity—The extent to which the External validity—The extent to which the results of an experiment can be generalizedresults of an experiment can be generalized

INTERNAL OR EXTERNAL INTERNAL OR EXTERNAL VALIDITY: A TRADE-OFF?VALIDITY: A TRADE-OFF?

Too much control reduces ability to generalizeToo much control reduces ability to generalize Too little control reduces ability to make Too little control reduces ability to make

causal statementscausal statements Attempt to find a good balanceAttempt to find a good balance

InternalValidity

ExternalValidity

Factors Jeopardizing Internal ValidityFactors Jeopardizing Internal Validity

HistoryHistory - The events occurring between the - The events occurring between the first and second measurements in addition to first and second measurements in addition to the experimental variable which might affect the experimental variable which might affect the measurement.the measurement.

ExampleExample: Researcher collects gross : Researcher collects gross sales data before and after a 5 day 50% off sales data before and after a 5 day 50% off sale. During the sale a hurricane occurs and sale. During the sale a hurricane occurs and results of the study may be affected because of results of the study may be affected because of the hurricane, not the sale.the hurricane, not the sale.

MaturationMaturation - The process of maturing which - The process of maturing which takes place in the individual during the takes place in the individual during the duration of the experiment which is not a duration of the experiment which is not a result of specific events but of simply growing result of specific events but of simply growing older, growing more tired, or similar changes.older, growing more tired, or similar changes.

ExampleExample: Subjects become tired after : Subjects become tired after completing a training session, and their completing a training session, and their responses on the Posttest are affected.responses on the Posttest are affected.

Pre-testingPre-testing - The effect created on the second - The effect created on the second measurement by having a measurement before measurement by having a measurement before the experiment.the experiment.

ExampleExample: Subjects take a Pretest and think : Subjects take a Pretest and think about some of the items. On the Posttest they about some of the items. On the Posttest they change to answers they feel are more change to answers they feel are more acceptable. Experimental group learns from acceptable. Experimental group learns from the pretest.the pretest.

Measuring InstrumentsMeasuring Instruments - Changes in - Changes in instruments, calibration of instruments, instruments, calibration of instruments, observers, or scorers may cause changes in observers, or scorers may cause changes in the measurements.the measurements.

ExampleExample: Interviewers are very : Interviewers are very careful with their first two or three interviews careful with their first two or three interviews but on the 4th, 5th, 6th become fatigued and but on the 4th, 5th, 6th become fatigued and are less careful and make errors.are less careful and make errors.

Statistical RegressionStatistical Regression - Groups are chosen because - Groups are chosen because of extreme scores of measurements; those scores or of extreme scores of measurements; those scores or measurements tend to move toward the mean with measurements tend to move toward the mean with repeated measurements even without an experimental repeated measurements even without an experimental variable.variable.

ExampleExample: Managers who are performing poorly are : Managers who are performing poorly are selected for training. Their average Posttest scores selected for training. Their average Posttest scores will be higher than their Pretest scores because of will be higher than their Pretest scores because of statistical regression, even if no training were given.statistical regression, even if no training were given.

Differential SelectionDifferential Selection - Different individuals or - Different individuals or groups would have different previous knowledge or groups would have different previous knowledge or ability which would affect the final measurement if ability which would affect the final measurement if not taken into account.not taken into account.

ExampleExample: A group of subjects who have viewed a TV : A group of subjects who have viewed a TV program is compared with a group which has not. program is compared with a group which has not. There is no way of knowing that the groups would There is no way of knowing that the groups would have been equivalent since they were not randomly have been equivalent since they were not randomly assigned to view the TV program.assigned to view the TV program.

Experimental MortalityExperimental Mortality - The loss of subjects from - The loss of subjects from comparison groups could greatly affect the comparison groups could greatly affect the comparisons because of unique characteristics of comparisons because of unique characteristics of those subjects. Groups to be compared need to be the those subjects. Groups to be compared need to be the same after as before the experiment.same after as before the experiment.

ExampleExample: Over a 6 month experiment : Over a 6 month experiment aimed to change accounting practices, 12 accountants aimed to change accounting practices, 12 accountants drop out of the experimental group and none drop out drop out of the experimental group and none drop out of the control group. Not only is there differential of the control group. Not only is there differential loss in the two groups, but the 12 dropouts may be loss in the two groups, but the 12 dropouts may be very different from those who remained in the very different from those who remained in the experimental group.experimental group.

Interaction of FactorsInteraction of Factors, such as Selection , such as Selection Maturation, etc. - Combinations of these Maturation, etc. - Combinations of these factors may interact especially in multiple factors may interact especially in multiple group comparisons to produce erroneous group comparisons to produce erroneous measurements. measurements.

Factors Jeopardizing External Validity Factors Jeopardizing External Validity or Generalizabilityor Generalizability

Pre-TestingPre-Testing -Individuals who were pretested might -Individuals who were pretested might be less or more sensitive to the experimental variable be less or more sensitive to the experimental variable or might have "learned" from the pre-test making or might have "learned" from the pre-test making them unrepresentative of the population who had not them unrepresentative of the population who had not been pre-tested.been pre-tested.

ExampleExample: Prior to viewing a film on : Prior to viewing a film on Environmental Effects of Chemical, a group of Environmental Effects of Chemical, a group of subjects is given a 60 item antichemical test. Taking subjects is given a 60 item antichemical test. Taking the Pretest may increase the effect of the film. The the Pretest may increase the effect of the film. The film may not be effective for a nonpretested group.film may not be effective for a nonpretested group.

Differential SelectionDifferential Selection - The selection of the subjects - The selection of the subjects determines how the findings can be generalized. Subjects determines how the findings can be generalized. Subjects selected from a small group or one with particular selected from a small group or one with particular characteristics would limit generalizability. Randomly chosen characteristics would limit generalizability. Randomly chosen subjects from the entire population could be generalized to the subjects from the entire population could be generalized to the entire population. entire population.

ExampleExample: Researcher, requesting permission to : Researcher, requesting permission to conduct experiment, is turned down by 11 corporations, but conduct experiment, is turned down by 11 corporations, but the 12th corporation grant permission. The 12th corporation is the 12th corporation grant permission. The 12th corporation is obviously different then the others because they accepted. obviously different then the others because they accepted. Thus subjects in the 12th corporation may be more accepting Thus subjects in the 12th corporation may be more accepting or sensitive to the treatment.or sensitive to the treatment.

Experimental ProceduresExperimental Procedures - The experimental - The experimental procedures and arrangements have a certain amount procedures and arrangements have a certain amount of effect on the subjects in the experimental settings. of effect on the subjects in the experimental settings. Generalization to persons not in the experimental Generalization to persons not in the experimental setting may be precluded.setting may be precluded.

ExampleExample: Department heads realize they : Department heads realize they are being studied, try to guess what the experimenter are being studied, try to guess what the experimenter wants and respond accordingly rather than respond to wants and respond accordingly rather than respond to the treatment.the treatment.

Multiple Treatment InterferenceMultiple Treatment Interference - If the subjects - If the subjects are exposed to more than one treatment then the are exposed to more than one treatment then the findings could only be generalized to individuals findings could only be generalized to individuals exposed to the same treatments in the same order of exposed to the same treatments in the same order of presentation.presentation.

ExampleExample: A group of CPA’s is given : A group of CPA’s is given training in working with managers followed by training in working with managers followed by training in working with comptrollers. Since training training in working with comptrollers. Since training effects cannot be deleted, the first training will affect effects cannot be deleted, the first training will affect the second.the second.

Tools of Experimental Design Used to Control Tools of Experimental Design Used to Control Factors Jeopardizing ValidityFactors Jeopardizing Validity

Pre-TestPre-Test - The pre-test, or measurement before the - The pre-test, or measurement before the experiment begins, can aid control for differential experiment begins, can aid control for differential selection by determining the presence or knowledge selection by determining the presence or knowledge of the experimental variable before the experiment of the experimental variable before the experiment begins. It can aid control of experimental mortality begins. It can aid control of experimental mortality because the subjects can be removed from the entire because the subjects can be removed from the entire comparison by removing their pre-tests. comparison by removing their pre-tests.

However, pre-tests cause problems by their effect on However, pre-tests cause problems by their effect on the second measurement and by causing the second measurement and by causing generalizability problems to a population not pre-generalizability problems to a population not pre-tested and those with no experimental arrangements. tested and those with no experimental arrangements.

Control GroupControl Group -The use of a matched or -The use of a matched or similar group which is not exposed to the similar group which is not exposed to the experimental variable can help reduce the experimental variable can help reduce the effect of History, Maturation, Instrumentation, effect of History, Maturation, Instrumentation, and Interaction of Factors. The control group and Interaction of Factors. The control group is exposed to all conditions of the experiment is exposed to all conditions of the experiment except the experimental variable. except the experimental variable.

RandomizationRandomization - Use of random selection - Use of random selection procedures for subjects can aid in control of procedures for subjects can aid in control of Statistical Regression, Differential Selection, Statistical Regression, Differential Selection, and the Interaction of Factors. It greatly and the Interaction of Factors. It greatly increases generalizability by helping make the increases generalizability by helping make the groups representative of the populations groups representative of the populations

Additional GroupsAdditional Groups - The effects of Pre-tests - The effects of Pre-tests and Experimental Procedures can be partially and Experimental Procedures can be partially controlled through the use of groups which controlled through the use of groups which were not pre-tested or exposed to experimental were not pre-tested or exposed to experimental arrangements. They would have to be used in arrangements. They would have to be used in conjunction with other pre-tested groups or conjunction with other pre-tested groups or other factors jeopardizing validity would be other factors jeopardizing validity would be present. present.

Group experimental designs…Group experimental designs…

1.1. single-variable single-variable

2.2. factorial factorial

single-variable designssingle-variable designs

……involve one manipulated independent variableinvolve one manipulated independent variable pre-experimentalpre-experimental quasi-experimentalquasi-experimental true experimentaltrue experimental

Experimental DesignsExperimental Designs

Pre-Experimental DesignPre-Experimental Design - loose in structure, - loose in structure, could be biasedcould be biased

True Experimental DesignTrue Experimental Design - greater control - greater control and refinement, greater control of validityand refinement, greater control of validity

Quasi-Experimental DesignQuasi-Experimental Design - not randomly - not randomly selectedselected

Pre-Experimental DesignPre-Experimental DesignAim of the Aim of the Research Research

Name of the Name of the Design Design

Notation Notation Paradigm Paradigm

Comments Comments

To attempt to To attempt to explain a consequent explain a consequent by an antecedent by an antecedent

One-shot One-shot experimental case experimental case study study

X » O X » O An approach that An approach that prematurely links prematurely links antecedents and antecedents and consequences. The least consequences. The least reliable of all reliable of all experimental experimental approaches. approaches.

To evaluate the To evaluate the influence of a influence of a variable variable

One group pretest-One group pretest-posttest posttest

O » X » O O » X » O An approach that An approach that provides a measure of provides a measure of change but can provide change but can provide no conclusive results. no conclusive results.

To determine the To determine the influence of a influence of a variable on one variable on one group and not on group and not on another another

Static group Static group comparison comparison

Group 1: X » O Group 1: X » O

Group 2: - » O Group 2: - » O

Weakness lies in no Weakness lies in no examination of pre-examination of pre-experimental experimental equivalence of groups. equivalence of groups. Conclusion is reached Conclusion is reached by comparing the by comparing the performance of each performance of each group to determine the group to determine the effect of a variable on effect of a variable on one of them. one of them.

True Experimental DesignTrue Experimental Design Aim of the Aim of the Research Research



Comments Comments

To study the effect of To study the effect of an influence on a an influence on a carefully controlled carefully controlled sample sample

Pretest-posttest control Pretest-posttest control group group

R - - [ O » X » O R - - [ O » X » O

[ O » - » O [ O » - » O

This design has been This design has been called "the old called "the old workhorse of workhorse of traditional traditional experimentation." If experimentation." If effectively carried out, effectively carried out, this design controls for this design controls for eight threats of eight threats of internal validity. Data internal validity. Data are analyzed by are analyzed by analysis of covariance.analysis of covariance.

Experimental design (cont)Experimental design (cont)

Aim of the Aim of the Research Research



Comments Comments

To minimize the effect To minimize the effect of pretesting of pretesting

Solomon four-group Solomon four-group design design

R - - [ O » X » O R - - [ O » X » O

[ O » - » O[ O » - » O

[- » X » O[- » X » O

[ - » - » O [ - » - » O

This is an extension of This is an extension of the pretest-posttest the pretest-posttest control group design control group design and probably the most and probably the most powerful experimental powerful experimental approach. Data are approach. Data are analyzed by analysis analyzed by analysis of variance on posttest of variance on posttest scores scores

Experimental design (cont) Experimental design (cont)




Comments Comments

To evaluate a situation To evaluate a situation that cannot be that cannot be pretested pretested

Posttest only control Posttest only control group group

R - - [ X » O R - - [ X » O

[ - » O [ - » O

An adaptation of the An adaptation of the last two groups in the last two groups in the Solomon four-group Solomon four-group design. Randomness is design. Randomness is critical. Probably, the critical. Probably, the simplest and best test simplest and best test for significance in this for significance in this design is the t-test. design is the t-test.

Quasi-Experimental DesignQuasi-Experimental Design




Comments Comments

To investigate a To investigate a situation in which situation in which random selection and random selection and assignment are not assignment are not possible possible

Nonrandomized Nonrandomized control group pretest-control group pretest-posttest posttest

O » X » O O » X » O

O » - » O O » - » O

One of the strongest One of the strongest and most widely used and most widely used quasi-experimental quasi-experimental designs. Differs from designs. Differs from experimental designs experimental designs because test and because test and control groups are not control groups are not equivalent. Comparing equivalent. Comparing pretest results will pretest results will indicate degree of indicate degree of equivalency between equivalency between experimental and experimental and control groups. control groups.

Quasi-Experimental DesignQuasi-Experimental Design (cont) (cont)

Aim of the Aim of the ResearchResearch

Name of the Name of the DesignDesign

Notation Notation ParadigmParadigm

CommentsComments

To determine the To determine the influence of a variable influence of a variable introduced only after a introduced only after a series of initial series of initial observations and only observations and only where one group is where one group is available available

Time series Time series experiment experiment

O » O » X » O » O O » O » X » O » O If substantial change If substantial change follows introduction of follows introduction of the variable, then the the variable, then the variable can be suspect variable can be suspect as to the cause of the as to the cause of the change. To increase change. To increase external validity, external validity, repeat the experiment repeat the experiment in different places in different places under different under different conditions. conditions.





CommentsComments

To bolster the validity To bolster the validity of the above design of the above design with the addition of a with the addition of a control group control group

Control group time Control group time series series

O » O » X » O » O O » O » X » O » O

O » O » - » O » O O » O » - » O » O

A variant of the above A variant of the above design by design by accompanying it with accompanying it with a parallel set of a parallel set of observations without observations without the introduction of the the introduction of the experimental variable. experimental variable.





CommentsComments

To control history in To control history in time designs with a time designs with a variant of the above variant of the above design design

Equivalent time-Equivalent time-samples samples

[X1 » O1] »[X0 » O2] [X1 » O1] »[X0 » O2] » [x1 » O3] » [x1 » O3]

An on-again, off-again An on-again, off-again design in which the design in which the experimental variable experimental variable is sometimes present, is sometimes present, sometimes absent. sometimes absent.

the basic steps of research

Documents

experimental variable

criterion variable

active variable

experimental comparison1

experimental treatment

independent variablesyn

numerical data

new approach