Research Design

Week 6 Part 1

8-9 February 2011

PPAL 6200

Outline

• Observation vs. Experiment

• Subjects, Factors, Treatments

• Good and Bad Experiments

• Randomized Comparative Experiments– Their Logic

• Cautions about Experiments

• Matched Pairs and other Block Designs

Observation vs. Experiments

• An Experiment is a special kind of research study that imposes or applies a treatment in order to observe responses so as to determine if the treatment causes a change in response

• An observational study observes variation in individual variables of interest but does not attempt to manipulate them in order to influence or produce responses

• In many cases social scientists are interested in, it is probably not ethical to conduct genuine experiments.

Subjects, Factors, Treatments

• The book describes a “study” as “an experiment when we actually do something to people, animals,… in order to observe the response.”

• Not sure. Another view is that a study is any systematic investigation.

• Some key words we have to remember– Subject– Explanatory Variables or Factors, or

Independent Variables– Response Variables, or Dependent Variables– Treatment (can be a combination of Factors)

Experiments Good and Bad

• The benefit of experiments is that they allow for “control”, we try to manipulate one or some variables (the treatment) so as to produce change in another variable (the response) while holding everything else constant

• If all the variables that can impact the response are not controlled, the results can be compromised.

Three types of Control

• Physical control

• Statistical control

• Logical

The Double Blind Randomized Comparative Experiment

• This is the “gold standard” in research design often used in medical and pharmaceutical research.

• You are interested if treatment “x” can cure a disease.• You draw a random sample of the population of those

suffering the disease and someone separate from the researchers randomly assigns individuals to either the experimental or control group.

• You then administer the treatment to the experimental group but provide either a placebo, or more likely, an already approved treatment to the control group so that they also don’t know if they are receiving the treatment or not.

• Neither researchers nor subjects know who is in the treatment group and who is in the control group

• You then compare the results for the two groups

The Logic of a Randomized Comparative Experiment

• Randomized selection and assignment to treatment and control groups mean all variations among the cases ought to be equally represented in each group

• Therefore, any differences in average response must be due either to the treatment or (as occasionally discovered pure random chance).

• However, if you use enough subjects random chance ought not to play a role.

• As we noted before, there is always a chance that results are the product of pure chance.

• A result that is so strong that it is unlikely to be the product of pure random chance is said to be statistically significant.

Matched pairs and other block designs

• Sometimes the gold standard is not possible but other experimental designs are.

• Essentially this is like a randomized comparative experiment except that the assignment of individuals is done one pair at a time so randomization is not as thorough.

Some things not in the book

• In order to develop a social science project we need to look at some of the steps involved in moving from an idea to a completed project.

The research development processStage 1

• A Problem or Observation

• A Question

• Background Research– Generally a literature of previous research on

topics related to your question.

Stage 2

• A Hypothesis– A statement that something (the explanatory

or independent variable[s]) causes or influences variation in something else (the dependent or response variable), and possibly how this is done.

• To do this you must have a conceptual definition of your variables

Stage 3

• Operationalizing your variables – How are you going to measure your variables– How are you going to collect those

measurements

How you go about collecting data depends on the problem and question you began with

• In many cases the literature review you have conducted will reveal not only what people have found on topics related to the question you have, but also how people have gone about studying these topics as well.

Common Research Designs

• Small “n” vs. Large “n” designs• Small “n”

– Small “n” comparative• Mill’s Methods of Agreement and Disagreement

– The Case Study– Small “n” elite

• Position of individuals

– Participant Observer• Researcher’s own senses

– Key point, in all of these control is logical

• Large “n” Designs– Survey Research (Opinions)– Econometric (National Accounts and Policies)– Demographic.– Quantitative Content Analysis– All of these can involve either original data

collection or the secondary analysis of previously collected data.

– Key point, in all of these control is exercised statistically.

If you do collect original data…

• Two things you should be concerned about• The validity and reliability of your

operationalizations– Pilot Studies

• The comparability of your work to that of others• Make life simpler for yourself, look for questions

that have been previously used by other researchers which have been field tested and validated, and to which comparisons of findings can be made.

Stage 4

• Executing your design

Stage 5

• Analyzing your data

Stage 6

• Drawing your conclusions

Stage 7

• Writing your report

A funny thing called ethics

• As public servants are required to work in the public interest, this means respecting whatever codes of research ethics are in place in your work place

• The gold standards of ethics – informed consent

The Tri-Council Policy

• http://www.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/

• Sets out the basic policies that should govern research on human subjects at all institutions that receive funding from the Federal government, such as York University.

• Sets out a general standard of ethical conduct built around “informed consent” of all subjects who participate in research.