1 designing experiments suppose we wanted to design an experiment to see if caffeine affects pulse...

1

Designing Experiments

Suppose we wanted to design an experiment to see if

caffeine affects pulse rate (in this class).

What is the explanatory variable (factor)?

caffeine

What is the response variable?

pulse rate

What will be the experimental units?

the students in this class

2


Here is an initial plan: measure initial pulse rate give each student some caffeine wait for a specified time measure final pulse rate compare final and initial rates

3


What are some problems with this plan?

If there is a change in pulse rates, we do not know if caffeine was the cause. For example, suppose I told a joke while we were waiting, and every one laughed so hard that their pulse rates went up. Or, suppose we took notes and everybody’s pulse rate slowed down to sleeplike levels.

4


Some problems can be easily solved by including a ___________ which does not receive caffeine. In our experiment, we can accomplish this by using 2 ________ of caffeine: no caffeine and some caffeine. For example, we could assign each member one of two ___________: Regular Coke or Caffeine Free Coke.

5


Why don’t we give Coke to one group and nothing to the other group?

Often times applying any treatment can create a change in the response variable. For example, when a child gets hurt, they feel better when their wound is kissed or covered with a band-aid, even though neither of those treatments actually take away the pain.

6


In our study, if only one group got a treatment, the fact that they were chosen to receive free soda might make their pulse increase before the caffeine even hits their bloodstream!

The ________________ occurs when subjects in an experiment know they are receiving a treatment. This knowledge may cause a change in the response variable which ___________ the effect of the treatment. In other words, we will not know which caused the change in the response variable: the explanatory variable or the placebo effect.

7


Having every subject receive a treatment ensures that the placebo effect will treat both groups the same. Then, any difference between their pulse rates can be attributed to the _______________________ and not the excitement of being in an experiment.

Of course, it is essential that the subjects do not know which treatment they are receiving! When a person doesn’t know who is receiving which treatment, that person is ________.

8


There are two classes of individuals who can influence the results of an experiment: those who directly participate in the experiment

(subjects, treatment administrators, etc.)

those who evaluate the results

When every individual in one of these classes is blinded, the experiment is called ___________. If every individual in both classes is blinded, then the experiment is ________________.

9


Can our experiment be run in a D________ B_______ manner?

Yes, if the subjects, the people handing out the treatments, and the people measuring the pulse rates all do not know which treatment is which.

But doesn’t someone need to know which is which? Yes, but that person should not have any interaction with the subjects at any point of the experiment.

10


Four Key Principles of a Good Experiment:

THE BIG IDEA: Our goal when designing an experiment is to make the treatment groups as similar as possible, with the exception of the treatments. Then, if there is a change in the response, it can be attributed to the explanatory variable and not any other extraneous variables.

11


An _______________ is one that is not of interest in the current study but is thought to affect the response variable.

For example, sugar is an extraneous variable since it may affect pulse rates. If one treatment group was given regular Coke and the other treatment group was given caffeine free Diet Coke, then sugar and caffeine would be confounded. If there was a difference in the average pulse rates of the two groups after receiving the treatments, we wouldn’t know which variable caused the change, and to what extent.

12


To prevent sugar from becoming a __________ variable, we need to make sure that both treatment groups get the same amount of sugar.

13


Principle #1: _________________ means holding extraneous variables constant for all treatment groups so that their effects are not confounded with the explanatory variable. temperature of drink amount of drink drinking rate amount of sugar (no diet) waiting time between treatment and pulse reading pulse rate before experiment (need to be resting)

room temperature carbonation how we measure pulse

14


If we do not control these extraneous variables by making them the same for all treatment groups, they could confound the effects of the caffeine on pulse rates.

For example, we may not be able to tell if it was the caffeine or the temperature that causes the higher pulse rate.

15


Principle #2: _________ is when subjects are divided into groups (blocks) based on some extraneous variable and then separated into different treatment groups.

What if men react to caffeine differently than women?

If more men end up in the experimental group and more women end up in the control group, then gender and caffeine will be confounded. We will not know which variable caused the change in pulse rates, gender or caffeine.

16


How can we eliminate this potential confounding variable? Eliminate one gender from the study, but then we

could only draw conclusions about one gender. Make sure there is a representative number of men

and women in each treatment. For example, if there are 20 women and 30 men in the experiment, then the experimental group should have10 women and 15 men and the control group should have the same.In this example, we have formed 2 blocks: men and women. Then, we assigned treatments to the subjects within each block.

17


Blocking in experiments is similar to stratification in sampling.

Blocking reduces the variability of the results, just like stratifying.

Blocks should be chosen like strata: the units within the block should be similar, but different than the units in the other blocks. You should only block when you expect that the subjects in one block will have a different response than subjects in other blocks.

18


What are some other extraneous factors that we can block for?

age, weight, initial pulse rate, etc.

You should try to make the blocks as small as possible. Ideally, the size of the block should be the same as the number of treatments. For example, if there are 3 treatments, then there should be 3 subjects in each block. If each block has only 2 subjects, then the subjects are called a ______________.

19


Principle #3: _______________ is random assignment of subjects to treatments to ensure that the experiment doesn’t systematically favor one treatment over the other.

What about all of the other extraneous variables we do not think of? What about the variables we cannot directly control or block for? amount of food eaten before experiment caffeine tolerance

20


If we randomly assign subjects to treatments, this should _______________ (but not eliminate) the effects of these variables since their effects should be spread equally between the treatment groups.

Note: We must ALWAYS randomize since there will always be extraneous variables we do not consider.

21


How do we randomize?

Draw names from a hat. The first half chosen are in one group, the remaining names in the other.

Number the class from 01-32. Then, generate random numbers without replacement until half are chosen for one group. The remaining names go in the other group.

For matched pairs, we can flip a coin to determine which subjects go into which group. If its heads, the first person in the pair goes to A and the other to B. If its tails, it’s the opposite.

22


If you do not use blocking when dividing the subjects, the result is a _______________________________.

If you incorporate blocking in your design, it is called a _________________________ (every subject is assigned to a block based on some characteristics and the members of the block are randomly assigned to the different treatments).

23


Principle #4: ____________ means ensuring that there is an adequate number of observations in each treatment group.

If each treatment group only had one experimental unit, then we would not be able to conclude that any changes in the response are due to the treatments. It is also possible that some characteristic of the unit was the cause of the change.

Ex: our coke experiment

24


Increasing the __________________ makes randomization more effective. The more subjects we have, the more balanced our treatment groups will be. For example, if we have 10 subjects and only 2 have a certain unknown characteristic, it is quite likely that both of those subjects will end up in the same treatment group simply by chance.

25


However, if we have 100 subjects and 20 have the characteristic, it is very unlikely for all 20 to end up in the same group. There is a much better chance that the groups will be close to balanced (10/10, 9/11, 11/9, etc.) when the sample size is larger.

If you were flipping a coin and wanted to get as close as possible to 50%, you would decide to flip the coin more than a few times!

26


Note: Replication can also refer to repeating the experiment with different subjects. This can help us be more confident applying the results of our experiment to a ________________________.

SUMMARY: With control, blocking, randomization, and replication, each treatment group should be nearly identical, and the effects of extraneous variables should be the same in each group. Now, if changes in the explanatory variable are associated with changes in the response variable, we can conclude that it is a cause-and-effect relationship.

27

Caffeine Experiment Design

Before Treatment AfterTreatment Treatment

Questionrandom Administerassignment Treatmentto group by someone

Does Measure CAFFEINECaffeine Resting Measure Collect & Analyze ConclusionsAffect Pulse WAIT Resting Comparative &Pulse Rate Pulse Data ResultsRate? Rate by someone by someone

Againrandom Administerassignment Placeboto group by someoneNO CAFFEINE

Exp Units don't someone someone someone know who is in could be could be could bewhich group blind blind blind

29

End of Designing Experiments

30

More Designing Experiments

Not all experiments have CONTROL GROUPS or use a PLACEBO, as long as there is comparison. For example, if you are testing a new drug, it is usually compared to the currently used drug, not a placebo. Or, you can do an experiment to compare four brands of paint without using a placebo.

31


There are ethical issues to consider when doing experiments: smoking and lung cancer: we cannot force people to

smoke, but that would be the best way to prove smoking causes lung cancer.

many medical experiments are ended early if the experimenters discover that one treatment is much more effective: ARTICLE: Angioplasty Protocol Questioned: LA Times, 9-6-05

“The Tuskegee Study of Untreated Syphilis in the Negro Male”

32


The ethical rules that govern medical experiments are a relatively recent development.“The Tuskegee Study of Untreated Syphilis in the Negro Male” (1932 – 1972):- Before 1945, there was no effective treatment. The ones attempted were dangerous. Initially, the study addressed the question whether untreated syphilis is preferable to these remedies.- In 1945, a cure for syphilis was discovered. However, the study was continued and treatment was withheld.- By the end of the study, 128 out of 399 died of syphilis or its complications.

33


Even if caffeine has no effect on pulse rates, the average pulse rate of the two groups will probably be slightly different. Thus, whenever we do an experiment and find a difference between two groups, we need to determine if this difference occurred because of RANDOMIZATION VARIABILITY or because there really is a difference in the treatments.

The results of an experiment are called STATISTICALLY SIGNIFICANT if they are unlikely to occur by random chance.

34


AP Test, 1999, Question 3:

35


36


37

The Scope of Inference

Not all experiments and their resultant hypothesis tests use random samples. For example, in a study at UNC hospital, the researchers tried to determine the minimum daily requirement of choline for adults. They advertised at homeless shelters and natural food stores for volunteers. The subjects will spend 71 consecutive days in a hospital room and be paid $4,500 for their time and blood.

38


These subjects are just the first 80 people who are willing to participate and who pass whatever general health requirements are necessary. They are not a random sample from any population. Many medical studies proceed along these lines. The randomization lies in the random assignment of treatment to the experimental unit (subject), not in the random selection of subjects from a population. With a random assignment of treatment to subject, the researchers can make causal inferences from the results of their study. They cannot make inferences to a larger population. It is the inference to the population from which the sample was drawn that is lost when techniques of random sampling are not used.

39


On the other hand, in observational studies, there are generally random samples from distinct populations, so inferences can be made to those populations. However, causality cannot be attributed, since there was no random assignment to treatment groups.

40


The scope of inference refers to the type of inferences (conclusions) that can be drawn from a study. The types of inferences we can make (inferences about the population and inferences about cause-and-effect) are determined by two factors in the design of the study: how the subjects were selected from the population how the subjects were assigned to groups.

41


Allocation of Subjects to Groups

Randomized Not randomized

Selection of subjects from the population

Random Inferences about pop. – YES Inferences about cause and effect – YES

Some experiments are here.

Inferences about pop. – YES Inferences about cause and effect – NO

Most obs. studies are here.

Not random

Inferences about pop. – NO Inferences about cause and effect – YES

Most experiments are here.

Inferences about pop. – NO Inferences about cause and effect – NO

Some obs. studies are here.

42


Suppose a dentist wants to know if a daily dose of 500 mg of vitamin C will result in fewer canker sores in the mouth than taking no vitamin C.

43


Case 1) The dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who has complained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

44


What can we conclude?Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population. Since the study was observational, with subjects not randomly assigned to treatments, no causal inference can be made. We just know that for these patients, those who take vitamin C have fewer canker sores than those who don't. We don't know why, and we don't know if this result would be consistent with another group.

45


Case 2) A dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

46


What can we conclude?Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population. However, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C. We don't know if this result would be consistent with another group, but we believe we know why, for this group, the proportions differ.

47


Case 3) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who has complained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

48


What can we conclude?Since the patients selected were a random sample of dental patients in town, we can infer that the results observed in this experiment would be consistent with results from the whole population of dental patients in this town. However, since the study was observational, with subjects not being randomly assigned to treatments, no causal inference can be made. We believe that, for the population of dental patients in this town, those taking vitamin C have fewer canker sores than those who didn't. We don't know if it is the vitamin C that causes this reduction or some other confounding variable. We cannot conclude that for the general population, those taking vitamin C have fewer canker sores, since the sample was only of dental patients.

49


Case 4) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

50


What can we conclude?Since the patients selected were a random sample of dental patients in town, we can infer than the results observed in this experiment would be consistent with results from the whole population of dental patients in this town. Moreover, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C. We believe that for the population of dental patients in this town, taking vitamin C results in fewer canker sores.

51

Our Experiment

52

Does Caffeine Affect Pulse Rate?

An AP Stats Experiment

With Blocking by Gender

53

Before TreatmentTreatment

random Administerassignment Treatmentto group by someone

CAFFEINE CAFFEINE

Question Block by Matched Pairs random AdministerGender by Pluse assignment TreatmentMales Rate to group by someone

NO CAFFEINE NO CAFFEINE

Does MeasureCaffeine Resting Affect Pulse Pulse Rate1

Rate? (R1) random AdministerBlock by Matched Pairs assignment TreatmentGender by Pluse to group by someone

Females Rate CAFFEINE CAFFEINE

random Administerassignment Treatmentto group by someone

NO CAFFEINE NO CAFFEINE

54

The Caffeine Experiment

Initial pulse rate: ____________________

Drink you were assigned to: ___________________

55


Experiment Summary: Response variable:

pulse rateExplanatory variable:

caffeineExperimental units:

students in this AP Statistics classTreatments: caffeine (the quantity in a cup of Regular Coke) very little caffeine (the quantity in a cup of

Caffeine-Free Coke)

56


Variables that were directly controlled: amount of sugar drink temperature environmental conditions while drinking

Variables that were blocked: gender initial pulse rate.

57


What other extraneous factors can you think of? genetic factors tolerance to caffeine what or if students ate in the morning …

What was done about these other extraneous factors? Randomly allocating students in each matched pair should balance the groups with respect to any other factor.

58


How was the replication principle applied in the experiment? There were several matched pairs (more would

have been better). Class size large lots of matched pairs.

Unfortunately, the experiment won’t be repeated with two classes, as there is only 1 AP Stat class this year.

59

Treatment After Treatment

Administer Measure CalculateTreatment New Resting Differenceby someone Pulse Rate in R2 -

CAFFEINE R2 R1 CalculateDifferencein Pulse

Administer Measure Calculate A - BTreatment New Resting Differenceby someone Pulse Rate in R2 - Record Open

NO CAFFEINE R2 R1 Differences Envelopefor Alland Make

WAIT Compare Conclu-with Dot sions

Administer Measure Calculate PlotTreatment New Resting Differenceby someone Pulse Rate in R2 -

CAFFEINE R2 R1 CalculateDifferencein Pulse

Administer Measure Calculate A - BTreatment New Resting Differenceby someone Pulse Rate in R2 -

NO CAFFEINE R2 R1

60


Describe: Shape, Center, Bias, Variability (Spread)

Combined GendersO O

O O O X X O X O X O-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

61


AP Test, 2000, Question 5:

62


63


64


1 designing experiments suppose we wanted to design an experiment to see if caffeine affects pulse...

Documents

designing experiments

pulse rates

experiment subjects

initial pulse rate

everybodys pulse rate

class slide

caffeine free coke

response variable