proposed rearrangements and additions new order...

Module 2 – Proposed Rearrangements and Additions

New Order Topic L.O. Comment Mathematics Required Software? Stage Corrections, Changes to Existing Material

BN2.1 Now Showing: Sampling Introduction Five Questions None No Engage

BN2.2 Sampling Sense Current BN2.2 Multiplication, Percentages No Engage Changed title from “A Few MOE Basics” to get MOE out of the way

BN2.3 Determined Polling Current BN2.5 None No Engage

BN2.4 Now Showing: Language and Techniques Five Questions None No Engage

BN2.5 Prudent Survey Design Current BN2.1 None No Engage Remove the” Have a look at this …” and remove Exhibit 2. Increase room to write

BN2.6 What’s Simple About an SRS? New Counting, proportional reasoning No Engage

BN2.7 Are Online Reviews Statistical Samples? New None No Reflect

BN2.8 Random or Representative? New Fractions and Percentages No Reflect

BN2.9 Response Substitution New Reading graph No Reflect

BN2.10 Research Randomizer New Averages Yes Extend

BN2.11 Purposive Sampling New None No Extend

BN2.12 How Do National Polls Sample? New Proportional reasoning No Extend

BN2.13 Now Showing: Confidence Intervals Five Questions Addition, Multiplication, Sqr Rt No Engage

BN2.14 MOE Information New Addition, Multiplication, Sqr Rt No Engage

BN2.15 MOE in Practice New Addition, Multiplication, Sqr Rt No Engage

BN2.16 Essential Language Current BN2.4 Addition, Multiplication, Sqr Rt No Engage Change name to “Practicing What You Know”

BN2.17 Making Decisions with Confidence Intervals New Addition, Multiplication, Sqr Rt No Engage

BN2.18 Mathematically Organic Bells Current BN2.6 None No Engage First paragraph: “actually” should be “actual”

BN2.19 Confidence in Repetition Current BN2.10 Percentages, Fractions No Reflect Item 4: “Record you …” should be “Record your …”

BN2.20 A Challenging Interpretation New None No Reflect

BN2.21 Corn Hole Computations Current BN2.9 Area of square, circle No Reflect

BN2.22 Empirical Rule New Graphs, means, std deviations Yes Extend

BN2.23 Catch Me if You Can Current BN2.13 Add., multp, divide, sqr root No Extend

BN2.24 Confidence Intervals for Means New Means, std deviations Yes Extend

BN2.25 Now Showing: When MOE Doesn’t Apply Five Questions None No Engage

BN2.26 No MOE Today Current BN2.14 Percentages No Engage

BN2.27 Non-Sampling Errors: The Elephant Current BN2.15 None No Engage

BN2.28 Trying to Tame Non-Sampling Current BN2.16 None No Engage

BN2.29 Do Incentives Help? New Fractions No Reflect

BN2.30 Wording Matters – Seriously! New None No Reflect

BN2.31 The Role of Race in Survey Data New None No Extend

What About?

BN2.7, 2.8 Alternatives to BN2.18 (new order) I suggest adding to 2.18 (new numbering) as options. They will need to be reworded some.

BN2.11,2.12 Alternatives to BN2.20 (new order) I suggest adding to 2.19 (new numbering) as options. They will need to be reworded some.

BN2.3 Now Showing Sampling Content Remove. Now have the “Now Showings” distributed.

BEYOND THE NUMBERS 2.1_ LEARNING OUTCOME _

Now Showing: Sampling Introduction

Name: Section Number:

To be graded, all assignments must be completed and submitted on the original book page.

Sampling Content Videos Answer the following questions while watching the content video on the Introduction to sampling. 1. What is the goal of sampling?

2. What was the problem with the sampling done by the Literary Digest in 1936?

3. What proportion of people in the Ann Landers poll said they would not have children again if they had it to do over?

4. In what sense was the follow-up Newsday poll likely to be a more accurate representation of public opinion on the issue of having children?

5. Give an example not on the video of a biased sample.


Now Showing: Language and Techniques



Sampling Content Videos Answer the following questions while watching the content video on the Sampling Language and Techniques. 1. Define and distinguish “sample” and “population.”

2. Define and distinguish “statistics” and “parameter.”

3. Explain what a simple random sample is.

4. What are the two keys to having confidence in your parameter estimate?

5. What is the difference between a cross-sectional sample and a simple random sample? Give examples of each.

BEYOND THE NUMBERS 2.6_ LEARNING OUTCOME 5

What’s Simple about an SRS?



Background The definition of a simple random sample (SRS) can be confusing: a sample of size n, chosen in such a way that all samples of size n have the same chance of being chosen. It doesn’t help that the word is used lots of ways, but when it comes to selecting a simple random sample we have to be very careful to know its technical meaning. We will explore these issues in this set of activities.

Exhibit 1

Random Evolution On November 30th, 2012 National Public Radio’s Neda Ulaby ran a short segment entitled “That's So Random: The Evolution of an Odd Word.” You can likely find the short audio here at the link here: http://www.npr.org/2012/11/30/166240531/thats-so-random-the-evolution-of-an-odd-word . If not, look around for it elsewhere or contact your instructor. Question Give two uses of the word “random” from the audio that are different than the technical definition that is given above. Exhibit 2

Careful Counting The audio above ends with Charlie McDonnell (of the British “Fun Science” videos) noting that “every now and then, at random, you end up with something awesome.” We might take that to mean that every now and then a simple random sample will be representative of our population with respect to a certain list of demographics. Let’s look at that some more. Suppose you have a population with two men, one a Republican and one a Democrat; and two women, one a Republican and one a Democrat.

Questions 1. List all possible sample of size two. Make sure your notation makes it possible to distinguish all

four members of the population. 2. If you took a simple random sample of size two from that population of size four, what has to be

the chance that any sample of size two is chosen? 3. Suppose for a sample of size two to be “representative” of the population, it has to have exactly

one man and one woman, and one Democrat and one Republican. What is the chance of selecting a simple random sample of size two from this population that is representative in this sense of the word?

Exhibit 3

Social Media Sampling Suppose you have 113 “Friends” on Facebook and you want to choose a simple random sample of 20 of these. Answer the following: 1. What is your population? 2. Describe in detail how you would select the sample.

BEYOND THE NUMBERS 2.7_ LEARNING OUTCOME

Are Online Reviews Statistical Samples?



Exhibit 1

Bravos for Bucks By early 2012 the VIP brand Kindle Fire cover had received 4,945 reviews on Amazon for a nearly perfect average rating of 4.9 out of five. That’s quite impressive. It is tempting to think of the reviews that products receive online, especially at major sites like Amazon, to be somewhat representative of consumer experiences, even though we already know that voluntary responses are often biased. But are product reviews even less accurate? In his 2012 Time article “9 Reasons Why You Shouldn’t Trust Online Reviews,” Brad Tuttle writes “You shouldn’t believe everything you read. And if you’re reading online reviews of products, hotels, restaurants, or local businesses or services? Then you should believe even less.” You can likely find the entire article at the site below. If the link is broken go to your library to access a copy. After you have read the article, please answer the questions that follow. http://business.time.com/2012/02/03/9-reasons-why-you-shouldnt-trust-online-reviews/

Questions 1. List three reasons the article gives for why you should be very cautious about online reviews. 2. What was VIP doing to boost the ratings of their Kindle cover? Be specific. 3. How well did people do, versus computer algorithms, in the study to see how well fake reviews

could be spotted?

Exhibit 2

Bosom Bezos Buddy Fixing the problem with biased online product reviews is not going to be easy to do. But let’s pretend just for the moment that the founder and CEO of Amazon, Jeff Bezos, hired you to offer a solution that captures what you know about statistical sampling. Offer Mr. Bezos a sampling plan that is statistically defensible. Make sure you identify your population, your samples and how you plan to select them, as well as the parameter and statistic(s) of interest. You also should explain the reasons your sampling plan is better than what is currently being done and how it will correct for some of the problems you saw in Exhibit 1.


Random or Representative?



Gulliver Travels Suppose you know the following demographics on the population of 900 citizens in the small town of Gulliver, Michigan*.

Question 1 You want to know what proportion of the 900 residents support legalizing marijuana. You have enough money to interview 90 residents. Working much the way Gallup did in the 1930’s you want your sample of 90 to exactly mirror the distribution of subjects in the population, at least along the lines of gender, income, and political affiliation. How many people would your sample place in the groups shown below? If a calculation suggests you need a partial person (e.g. 6.4 persons), then just leave the number as it is, don’t round. *There really is a Gulliver, MI of about this size. The demographics are completely made up, however.

Gender 500 are Females

Income 20% make more than

$80,000 70% make between $40,000 and $80,000

10% make less than $40,000

Political

Affiliation 70%

Democrat 30%

Republican50%

Democrat50%

Republican30%

Democrat 70%

Republican Gender

400 are Males

Income 40% make more than

$80,000 50% make between $40,000 and $80,000

10% make less than $40,000

Political

Affiliation 60%

Democrat 40%

Republican50%

Democrat50%

Republican40%

Democrat 60%

Republican

Category Number of Persons Males Females Males making between $40,000 and $80,000 yearly Females making less than $40,000 per year who are Democrats Male Republicans making over $80,000 per year

Question 2 Suppose the cross-sectional sample taken above was successful in the sense that the sample was perfect microcosm of the larger population with respect to the legalization of marijuana. Is there any uncertainty involved in that sample-based estimate of the proportion of people in Gulliver who favor the legalization of marijuana? Explain. Question 3 After taking this class the Gulliver pollster worries she should had taken a simple random sample instead. Explain how an SRS of size 90 could have been taken from this population. Question 4 A carefully chosen simple random sample may not be representative of the population. Explain how this could be.

FYI for Draft. Just so we know what the students will be seeing.

Gender 50 are Females

Income 10 make more than

$80,000 35 make between

$40,000 and $80,000 5 make less than

$40,000

Political Affiliation

7 Democrat

3 Republican

17.5 Democrat

17.5 Republican

1.5 Democrat

3.5 Republican

Gender

40 are Males

Income 16 make more than

$80,000 20 make between

$40,000 and $80,000 4 make less than

$40,000

Political Affiliation

9.6 Democrat

6.4 Republican

10 Democrat

10 Republican

1.6 Democrat

2.4 Republican


Response Substitution



Background Two researchers at the Kellogg School of Management at Northwestern defined a new form of bias in surveys*. Professors Gal and Rucker call this bias “response substitution” and refer to it as the “process whereby some respondents frame their answers to questions in a survey in such a way as to express their views on issues outside the survey’s scope—issues on which they have strong opinions. “ Here is one of three experiments they ran in an effort to see if the phenomenon was real.

Fondue Folly One-hundred and thirty-seven undergraduates were assigned to one of four treatments. Roughly ¼ were presented with Scenario 1 - about “Anne” - (below) and asked to rate Anne’s wastefulness first, and then her intelligence (Treatment 1). Another ¼ were also presented with Scenario 1, but rated her intelligence before rating her wastefulness (Treatment 2).

Scenario 1 - Anne has a dinner party for some friends. She buys a fondue set and ingredients for a fondue dinner. The dinner party costs her $250. Her friends enjoy the meal and have a good time. She never uses the fondue set again.

The remaining students were presented with Scenario 2 – about “Jane” – (below) and asked to do the same kind of rankings: half of the remaining ranking Jane’s intelligence first, then her wastefulness (Treatment 3) and the rest ranking her wastefulness first , then her intelligence (Treatment 4).

Scenario 2 - Jane has a dinner party for some friends. She reserves dinner at a fondue restaurant. The dinner party costs her $250. Her friends enjoy the meal and have a good time.

All rankings were on a five-point scale, with 5 being the highest perception of intelligent (or wastefulness). A plot of the average ratings in each treatment appears on the following page. *Journal of Marketing Research (JMR); Feb. 2011, Vol. 48 Issue 1, p185

Questions 1. Make sure you can read the

graph correctly. Please write the correct average rating alongside the corresponding treatment.

Treatment Avg. Rating

1 2 3 4

2. Professors Gal and Rucker used a plot very much like this one to argue that these results are

“consistent with response substitution because wastefulness led to more negative perceptions of intelligence when participants did not have an opportunity to provide their attitude toward wastefulness.” Explain very carefully how the plot supports that conclusion.

3. How might you try to address response substitution when designing a survey?

2.16

2.67

3.06 3.02

2

2.5

3

3.5

IntelligenceRated First

WastefulnessRated First

Inte

llige

nce

Rati

ng

Anne's Scenario

Jane's Scenario


Research Randomizer



Background A simple random sample is the easiest kind of “statistically correct” sample to think about. But how do you actually select an SRS? One useful tool is Research Randomizer at http://www.randomizer.org/ . These activities are designed to allow you to get familiar with that tool.

Exhibit 1

No-Stumble Sampling Data from the NHTSA’s 1998 San Diego field sobriety test validation study are in the appendix of this workbook. There are 296 participants in this study, so there are 296 “Case” numbers shown, but they don’t run sequentially from 1 to 296. Your job is to use Research Randomizer to select a sample of 20 cases from this data set. Questions 1. Explain how you plan to identify the cases for Research Randomizer. 2. What entries did you use for the following Research Randomizer fields?

How many sets of numbers do you want

to generate?

How many numbers

per set?

Number range

(e.g. 1-50):

3. For the 20 cases selected, fill out the following chart:

4. What is the average BAC (“Blood Alcohol Content”) of the 20 selected?

5. What proportion of cases in your sample had BACs at or above the legal limit of 0.04?

Exhibit 2

Social Media Sampling Revisited Suppose you have 113 “Friends” on Facebook and you want to choose a simple random sample of 20 of these to ask a survey question you have constructed. Question Carefully explain how could you use Research Randomizer to select your sample?

Case Number Actual BAC Case Number Actual BAC


Purposive Sampling



Exhibit

Probability in Peril A simple random sample is a type of “probability” sample. The upside of probability samples is that is it possible to develop rigorous mathematical assessments of how good sample-based estimates of parameters are. Another type of sampling is “purposive sampling.” It is not probabilistic and, as a result, critically useful ties with probability theory are cut and numerical assessments of sampling integrity are typically just not available. Still yet, one often sees these types of samples particularly in social science studies, so it is important to know what they are. You will look at four in this activity:

Convenience Sampling Snowball Sampling Heterogeneity Sampling Expert Sampling

Use your research skills to look up each of these types of samples and answer the questions below.

Questions 1. Define what is meant by a convenience sample and give an example.

2. Define what is meant by a snowball sample and give an example.

3. Define what is meant by a heterogeneity sample and give an example.

4. Define what is meant by an expert sample and give an example. 5. Give some well-stated reasons why you think inference from these types of samples to a larger

population would be difficult to do?


How Do National Polls Sample?



Exhibit 1

Getting Gallup The following is an excerpt from the Gallup Poll site “How does Gallup polling work” http://www.gallup.com/poll/101872/how-does-gallup-polling-work.aspx

…. The majority of Gallup surveys in the U.S. are based on interviews conducted by landline and cellular telephones. Generally, Gallup refers to the target audience as "national adults," representing all adults, aged 18 and older, living in United States. The findings from Gallup's U.S. surveys are based on the organization's standard national telephone samples, consisting of directory-assisted random-digit-dial (RDD) telephone samples …. A computer randomly generates the phone numbers Gallup calls from all working phone exchanges (the first three numbers of your local phone number) and not-listed phone numbers; thus, Gallup is as likely to call unlisted phone numbers as listed phone numbers.

Questions 1. What is the actual population being addressed by a Gallup telephone survey? Be very precise with

your answer. 2. In what sense can a directory-assisted random-digit-dial sample taken by Gallup be thought of as

a simple random sample? Be very specific.

Exhibit 2

Weighting Room In 2012 The National Journal reported: "Critics allege that pollsters are interviewing too many Democrats -- and too few Republicans or independents -- and artificially inflating the Democratic candidates' performance." Let’s make up some data so we can better understand the issue being surfaced. Suppose a simple random sample of voters yielded the following poll results: Questions 1. What proportion of likely voters overall (Democrats and Republicans combined) planned to vote

for Obama? 2. Do you think the proportion you provided above is an underestimate or an overestimate? How is

this related to the claim in the National Journal? 3. Suppose we assume that in the larger population likely voters are split 50/50 as Democrats and

Republicans. Let’s assume that we had 50 Democrats and 50 Republicans in our sample, instead of 80 and 20. Re-compute the proportion be of likely voters who planned to vote for Obama. What percent decrease is this from the computation you did for Question1?*

*Comment: polling organizations often reweight in a similar, but more complicated, manner so as to adjust estimates to

better fit known population demographics. However, weighting election polls by party ID is very controversial because of

the uncertainty of what the actual voter turnout is going to look like.

Results of a random sample of 100 likely

voters

Planned to Vote

“Obama”

Planned to Vote

“Romney” 80 Democrats 70% 30%

20 Republicans 20% 80%


Now Showing: Confidence Intervals



Sampling Content Videos Answer the following questions while watching the content video on the Sampling: Confidence Intervals 1. What is sampling variability and why is it important to understand?

2. What is a sampling distribution and what characteristics will it have if formed from a simple random sample?

3. What is the formula for an 80% confidence interval for a population proportion?

4. What is the correct interpretation of a 95% confidence interval?

5. Specify an interval to the right of the parameter p where you can expect about 5% of all the sample proportions to fall.


MOE Information



Exhibit

Inference Assumed Title: A Shifting Landscape: A Decade of Change in American Attitudes about Same-sex Marriage and LGBT Issues (Feb. 26th, 2014) Authors: Robert Jones, Daniel Cox, and Juhem Navarro-Rivera Source: http://publicreligion.org/site/wp-content/uploads/2014/02/2014.LGBT_REPORT.pdf The following is an excerpt from the Executive Summary of this report:

Support for same-sex marriage jumped 21 percentage points from 2003, when Massachusetts became the first state to legalize same-sex marriage, to 2013. Currently, a majority (53%) of Americans favor allowing gay and lesbian couples to legally marry, compared to 41% who oppose. In 2003, less than one-third (32%) of Americans supported allowing gay and lesbian people to legally marry, compared to nearly 6-in-10 (59%) who opposed.

Near the end of the report, the authors add:

Results of the survey were based on bilingual (Spanish and English) telephone interviews conducted between November 12, 2013 and December 18, 2013, by professional interviewers under the direction of Princeton Survey Research Associates. Interviews were conducted by telephone among a random sample of 4,509 adults 18 years of age or older in the entire United States (1,801 respondents were interviewed on a cell phone, including 977 without a landline phone). The margin of error is +/- 1.7 percentage points for the general sample at the 95% confidence level. In addition to sampling error, surveys may also be subject to error or bias due to question wording, context, and order effects.

Questions 1. Using the MOE given in the article, construct a confidence interval for the true

proportion of all Americans in favor of allowing gay and lesbian couples to legally marry.

2. In the Executive Summary the statement is made that “… a majority (53%) of Americans favor allowing gay and lesbian couples to legally marry ….” Were all Americans asked? How could the statement be changed so that it is more precisely reflects the sample data?

3. Suppose you were to take another random sample of Americans at the same point in time and ask the same question. Would you be sure to find that more than 50% in the sample were in favor of allowing gay and lesbian couples to marry? Why or why not?

4. Offer and defend some reasons why the following statement appears in this article: “1,801 respondents were interviewed on a cell phone, including 977 without a landline phone.”


MOE in Practice



Exhibit

Is it just me, or is it warm in here? Title: Americans Do Care Authors: Annie Leonard Source: New York Times May 8, 2014

http://www.nytimes.com/roomfordebate/2014/05/08/climate-debate-isnt-so-heated-in-the-us/americans-do-care-about-climate-change

Americans do care about climate change. Polls showing lower levels of concern than in some countries don't tell the whole story. I travel widely around the U.S., attending meetings at schools, churches and community gatherings. Everywhere I go, I see people who are not only concerned about climate change, but are actively working on solutions. Nearly two-thirds (67%) of Americans accept the scientific evidence of global warming; fewer than one in six remain in denial. Two-thirds of Americans, including a majority of Republicans, want stricter limits on air pollution from power plants.

The full report referenced by Leonard’s article tells us that the original survey was by the Pew Research Center and was conducted Oct. 9-13, 2013, with a (95%) margin of sampling error of about 2.9% associated with the entire sample. Questions 1. What are the sample and the statistic for the Pew Center poll?

2. What are the population and the parameter?

3. Using the MOE given in the article, construct a confidence interval for the true proportion of all Americans who accept the scientific evidence of global warming.

4. When the data were broken down into subgroups, such as Republicans and Democrats,

the associated MOEs increased. Explain why that makes sense. 5. Polls often select something equivalent to an SRS of, say, “likely voters” and then break

that selection down into smaller subgroups, say “men” and “women.” Margins of error are then computed on the smaller subgroups using the same formula as for the original sample, only with a different sample size. Offer an objection to that and defend your answer.

BEYOND THE NUMBERS 2.17 (LEARNING OUTCOME 5, 9,10)

Decisions with Confidence Intervals



Exhibit

Sticker Shock Title: Great Jobs, Great Lives: The 2014 Gallup-Purdue Index Report

Authors: Gallup Organization and Purdue University

Source: Download access at http://products.gallup.com/168857/gallup-purdue-index-inaugural-

national-report.aspx

Does it really matter so much where you go to college? Not so much, this study found. Five areas of well-being for college graduates were measured: purpose well-being, social well-being, financial well-being, community well-being and physical well-being. The study concluded “… that the type of schools these college graduates attended -- public or private, small or large, very selective or less selective -- hardly matters at all …. Just as many graduates of public as not-for-profit private institutions are thriving -- which Gallup defines as strong, consistent, and progressing -- in all areas of their well-being.” The percent thriving for each institution type is shown in the bar chart below.

These results were obtained from internet surveys conducted Feb. 4 – March 7, 2014. The margin of error is estimated to be 1%.

11.00%

10.00%

12.00%

10.00%

4.00%

0.00% 5.00% 10.00% 15.00%

Public (selective)

Public (non-selective)

Private not-for-profit (selective)

Private not-for-profit (non-selective)

Private for-profit

Percent Thriving Well‐Beingin all Five Elements

Questions 1. Using a pair of confidence intervals, argue that the true thriving rates for graduates of Private not-

for-profit selective institutions is “95% likely” to be no different than the true thriving rates for graduates of Public not-for-profit selective institutions. Show all your work.

2. Using a pair of confidence intervals, argue that the true thriving rates for graduates of Private for-

profit institutions is “95% likely” to be different than the true thriving rates for graduates of Public not-for-profit non-selective institutions. Show all your work.

3. Why can’t you make an argument similar to the one you made above to show that the true thriving rates for graduates of Private not- for-profit selective institutions is “95% likely” to be no different than the true thriving rates for graduates of Public not-for-profit non-selective institutions? Still, the authors of the study concluded there were no differences across the board. What were they likely referring to? Be very focused with your answers.


A Challenging Interpretation



Exhibit

A Common Problem Psychology researchers* were interested in potential misinterpretations of what it means to claim “confidence” in a confidence interval. They collected data from 442 undergraduate students, 34 graduate students and 118 of their research-active colleagues. All subject were presented with a “… fictitious scenario of a professor who conducts an experiment and reports a 95 % CI for the [population proportion] that ranges from 0.1 to 0.4. Neither the topic of study nor the underlying statistical model used to compute the CI was specified in the survey.” The subjects were then asked to specify whether they Agreed or Disagreed with each of the following statements as an interpretation of that confidence interval (CI). Here is what the investigators found. Percentage of Subjects Agreeing with the Statement* Statement

First Year Students (n=442)

Master Students

(n=34)

Researchers (n=118)

1. The probability that the true proportion is greater than 0 is at least 95%

51% 32% 38%

2. The probability that the true proportion equals 0 is smaller than 5%

55% 44% 47%

3. There is a 95% probability that the true proportion lies between 0.1 and 0.4

58% 50% 59%

4. We can be 95% confident that the true proportion lies between 0.1 and 0.4

49% 50% 55%

5. If we were to repeat the experiment over and over, then 95% of the time the true proportion falls between 0.1 and 0.4

66% 79% 58%

* Hoekstra, R., Morey, R., Rouder, J., and Wagenmakers, E-J. “Robust misinterpretation of confidence intervals,”

Psychon Bull Rev. Published online January 14, 2014. http://www.ejwagenmakers.com/inpress/HoekstraEtAlPBR.pdf . Table has been reformatted, one statement omitted, and “mean” changed to “proportion” throughout.

All of the statements are wrong! Students and professionals alike find the awkward interpretation of a confidence interval to be challenging. Yet, it is not acceptable to step away from the challenge because these intervals are used everywhere as a kind of statistical seal of approval for the results of a survey or an experiment. Questions 1. What is wrong with Statement 1? Be very clear. Look back at your content on confidence intervals if

necessary to make sure you have the right interpretation. 2. What is wrong with Statement 2? Be very clear. Look back at your content on confidence intervals if




necessary to make sure you have the right interpretation.


The Empirical Rule



Background A bell-shaped distribution can mostly be characterized by where it peaks (mean) and how spread out it is (standard deviation). We already know that a bell-shaped sampling distribution is important to the construction of a margin of error and the associated confidence interval. However, bell-shaped distributions in general embody useful probabilistic information about the variable being described. The following well-known rule addresses this connection: Empirical Rule Suppose a bell-shaped distribution has mean μ and standard deviation σ. Then:

a. About 68% of all observations represented by that distribution will fall within one standard deviation of the mean.

b. About 95% will fall within two standard deviations of the mean. c. About 99.7% will fall within three standard deviations of the mean.

Graphically:

Exhibit 1

Face in Class Book In a 2012 Washington Post article entitled “Is college too easy? As study time falls, debate rises,” Daniel de Vise reports that “over the past half-century, the [average] amount of time college students actually study — read, write and otherwise prepare for class — has dwindled from 24 hours a week to about 15 …” No standard deviation is given, but let’s assume the standard deviation is 2.5 hours.

Questions

1. Suppose a college student is selected at random. Use to Empirical Rule to estimate how likely it is that this student studies between 10 and 17.5 hours per week.

2. Suppose a college student is selected at random. Use to Empirical Rule to estimate how likely it is that this student studies between 17.5 and 20 hours per week.

3. Suppose a college student is selected at random. Use to Empirical Rule to estimate how likely it is that this student studies more than 20 hours per week.

Exhibit 2

Class on Facebook A 2012 study looked at “the efficacy of social networking systems as instructional tools.” The study surveyed 186 students and asked them various questions about using social networking systems as an active part of the semester class structure. One question asked and answered by 181 of the 186 students, along with the results is shown below:

Question: There are no specific benefits that make Facebook a better forum for class discussions and announcements over a learning management system like Blackboard Response

Number of Subjects Choosing this Response

1 – Strongly Disagree 9 2 - Disagree 43 3 – Neutral/Undecided 59 4 – Agree 52 5 – Strongly Agree 18

Make sure you can read the table. There were 9 responses that were “1”, 43 that were “2”, etc. The standard deviation of these 181 answers is 3.15 and the standard deviation is 1.05.

Questions

1. Use the Empirical Rule to estimate how likely it is that an answer to this question will be in the interval 2.10 to 4.20. What was the actual percentage of answers in this sample in that interval?

2. Use the Empirical Rule to estimate how likely it is that an answer to this question will be above 4.20. What was the actual percentage of answers in this sample in that interval?

Buzzetto-More, N. “Social Networking in Undergraduate Education,” Interdisciplinary Journal of Information, Knowledge, and Management Volume 7, 2012

3. This example is a little different since only 5 outcomes (1-5) are possible and the Empirical Rule doesn’t strictly apply. It still provides useful estimates, however. Graph the distribution of these 181 answers and show that it is, indeed, bell-shaped. You must use computer software (such as Microsoft Excel or Apple Numbers) to create this plot. Cut and paste the plot (literally or electronically, depending on how you are instructed to turn this exercise in) in the space below.

4. Confirm that the mean is 3.15 and the standard deviation is 1.05, as claimed. You must use computer software (such as Microsoft Excel or Apple Numbers) to do this. Give detailed instructions to the reader explaining how you accomplished this.


Confidence Intervals for Means



Background This workbook is focused primarily on sample and population proportions. However, one can also compute confidence intervals for a population mean μ, based on a sample mean x. The formal is below:

x z∗s

√n

where s is the standard deviation of the sample data, n the sample size, and z* the confidence coefficient you first encountered with proportions. While this is not mathematically the exact formula, it will be fine for our purpose. Your instructor may or may not choose to clarify differences between t-and z-distributions at this point. The interpretation associated with this interval is just the same as it is for proportions, with the obvious change to “mean” language.

Exhibit

Inaugural Intervals A random sample of 10 U.S. Presidents was taken and the age at inauguration recorded. See the table on the right.

Questions

1. Find the sample mean of the ages of these ten Presidents, in days, at the time of their inauguration. You are required to use a software package as directed by your instructor (e.g. Microsoft Excel or Apple Numbers).

President Age at inauguration

James Madison 57 years,353 days

Martin Van Buren 54 years, 89 days

Millard Fillmore 50 years,183 days

Warren G. Harding 55 years,122 days

William McKinley 54 years, 34 days

William Howard Taft 51 years,170 days

George Washington 57 years, 67 days

Benjamin Harrison 55 years,196 days

Franklin D. Roosevelt 51 years, 33 days

Ulysses S. Grant 46 years,311 days

2. Find the sample standard deviation of these ten Presidents in days. You are required to use a software package as directed by your instructor (e.g. Microsoft Excel or Apple Numbers).

3. What is z* for a 90% confidence interval?

4. Compute a 90% confidence interval for the true average age (in days) of all U.S. Presidents (through President Obama) at the time of inauguration.

5. Carefully interpret the interval you computed in Question 4.

6. Do some research on your own and determine the true average age of all 44 U.S. Presidents, through President Obama. You are required to use a software package as directed by your instructor (e.g. Microsoft Excel or Apple Numbers). See if the interval you computed in Question 4 contains this true average. Whether it does or doesn’t, explain the chances of that outcome happening.


Now Showing: When MOE Doesn’t Apply



Sampling Content Videos Answer the following questions while watching the content video on When the MOE Doesn’t Apply. 1. What kind of error does the margin of error address?

2. What is the definition of a non-sampling error?

3. What was the issue with the question “Have you often, sometimes, hardly ever, or never felt bad because you were unfaithful to your wife?” How would this question affect errors in the survey results?

4. What are some strategies for reducing non-sampling errors?

5. Give your own example of a survey you have taken or know about that would potentially have a notable source of non-sampling error.


Do Incentives Help?



Exhibit 1

Pay to Play Early Title: Efficacy of Incentives in Increasing Response Rates Authors: Fahimi, M., Whitmore, R.W., Chromy, J.R., Siegel, P.H., & Cahalan, M.J. (2006). Source: Presented at 2nd International Conf on Telephone Survey Methodology, Miami, FL. http://www.rti.org/publications/abstract.cfm?pubid=6019 An internet search on the effect of incentives on increasing response rates will reveal a lot of research directed toward that end. The study by Fahimi, et al., at the Research Triangle Institute in Raleigh-Durham, North Carolina is particularly interesting. In Phase I of the study 1,197 subjects were contacted and asked to complete a survey. However, those subjects were randomly split into three subgroups. Subjects in the first group were not offered any monetary reward for early completion of the survey. Subjects in the second were offered $20, and those in the third, $30. Here is what the researchers found with respect to number of respondents:

Incentive Group (Early Response)

Number of Respondents

Number of Non-respondents

Group 1 (0$) 66 336 Group 2 ($20) 120 271 Group 3 ($30) 138 266

Total 324 873 Questions

1. Compute appropriate fractions from the table to support or refute a claim that incentives matter for early completion of a survey.

2. Compute appropriate fractions from the table to support or refute a claim that a high incentive is more effective than a low incentive at achieving early completion of a survey.

Exhibit 2

No Pay Replay There were 873 non-respondents to the survey described above. In Phase II of the study, these 873 were contacted a second time and asked to complete the survey. No additional incentives were offered and the offers of incentives for early completion had already expired. Here is what the researchers found.

Non-respondents from Phase I.

No Incentive Offered for Late Completion

Number of Respondents

Number of Non-respondents

Group 1 (0$) 109 227 Group 2 ($20) 91 180 Group 3 ($30) 96 170

Total 296 577 Questions

1. Compute appropriate fractions from the table to support or refute a claim that no-incentive follow-up requests are effective at increasing response rates.

2. Compute appropriate fractions from the table to support or refute a claim that the effectiveness of no-incentive follow-up requests depends on whether an incentive for early completion had originally been offered, and whether it was a large or small incentive.

Exhibit 3

Some Pay Saves Day There were 577 non-respondents still remaining after the second phase attempt to get the survey completed, with no incentives offered at that stage. In Phase III, these 577 were contacted again and asked to complete the survey. This time, however, the remaining non-respondents were randomly divided into two groups - one would receive no compensation for completion and the other $30. Here is what the researchers found. Non-respondents from Phase II.

Incentive Groups for Follow-ups.

Respondent

Nonrespondent NF1 ($0) 98 190

NF2 ($30) 135 154 Total 233 344

Questions

1. Compute appropriate fractions from the table to support or refute a claim that incentives matter when doing follow-ups for survey completion.

2. At the end of Phase III, 233 people responded. All of these 233 had already been contacted two other times over the course of the experiment. Explain why this could create confounding if fancy statistical methods are not used to mitigate.

3. Take a step back and look at the results in all three of these Exhibits. Sue what you have learned to describe how you would incentivize (or not) a survey you wanted to administer. Your answer should include comments about incentives for early completion as well as incentives for follow-up completion. Remember, you are not being asked to design an experiment. That’s what the authors above did. Rather you are being asked to use what you have learned from their study to decide whether you want an incentive plan attached to your survey and, if so, what that should look like.


Wording Matters --- Seriously!



Exhibit 1

Knotty Not Title: Poll on Doubt of Holocaust Is Corrected Authors: By Michael R. Kagay Source: New York Times, July 8, 1994 http://www.nytimes.com/1994/07/08/us/poll-on-doubt-of-holocaust-is-corrected.html Just how important is question wording in a survey? A mixed-up Roper poll in 1992 makes the answer clear. While this happened a long time ago, it remains as one of the most sobering examples of how seriously we have to take non-sampling errors. In the article referenced above, a fatal flaw in the original poll is discussed and those first results are compared with the results of a corrected follow-up poll. Please access this article and answer the following questions. Questions

1. What was the exact wording of the question asked in the original 1992 poll, and what percentage of those surveyed suggested said it was possible the Nazi extermination of the Jews never happened?

2. What was the problem with the original question and how was that problem corrected in the follow-up poll?

Exhibit 2

When you say it that way … Title: Question Wording Authors: Pew Research Center Source: http://www.people-press.org/methodology/questionnaire-design/question-wording/ The Pew Research Center is one of the most prolific polling organizations today. The referenced article is a piece provided by the Center as a short tutorial on the importance of question wording. Please access the article and answer the questions below. Questions 1. The primary example provided is one from 2003 when Americans are asked about the desirability

of ending Saddam Hussein’s rule in Iraq. Two slightly different questions were asked with dramatically different results. What is the take-home message from this example?

2. The article lists several tips for survey construction. What are two “important things to consider

in crafting survey questions?” Your answers can be short one-sentence summaries.

A.

B.

3. Should marijuana be legalized in your state? Suppose you are charged with constructing a survey

to collect public opinion on that question. Construct two versions. The first version should be free of wording bias. The second version should exhibit a gentle push toward legalization.

a. Balanced

b. Biased


The Role of Race in Survey Data



Exhibit

Does Race Still Matter? Title: Stereotype threat and race of interviewer effects in a survey on political knowledge Authors: Darrin Davis and Brian Silver. Source: American Journal of Political Science, Vol. 47, No. 1, January 2003, pp 33-45 http://www.researchgate.net/publication/228828389_Stereotype_threat_and_race_of_interviewer_effects_in_a_survey_on_political_knowledge The study referenced above was published in 2003, almost 35 years after a study suggested race was a big factor in non-sampling survey errors. In the original study, interviewers in Detroit asked Black residents “Do you personally feel that you trust most white people, some white people or none at all?” When the interviewer was White, 35% answered “most.” When the interviewer was Black, only 7% answered “most.” Are we beyond this problem now? Access the article above and read it. Access is free. You may want to ignore the more complicated statistical summaries, but you should read the non-technical textual parts of the paper in detail. Questions 1. Describe “social desirability bias” and explain how it affects the accuracy of survey data. 2. Explain the concept of “stereotype threat” in the context of the gathering of survey data.

3. Explain how the experiment these authors implemented tried to eliminate “social desirability bias” as a cause for any treatment differences that would be observed.

4. The authors asked seven political questions to two groups of subjects. Describe what

distinguished the two treatments. 5. The study also collected racial information on the interviewer and the interviewee. Explain

how this was done. 6. A primary deliverable of this study was to evaluate the effect of (perceived) race of

interviewer on political knowledge of Blacks and Whites. What did the study find?

proposed rearrangements and additions new order...

Documents