Dada Veloso-Beltrandadaspeaks.com

survey

primer nuts & bolts

case study

By a small sample we may judge the whole piece.

Miguel de Cervantes (1547-1616)

EvolutionaryLack sharp corners & sudden turns.

primer brass tackscase study

Answers reveal relationships to something they measure.

designing questions to be good measures

Qualitative

Intent Bubbles

I want to see the world through the eyes of my respondents.

I want to describe the context in a lot of detail. I want to show

how social change occurs. I’m interested in how things come to be.

I want to make sure others can repeat my findings.

I really spend a lot of time wondering how to measure things.

I wonder how small patterns generalize to big patterns.

I want to know what causes something else.

Quantitative

I really want my research approach to be flexible and able to change.

A standardized stimulus designed to convert fuzzy phenomenon into data. (Neill. 2014)

Study of a population through observation of its members.

"The survey is a systematic method for gathering information from entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.”(Groves et al. 2004, p.4)

Research Design Matrix

Mixed Method

Quantitative

QualitativeExplore

Theorize

Validate

Optimize

ConstructivistInterpretiveBig Picture

ObjectivistPositivistGranular

Depth of Insight, Small Sample, Longer Engagement, Personal

Broadness of Phenomena, Large Sample, Shorter Engagement, General

No

YesI know a lot about the topicI need to summarize the findings in numerical formatI don’t need a lot of detail and nuanceI need to generalize the results to the populationI need to predict the likelihood of a certain thing happeningI want to measure incremental change

I don’t really know much about the topicI don’t need to summarize the findings in numerical formatI need to communicate a lot of detail and nuanceI’m interested in case study informationI’m not planning on predicting the likelihood of anythingI don’t need to measure delta

Is survey the right research

method?

Mutually exclusive and exhaustive§ No overlap between answers§ All possible answers

No double-barreled questions§ Wrong: “Is your work permanent or casual? “§ Right: “What is your opinion of our response time?”

Focus on opinions or beliefs of participants§ Wrong: “How usable is our Web site?”§ Right: “What are users’ opinions of the site’s look and feel?”

Close-ended is best§ Cuts down on analysis time

Demographic questions at the end

Survey Question

Do’s

Survey Question

Don’ts

Inapplicable: does not apply to all respondents§ How many years can a worker of average health be employed in your trade?

Over-demanding: e.g., recall of time-consuming details§ What are the general physical, intellectual and moral conditions of life of

the working men and women employed in your trade?

Ambiguous: meaning must be clear to all respondents§ Describe wage increases during so-called prosperity periods.

Leading§ Does your employer or his representative resort to trickery in order to

defraud you of your earnings?

Loaded§ Have you ever known any rank and file workers who could retire from

employment at the age of 50 and live on the money earned by them as wage workers.

Probability Sampling

Simple random sampling: Choose the people to survey randomly without any order or preference, like taking an email list and choosing 10% of them in no specific order and without any preference.

More scientific. Randomness ensures higher chances of representation in the target population.

Systematic Random Sampling: More organized version of random sampling, where you take an email list and choose, say, every 3rd on the list.

Stratified Sampling: this is when you first divide the target population into smaller groups, like separating email addresses according to how much each one of them earns or their gender or education level. Then in each group you systematically choose whom to talk to, for example every 5th person on the list.

Cluster Sampling: here you divide the target population into groups that seem natural, which almost always means based on geographical spread. After having done that, you can perform a simple random sampling within each cluster.

Non-Probability

SamplingJudgment: this is when the person conducting the survey uses personal judgment or experience to choose the people to interview, like standing outside a supermarket and choosing to talk only to the people that look respectable.

Often done for the convenience, subjects who are easily accessible or require the least expense.Use this sampling when you not looking for the exact truth, but just a decent approximation of it. Useful as a way of testing or gauging the market in a very simple way.

Quota: Cut the group you want to talk to into smaller groups according to gender, income, or some other criteria, just like in stratified sampling. Then you choose whom to talk to according to your judgment of which ones are more representative, or simply who is easier and least costly to reach.

Snowball: Asking people whom you have just surveyed to refer you to other related people. Economical way of sampling a group that is difficult to locate, like people with a certain illness or uncommon hobby, preference, or interest, as those people will usually have a network.

Is survey a numbers game?Your subject population informs the game.

If the population is large but homogenous, your sample can be small. If the population is very diverse, you need a larger sample.

So it’s not just about the absolute number of people surveyed. What's more important is the mapping of the target population correctly, and then sampling accordingly.

primer brass tackscase study

#EpicFail“That’s not what I heard.”

1948 US Election

Survey George Gallup introduced the world to survey sampling by predicting the result of a previous election.

Gallup was a respected figure in the field. He not only pioneered polling firms but advanced social innovation – feeling the pulse of the public on issues that affected them.

Telephone polls held before the elections consistently showed Dewey as the leading candidate. Other polling firms such as Crossley and Roper predicted the same result.

Instead of facilitating a transition in government, the polls had misled the presidential candidates and all other politicians, the Washington bureaucrats, the media, and the public.

Critical Errors

Polling firms ended their polling two weeks before Election Day. Therefore, they missed the importance of measuring preferences just before the election.

Pollsters had no certain way of deciding who would stay home on Election Day and who would go and cast a vote.

The polling firms did not put much premium on the undecided voters, and therefore miscalculated their margins of error.

The pollsters’ use of quota sampling that biased responses towards Truman.

The Social Science Research Council committee concluded that the 1948 polls were not up to the standards of science.

Legacy

It brought the academe and social scientists into a field they previously did not pay attention to. This brought rigor into survey as a method for gathering information on public opinion.

Survey design became more sensitive to errors of sampling and brought attention to the importance of designing questions that elicit more accurate indicators of their target population.

The academe developed a more strategic method for survey sampling –probability sampling, to lessen errors of underrepresentation or misrepresentation.

The Gallup Poll of the 1948 US Presidential Elections proved to be a watershed event that revolutionized the science of survey.

Up to this point, we in the field of public opinion research have had to carry the ball ourselves with little help, but with plenty of criticism, from the social scientists. (Gallup 1948)

The more I worked on this report, the more I felt the debt that we owe to these men who have been willing to risk their own money in trying to learn something about American political behavior… And we have a responsibility, we in the universities, to do our best to help improve these techniques…. Because I believe in this work as an instrument of democracy. (Stouffer 1948. p 214.)

Afamoussamplingmistake

#TurningPoint“…predicting behavior on the basis of knowledge of attitude is a very hazardous venture.” (Dinerman. 1948)

If we knew what it was we were doing, it would not be called research, would it?

Babbie, Earl. (2011) The Practice of Social Research. Belmont CA: Wadsworth.

Creative Research Systems (2009). Survey design: How to begin your survey project.

Fowler, F. J., Jr. (2002). Designing questions to be good measures. In In F. J. Fowler, Survey research methods (3rd ed.) (pp. 76-103). Thousand Oaks, CA: Sage.

Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004). Survey methodology. Hoboken, NJ: John Wiley & Sons.

Howitt & Cramer (2011):

Chapter 33 - The analysis of a questionnaire / survey project (pp. 407-415)

Lindloff, Thomas & Taylor, Bryan. (2011). Qualitative Communication Research Methods. Thousand Oaks, CA: Sage.

Nardi, P. (2006). Developing a questionnaire (Ch. 4). In Doing survey research: A guide to quantitative methods (2nd. ed.) (pp. 66-106). Boston, MA: Pearson.

Trochim, W. M. K. (2006). Survey research.

Zetterberg, Hans (2004). US Election 1948: The First Great Controversy about Polls, Media, and Social Science. Paper presented at the WAPOR regional conference on "Elections, News Media and Public Opinion" in Pamplona, Spain, November, 24-26, 2004. http://www.zetterberg.org/Lectures/l041115.htm

References