survey methods - oiisdp 2015
TRANSCRIPT
1. Measure behaviors, attitudes, or a construct
2. Collect relatively large amounts of data
3. Efficiency (cost vs. output)
Steps in survey process• Step 1: Selecting/developing questions
• Step 2: Select method of administration
• Step 3: Pilot testing
• Step 4: Sampling
• Step 5: Writing up results
What to look for in a good measure
•Published
•Used in a similar study
•Good psychometric properties
•Internal consistency estimates (𝛂 >.80)
•Good factor structure
•Evidence of validity
What is Factor Analysis?•Factor analysis is all about simplification
•Allows us to understand large quantities of observable variables in terms of a smaller number of unobservable variables
•These unobservable variables are called “latent variables”
Homer likes to stick his hand in beehives
Homer encourages his daughter to smoke
Homer is distracted during a reactor
meltdown
• These are all directly observable phenomena
• It might be easier just to say he’s stupid
• Stupidity is a latent variable.
Factor StructureFactor Loadings for Exploratory Factor Analysis with Varimax Rotation of Facebook Activities
Items Factor 1 Communicating
Factor 2 Self-presentation
Factor 3 Social Information Seeking
Commenting 0.796
Liking 0.649
Sharing links 0.635
Status updates 0.628
Private messages 0.444
Chat 0.422
Tagging photos 0.826
Posting photos 0.811
Viewing photos 0.798
Checking up on 0.626
Creating or RSVP’ing to events
0.310
Survey ModePros Cons
Online
Lower cost Quickly collect data
Easy to modify Review results
Branching
Response rates Digital Inequalities
Contacting participants Data loss
PhoneHigh response rates
Interact with real person Diverse sampling
High cost Take longer
Cell phone numbers
Mail Convenient
Detailed responses Lower cost
Efficient
Take much longer Higher non completion
Compete w/mail Project management
Levels of Measurement•Nominal: Names. Has no order. Assignment is arbitrary
(1=East, 2=North, 3=South, etc.)
•Ordinal: Has order, but interval between scale points may be uneven (1st place vs 2nd place runners compared to 50th and 51st place). Arithmetic operations are impossible with ordinal data. Can count and order.
•Interval: Has order and equal intervals with an arbitrary zero point (years: year 1 AD is arbitrary - not when time began). Can add and subtract but not multiply and divide.
•Ratio: Same as interval data with a true zero point (income: 0 income is truly no income). Can conduct all operations.
Measurement Level Operations Examples
Nominal No Ordering Sex (Male, Female)
Ordinal Ordering, but not distance
Student Class Standing (Freshman,
etc.)
Interval Distance, but not ratios GPA
Ratio Ratios Number of Credit Hours
Clarity• Provide clear instructions
• Emphasize rating scale
• Never use compound questions
• Will you allow “don’t know” option?
QuestionsOpen Ended Closed Ended
Responses Greater variety of responses
InterpretationRespondents
interpret question the same way
Missing Data More likely to skip
Data Analyses No coding involved
Writing Questions• Avoid leading words
• The government should force you to pay higher taxes.
• Give mutually exclusive choices
• What is your age?
• 0-10
• 10-20
• 20-30
Writing Questions• Be direct
• How do you use the Internet?
• Cover all possible answer choices
• You indicated that you no longer use Facebook. Why not?
• My parents joined Facebook
• I like Instagram better
• I lost my password
Likert Scales• Make sure they are ordinal and perhaps even
interval
• 5 or 7? Doesn’t matter
• Balanced # of positives and negatives
• Use scales with similar anchors
• Reduces cognitive load but…
• Spurious covariance
Question Order• Easy/fun/engaging questions first
• Sensitive & demographics last
• Counterbalance
• Branch when you can
Populations & Samples•Population: Entire collection of all of the data of
interest
•Sample: A subset of the population
Population
Sample
Choosing a Sample•Random:
1. Each person is chosen entirely by chance
2. Each member of the population has an equal chance of being included in the sample
•Representative: The characteristics of the sample should match the characteristics of the population
Population: All college students who use Facebook
Sample: All students in an introductory psychology course who use Facebook
Sample Size•Larger sample sizes generally lead to increased
precision when estimating parameters.
•Sample must be large enough to detect differences in significance testing.
•Calculate the sample size required to yield a certain power for a test, given a predetermined Type I error rate (ử).
•Power: The probability that you will conclude there is no relationship when in fact there is.
Statistical Significance•Types of Error
•Type I Error (significance): the chance you will conclude there is a relationship when there is not.
•The chance that another random sample from the same population would result in a relationship as strong or stronger than the observed one, just by chance of sampling. Typically set at 5% (p < .05).
•Type II Error (power): the chance you will conclude there is no relationship when in fact there is. Typically set at 20%, corresponding to a power level of .80.
•Low power: where relationships which are real cannot be found to be significant (usually because sample size is too small).
•High power: where even trivially small relationships are found significant (because sample size is excessive).
Choosing your sample•Calculate the sample size required to yield a
certain power for a test, given a predetermined Type I error rate (ử).
•Figure out how to obtain a sample that is representative of your population of interest.
•Randomly sample your population.
•Simple random sampling: when you have a list which approximates all members of the population, then you draw from that list using a random number generator.
Effect Size•A measure of how strongly the independent
variables affect the dependent variables.
• p-value is not effect size!
•Cohen suggested:
•Small: 0.01
•Medium: 0.059
•Large: 0.138
“These results were highly significant”