workshop: 'self-administered mobile survey workshop' - dr michael bosnjak, free university...
DESCRIPTION
Workshop - Part Theory: Prof. Dr. Michael Bosnjak gives an introduction into the methodological foundations of self-administered mobile surveys. Topics include issues of coverage, sampling, non-response and measurement applied to mobile survey contexts. Special attention is devoted to methods and procedures to increase response rates, and to visual design effects.TRANSCRIPT
Self-Administered Mobile Surveys
MRC 2011 Workshop (Part 1)
London (UK)April 18th, 2011
Michael Bosnjak, PhD, Assoc. Prof. Free University of Bozen-Bolzano, School of Economics and Management
1
• Definitions of mobile surveys– Interviewer-administered surveys
• Interviews among mobile phone users• Interactive voice response surveys among mobile phone
users
– Self-administered surveys• SMS (text messaging) surveys• Browser-based surveys on mobile devices (e.g., mobile
phones having mobile Internet-access, Smartphones, etc.)
• Our focus: – Self-administered surveys AND– using a mobile phone AND– browser-based.
2
Mobile Surveys?Self-Administered Mobile Surveys?
Selected Applications3
Directly at point of sale: in shopping malls and at points of service
At public venues, such as concerts
At schoolyards, in universities & recreational facilities
At trade fairs
In training seminars
In workplaces without internet
access
En-route with bus or
train & at the airport
En-route with bus or train & at the airport
B2BB2C
Insights from difficult to
reach target groups, event/incident-based
surveys, immediacy
Overall Goal
• Providing a very brief introduction into the methodological foundations of self-administered mobile surveys (esp. sources of biases known from survey methodology)
• Summarizing key findings of an own methodological study series conducted between 2008-2011
• Discussing practical, evidence-based recommendations (esp. on measurement and nonresponse issues)
Agenda
• Background– Survey research: Overall aims and scope– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues– What can be presented/assessed?
– How usable are mobile question formats?– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
Background: Overall Aim of Surveys• Measuring ´true scores´, i.e. yielding unbiased
estimates for facts and/or latent variables.– Examples of factual questions to measure facts:
• Household-level income/expense estimates > Disposable income
• Behavioral frequency estimates > Behavior
– Examples of indicators supposed to measure latent variables:• Evaluative judgments > Attitudes• Behavioral likelihood scales > Intentions• Brand/product related attributes > Image
• Sources of errors in surveys:• Representation-related biases: Coverage, Sampling,
Nonresponse• Measurement-related biases/errors
6
Background: Total Survey Error7
Construct
Measurement
Response
Population
Sampling Frame
Sample
Respondents
Survey estimate
Measurement Representation
Coverage
Sampling
Nonresponse
Measurement
Measurement
Inappropriate implementation into a specific mode: Undesired design-related effects
Inappropriate operationalization (range restriction, reliability, validity)
Representative for the population in question?
Representative (valid) for the construct in
question?
Background: Survey Errors/Biases
• Coverage ErrorMembers of the target population have no chance of being selected in the sample (e.g., no access to the Internet, incomplete lists etc.). Error due to the fact that not every unit in the population is represented on the frame.
• Sampling Error... arises from the fact that not all members of the frame population are measured.
• Nonresponse ErrorThe responses of people who have not been surveyed are different from those who actually have participated in a survey.
• Measurement ErrorDeviation of the answers of respondents from their true values on the measure, e.g. due to inappropriate operationalizations of (latent) constructs, design features and context effects.
8
Mobile Survey Methodology: Study Series9
Mobile Study I(1.7.-2.9.08)
1. Web: Item development: Determinants of the willingess to participate in mobile surveys (Sozioland Web-Panel)2. Pre-Testing: Expert usability assessment at YOC 3. Web: Determinants of the willingess to participate S4 (YOC Mobile-Panel; 979 panelists, 272 participants)4. Olympic Games 2008 Mobile Survey(YOC Mobile-Panel; 979 panelists, 413 participants)5. Web: Usability of S4 from participants´ perspective(YOC Mobile-Panel; 413 panelists from S4, 187 completes)
Mobile Study II(29.9.-18.10.09)
Mobile Study III(March/April 2011)
8. Usability of voice capturing/recognition technology (presentation of results at tomorrow at MRC 2011, April 19, 2011)
6. Mobile survey: Evaluation of last vacation(Respondi Web-Panel; 3270 panalists, 540 completes)7. Web: Usability of S6 from participants´ perspective(Respondi Web-Panel; 540 panelists from S6, 318 completes)
www.mobileresearchconference.com10
Agenda
• Background– Survey research: Overall aims and scope – The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues– What can be presented/assessed?
– How usable are mobile question formats?– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
What can be presented/assessed? (I)12
Single choice
Multiple choice
Drop-Down menu
What can be presented/assessed? (II)13
TextfieldMatrix / Polarity profile
Voice / image /video capturing
How ´usable´ are standard formats?14
Einfachauswahl untereinander
Mehrfachauswahl untereinander
Geschlossene Auswahlliste
Textfeld einzeilig
Fragetyp mit Bild
65,00 73,75 82,50 91,25 100,00
87,9
74,7
82,7
87,3
89,2
Usability score (Range: 0-100 Punkte)
Frag
etyp
Subjective Usability AssessmentPost-hoc survey (Web) one week after mobile survey completion
Indicators for usability score: fluency, simplicity, ease of use
Observed
Item-NR
Drop-Out
45%
9%
9%
23%
Multiple choice
Single choice
Drop-Down menu
Textfield
Image map
Voice recognition / capturing ?
Sources: MS I and MS II combined
15
Technical Implementation: iPhone App
16
Technical Implementation: Android
GPS positioning: Privacy concerns?17
91%
9%
Yes (willing to disclose)No (not willing to disclose)
Acceptance of GPS-Location
among participants with
an iPhone (MS II; n=45)
Agenda
• Background– Survey research: Overall aims and scope – The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues– What can be presented/assessed?
– How usable are mobile question formats?– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
Nonparticipation: Industry perceptions?
• Survey among 327 market researchers about acceptance and use of mobile surveys in D/A/CH
• Top 3 advantages of mobile surveys:– 51%: Independence of time/location– 49%: Context-sensitive, fast surveys– 43%: Reachability of hard-to-reach, mobile target groups
• Top 3 barriers for mobile surveys:– 35%: Costs incurred to survey participants (data traffic)– 35%: Difficulties entering information (esp. open-ended
questions)– 33%: Software/platform heterogeneity
19
Mobile Research Mobile Research
Barometer
Februar 2011
Mobile Research Barometer 2/2011
Nonparticipation: Self-Reports?20
Post survey in Wave 3, open responses, N= 63
´True´ Reasons for (Non)Participation I
• What is the influence of the following potential determinants of the willingness to participate?1.Attitude towards participating2.Hedonic aspects (perceived enjoyment)3.Social aspects (subjective norm)4.Image and perceived self-congruity5.Perceived benefits and costs
• Hypothetical modelExtended technology acceptance model (Venkatesh et al., 2003)
• Prospective study design (MS I)– S1: Developing and optimizing measurement models– S3: Assessing all above mentioned determinants– S4: Olympic games mobile survey (non)participation
21
´True´ Reasons for (Non)Participation II22
Discussion and Overview
The results of this study indicate that the extended TAM we propose offers a suitable heuristicframework for explaining both intentions to participate in mobile surveys and actual participation.Of the six factors we propose as influential, the hedonic, affective, self-expressive, and trust-relatedones emerge as important determinants of the propensity to participate. Utilitarian aspects, such ascost considerations and the perceived usefulness of using the mobile mode for surveys, as well asconsiderations involving the perceived social pressure, surprisingly do not appear to exert a signif-icant influence.
These results therefore offer some suggestions about ways to influence people’s decision to par-ticipate in mobile surveys. Because the propensity to respond seems primarily a matter of hedonic,affective, self-expressive, and trust-related factors, survey researchers must address these four con-structs through persuasive appeals. For example, to enhance hedonic and affective factors, messagesmight focus on the positive consequences of participation, such as enjoying the survey itself. Self-expressive appeals mainly stress the lifestyle and value attributes of stereotypical idealized persons(Johar & Sirgy, 1991); therefore, testimonials by aspirational spokespersons could be effective in
Figure 3. Structural model of participation in mobile surveys with standardized path coefficients (N ! 272).Notes: Solid arrows indicate significant paths; dashed arrows represent insignificant influences. For clarity, thisfigure does not depict the measurement models and cross-correlations between exogenous variables. A fullcorrelation matrix appears in Appendix B.
357
Bosnjak et al. 357
at SSC MASTER on August 22, 2010ssc.sagepub.comDownloaded from
*std.β, sig. at α=.05, N= 272
Fit Indices (robust)SB-Χ²=407; df=296p<.05, Χ²/df= 1.37NNFI=.98RMSEA=.04(.03-.05)
Highest influences:
> Hedonic aspects> Self-congruity
Not relevant:
> Expected costs (!)> Opinions of others
´True´ Reasons for (Non)Participation III
• If hedonic factors outperform cost/benefit-related, then– ´exciting´ incentives (lottery drawing) should
increase participation rates – compensation for incurred costs should undermine
hedonic motivation (salience of costs is increased)
• MS I experiment, manipulating basic compensation (1 EUR, yes/no) and announced prize draw (100 EUR voucher, yes/no)
• Results confirmed our expectations (see Appendix):– highest access and participation ratesfor „lottery & no incentive condition“
23
Geschwindigkeit des Zugriffs auf Welle 1 und 2
Kum
ulie
rtert
proz
entu
aler
Ant
eil
Stunden seit Einladung
Speed of participation? (MS I)24
Faster responses for Mobile compared to Web:
approx. 35% Mobile versus aaprox. 10 % Web
For about 4.5 hours, Mobile response rates are higher
compared to Web
Speed of participation? (MS II)25
0
2
3
5
6
8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00
Mean response speed in hours for different contact/invitation time points (sent out via SMS)
MS I
Current context of participation? (MS I)26
At home busy with my PC
Watched TV
Worked at home
Read
In the office / at work
Preparing / eating a meal
On the move
Nothing disrupted 23,47%
8,16%
7,14%
11,22%
8,16%
10,20%
14,29%
17,35%
„Were have you been taking part in the
survey?“(Wave 3; N=116)
At home
In the office / at work
In a car
At the bus or train station
Using public transport
On the move (other reasons) 5,17%
2,59%
4,31%
6,90%
17,24%
63,79%
"Which activity did you have to disrupt to take part in the mobile survey?/ What have you done in
that very situation?“ (Wave 3; N=98; open responses)
Optimal length of mobile surveys? (MS II)27
0 %
25 %
50 %
75 %
100 %
non iPhone iPhone
68,9 %
10,3 %
31,1 %
89,7 %
onlinemobile
„Do you want to continue answering the survey mobile or online (in this case you
will get a link via email)?“
Participants: n= 540
0
7,5
15,0
22,5
30,0
Total iPhone Non iPhone
20,119,319,8
5,25,15,2
Min
nute
s
Part 1: Mobile- Initial SurveyPart 1 & 2: Mobile Survey
Agenda
• Background– Survey research: Overall aims and scope – The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues– What can be presented/assessed?
– How usable are mobile question formats?– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
Take-Home Messages & Discussion
• ... on mobile survey measurement:– Most ´classical´ closed-ended item formats can be included and
are in many cases sufficiently usable– Various measurement options ´beyond´ the usual self-
administered formats do exist (e.g. GPS positioning, multimedia upload)
– Open-ended text may need to be replaced by voice capturing/recognition (to be discussed tomorrow)
• ... on mobile survey (non)response:– Industry perceptions and self-perception of potential mobile survey
participants on the reasons for nonresponse may be misleading– Most probable motivators: anticipated enjoyment, image– Boomerang effects for (over-)compensation– Fast responses, given in various contexts– ´Optimal length´ may not exist, various factors appear to
influence the willingness to spend time on (mobile) surveys
29
Thank you!
[email protected]://www.bosnjak.eu
Michael Bosnjak, PhD, Assoc. Prof. Free University of Bozen-Bolzano, School of Economics and Management
30
Appendix
31
Acceptance and users‘ behaviorInfluencing participants‘ behavior: design
Basic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile survey
yesyes nono
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
yes no yes no
in theSMS Group 1 Group 2 Group 3 Group 4
on the survey landing page
Group 5 Group 6 Group 7 Group 8
Ince
ntiv
e in
form
atio
n (ti
min
g)
Basic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile survey
yesyes nono
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
yes no yes no
in theSMS
Group 1
8,9%Group 2
17,3%Group 3
21,2%Group 4
12,3%
on the survey landing page
Group 5 Group 6 Group 7 Group 8
Acceptance and users‘ behaviorInfluencing participants‘ behavior
Ince
ntiv
e In
form
atio
n (T
imin
g)
Landing page
access
Groups not relevant, first contact on landing page
Acceptance and users‘ behaviorInfluencing participants‘ behavior
Reminder
Response rates maximized with price draw (group 3), additional compensation undermines motivation (see group 1: 1€ and price draw).
Response rates in wave 2 against time
Cum
ulat
ed re
spon
se ra
te
Hours since SMS invitation
SMS informationGroup 1 (1 EUR + price draw)Group 2 (1 EUR)Group 3 (prize draw)Group 4 (no incentive information)
Basic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile survey
yesyes nono
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
Prize draw(100 € Amazon voucher)
yes no yes no
in theSMS
Group 1
5,9%Group 2
12,8%Group 3
14,4%Group 4
9,2%
on the survey landing page
Group 5
9,8%Group 6
9,6%Group 7
9,1%Group 8
10,5%
Acceptance and users‘ behaviorInfluencing participants‘ behavior
Ince
ntiv
e In
form
atio
n (T
imin
g)
All questions answered
Nonresponse issues: Background
Why increasing response rates to surveys?
true difference nonresponse
error
nonresponse
rate
Black Boxyr ! = statistic of interest for respondentsyt ! = statistic of the total sampleynr ! = statistic of interest for nonrespondents 36
Nonresponse: Background:
Types of nonresponse
Source: Bosnjak (2001)37
Nonresponse: Background:
Generic reasons for nonresponse
• Failure to deliver the survey request• Spam guards
• Unused or infrequently checked e-mail addresses
• Non-availability during fielding period
• Inability to provide the requested data• Lack of knowledge
• Insufficient information readily available
• Noncompliance: Refusals to survey requests
38
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
39
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:Respondents are motivated by the monetary benefits promised/expected.
Actionable recommendations:„Pay respondents“ according to the time/effort invested
Caveats:• Peoples´ price points vary greatly and are unknown a-priori• May largely increase non-response bias •Undermines intrinsic motivation and may increase measurement error (low survey involvement)• Promised monetary incentives NOT consistently effective (!)
40
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:Some values are systematically related to the propensity to respond (higher order needs, civit duty orientation, etc.)
Caveats:• Effects small (if any)• Actionable recommendations?
41
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:Certain aspects of the survey announcement and survey implementation do induce compliant behavior:1. Reciprocity2. Scarcity3. Authority4. Consistency5. Consensus6. Liking
Actionable recommendations:Can be derived from persuasion literatures, but specific prescriptive models on how to tailor them toward survey situations are rare.
Groves, Cialdini & Couper (1992); Cialdini (2008);http://www.influenceatwork.com/
42
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:Larger response propensity if communication style reflects positive regard and avoids adult-to-child communication styles.
Caveats:• Limited scope• Empirical evidence scarce• Covered by other theories (compliance heuristics, social exchange)
Comley (2006)43
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:The propensity to respond to surveys is primarily a function of three factors:• Attitude to participate• Subjective norms• Perceived behavioral control • Moral obligation
Actionable recommendations:If weights are known for a specific population/sample: Enables the researcher to design survey participation requests
Caveats: Restricted to optimize survey announcements
Bosnjak (2002); Bosnjak, Tuten & Wittmann (2005)44
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale:Respondents are differentially motivated by • different aspects of the survey (leverage, e.g. type of incentives) and by • how much emphasis is put on each aspect by the surveyor (salience, e.g. preference for certain incentives )
Actionable recommendations:Because of the interaction between leverage*salience, improving response rates is not always desirable! Nonresponse bias may be influenced by leverage*salience interaction.
Groves, Singer & Corning (2000)45
Nonresponse: Theory: Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
Rationale: Survey participation as social exchange: The likelihood of responding is greater when the respondent trusts that the expected rewards will outweigh the anticipated costs of responding.
Actionable recommendations:Tailored Design Method, a well-developed set of practical recommendations on all aspects of survey design/implementation, aimed at:•establishing trust•increasing participation benefits•decreasing participation costs
Dillman, Smyth & Christian (2009)46
Nonresponse: Theory: TDM-based recommendations (selection)
To establish trust To increase benefits of participation
To decrease costs of participation
•Obtain sponsorship by legitimate authority•Provide a token of appreciation in advance•Make the task appear important•Ensure confidentiality and security of information
•Provide information about the survey•Ask for help or advice•Show positive regard•Say thank you•Support group values•Give tangible rewards•Make the questionnaire interesting•Provide social validation•Inform people that opportunities to respond are limited
•Make it convenient to respond•Avoid subordinate language•Make the questionnaire short and easy to complete•Minimize requests to obtain personal or sensitive information•Emphasize similarity to other requests or tasks to which a person has responded
Dillman, Smyth & Christian (2009, p. 38) 47
Nonresponse: Evidence: Mail surveys:
Effective methods & procedures I
• Most effective factors in mail surveys (only factors under the researchers full control listed):
• Personalization of requests to participate(Dillman, 1978, 2000; Edwards et al., 2007; Fox, Crask, & Kim, 1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991; Yu & Cooper, 1983)
• Prepaid monetary incentives(Church, 1993)
• Number of contacts made (esp. if prenotifier is included)(Armstrong & Lusk, 1987; Edwards et al., 2007; Fox et al., 1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991; Yu & Cooper, 1983)
➡ Integrated and refined within the Total-Design-Method (Dillman, Smyth & Christian, 2009)
48
Nonresponse: Evidence: Mail surveys:
Effective methods & procedures II
• Effective, but not covered because of limited control:
• Survey topic / topic involvement
• Length
• Sponsorship (University / commercial)
• Factors reducing response rates (1, 2: Edwards et al., 2007; 3: Singer, Hippler & Schwarz, 1992):
1. Starting with the most general question (e.g. demographics)
2. Opportunity to opt-out of the study
3. Over-emphasizing data protection/confidentiality
•Partly covered later for Web surveys:
• Questionnaire design effects on nonresponse
49
Nonresponse: Evidence: Web surveys:
Effective methods & procedures III
• Personalization:
• Personal salutation (name) is effective (esp. for powerful sender) (e.g., Heerwegh, et al., 2005; Joinson & Reips, 2007)
• (Monetary) Incentives:
• In general effective but small overall effect (Göritz, 2006)
• Pre-paid monetary incentives need to be tangible to be effective(Birnholtz et al., 2004; Bosnjak & Tuten, 2003)
• Lotteries esp. effective, timing important (immediate notification)(Bosnjak & Tuten, 2003; Tuten, Galesic & Bosnjak, 2004)
•Contact features:
• No of contacts very effective (Cook, Heath & Thompson, 2000)
• SMS prenotifier very effective (Bosnjak et al., 2008)
50