voice recognition in self-administered mobile surveys' - free university of bozen-bolzano and...
DESCRIPTION
Text versus voice? A number of approaches have been suggested to optimise mobile survey response rates. Whilst addressing survey design can help for closed questions, what about open-ended ones? The presenters will share the results of an experimental study comparing classical text-input formats in self-administered mobile surveys with an alternative, voice-recognition-based procedure in mobile survey projects. They will report on the relative differences of these two formats on usability and non-response levels, and will provide practical recommendations for mobile survey projects.TRANSCRIPT
Globalpark. Manage what matters.April 19, 2011 Mobile Study III1
VOICE RECOGNITION IN SELF-ADMINISTERED MOBILE SURVEYS
LORENZ GRÄF, SVEN SCHERRER, MARC HAUPT, MICHAEL BOSNJAK, MARIO CALLEGARO
Globalpark. Manage what matters.April 19, 2011 Mobile Study III2
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
Globalpark. Manage what matters.April 19, 2011 Mobile Study III3
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
Globalpark. Manage what matters.April 19, 2011 Mobile Study III
USABILITY OF MOBILE SURVEYS: PAST FINDINGS
4
Einfachauswahl untereinander
Mehrfachauswahl untereinander
Geschlossene Auswahlliste
Textfeld einzeilig
Fragetyp mit Bild
65,00 73,75 82,50 91,25 100,00
87,9
74,7
82,7
87,3
89,2
Usability score (Range: 0-100 Punkte)
Frag
etyp
Subjective Usability AssessmentPost-hoc survey (Web) one week after mobile survey completion
Indicators for usability score: fluency, simplicity, ease of use
Observed
Item-NR
Drop-Out
45%
9%
9%
23%
Multiple choice
Single choice
Drop-Down menu
Textfield
Image map
Globalpark. Manage what matters.April 19, 2011 Mobile Study III5
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
Globalpark. Manage what matters.April 19, 2011 Mobile Study III6
OVERALL GOAL Improving mobile input formats requiring rich, non-standardized information, which is usually captured using text input in other self-administered survey modes. Alternative to (typed) text-input formats:
Embedding voice input into self-administered mobile surveys, similar to CATI or esp. IVR surveys
Today: Subjective usability assessments as indicators of a potential improvement text versus voice Next step: Analyzing objective / unobtrusive data on nonresponse (item-nonresponse, drop-out)
Globalpark. Manage what matters.April 19, 2011 Mobile Study III7
TECHNICAL IMPLEMENTATION: IPHONE APP
Globalpark. Manage what matters.April 19, 2011 Mobile Study III8
TECHNICAL IMPLEMENTATION: ANDROID
Globalpark. Manage what matters.April 19, 2011 Mobile Study III9
RESEARCH QUESTION(S) TESTEDDifferences between
iPhone-App-based voice capturingversusAndroid-based voice recognitionversus
Classical (touchscreen-based) text inputon the following dependent variables:
usability assessment immediately following a specific voice/text input format, (objective indicators of nonresponse, esp. item-nonresponse and drop-out,)
retrospective usability assessement Ø 5 days following participation?
Globalpark. Manage what matters.April 19, 2011 Mobile Study III10
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
11 Globalpark. Manage what matters.
MINGLE PANEL – MOBILE USERS
04/11/11 Mobile Study III
N=10518
12 Globalpark. Manage what matters.
MINGLE PANEL – SMARTPHONE OS
04/11/11 Mobile Study III
N=3136
Globalpark. Manage what matters.April 19, 2011 Mobile Study III13
PARTICIPANTS
Globalpark. Manage what matters.April 19, 2011 Mobile Study III14
PARTICIPANTS
79,2 %
14,2 %6,6 %Sales
EmployedStudentUnemployed
Employment Status
Globalpark. Manage what matters.April 19, 2011 Mobile Study III15
PROCEDUREPart 1: Pre-Survey (23.03. – 28.03.2011, 6 Days, time of invitation: 5 PM GMT)• Device Screening
• Android: Technical requirements check
• iPhone: Instructions App installation
• Pre-commitment
Part 2: Main survey (28.03. – 03.04.2011, 7 Days, time of invitation: 5 PM GMT)
• Test drive: Voice recording / voice recognition
• Three text/voice input items (short: single number, medium: list three items, long answers: elaborated text/answer)
• Usability assessment following each text/voice input item
Part 3: Post survey (05.04. – 09.04.2011, 5 days, time of invitation: 8 AM GMT)
• Retrospective usability assessment
• Evaluation of Main-Survey Drop-Out• Preferred input method short / long entries
Globalpark. Manage what matters.April 19, 2011 Mobile Study III
PROCEDURE (PARTS 2 AND 3)
16
Respondi panel members (N= 235)
iPhone App Voice Capturing Group
(n1.1= 126)
Android Voice Recognition Group
(n2.1= 60)
Android Text Input Group (n3.1= 49)
Immediate usability assessment for three corresponding questions.
(n1.2= 87) (n2.2= 36) (n3.2= 38)
Retrospective usability assessment (after Ø 5 days, Web-based).
(n1.3= 79) (n2.3= 36) (n3.3= 34)
Globalpark. Manage what matters.April 19, 2011 Mobile Study III
DEPENDENT MEASURESImmediate usability assessment: Mean score (transformed to 0 -100 range) of three identically worded, closed-ended single items evaluating the extent to which (iPhone) voice recording / (Android VR) voice recognition / (Android text) text input was appropriate to capture the intended response („How do you evaluate the performance of the voice- recognition/-capturing /touchscreen keyboard feature?“, scaled from ´very good´ thru ´very poor´).
Retrospective usability assessment (reminder screenshots were displayed):Mean score (transformed to 0 - 100 range) of the following four indicators: ease of use, fluency, usability, unintended entries .
Non-response (work in progress):
Item-nonresponse: Aggregated item-nonresponse percentage for all three questions with experimentally manipulated input formats (per experimental condition)
Drop-Out: Overall drop-out rates (per experimental condition)
17
Globalpark. Manage what matters.April 19, 2011 Mobile Study III18
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
Globalpark. Manage what matters.April 19, 2011 Mobile Study III19
USABILITY ASSESSMENT; IMMEDIATE
Immediate usability assessment best and most homogeneous for Android-based
text input (!)
Globalpark. Manage what matters.April 19, 2011 Mobile Study III20
USABILITY ASSESSMENT: RETROSPECT
Retrospective usability assessment best for
Globalpark´s iPhone-App-based voice capturing
technology.
Globalpark. Manage what matters.April 19, 2011 Mobile Study III
MISC: INPUT FORMAT PREFERENCES
21
Physical KeyboardVoice Capt/RecTouchscreenDon't Know
Short Entry Long Entry0 %
50 %
100 %
Short Entry Long Entry
57 %76 %
9 %
9 % 33 %13 %
iPhone App Android
Globalpark. Manage what matters.April 19, 2011 Mobile Study III22
SUMMARY OF RESULTSImmediate usability assessment:
Android touchscreen > better than > iPhone App VC > better than > Android VR
Android touchscreen ratings most homogeneous (largest agreements)
Retrospective usability assessment:iPhone App VC > better than > Android touchscreen > better than > Android VR
Android VR shows largest dispersion among ratings (largest disagreement)
Additional result:The larger the amount of text information to be entered, the more preferred are voice input formats on the expense of the touchscreen-based version, but only for iPhone participants (not for Android participants).
Globalpark. Manage what matters.April 19, 2011 Mobile Study III23
AGENDA
1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook
Globalpark. Manage what matters.April 19, 2011 Mobile Study III24
DISCUSSION AND OUTLOOKFurther analyses using data from the current study:
Item-Nonresponse analysis
Drop-Out analysis (partly done, see Appendix)
Quality of content between groups?
Technology and/or selection effects? For instance,iPhone users are more experienced with Apps and more ´attached´ to their Smartphone, ´causing´ more favourable post-hoc evaluations of App-based input formats;men versus women systematically overrepresented in certain groups, other systematic differences?
Next steps:
Results consistent over time and in the context of ´real life´ applications?
Please feel free to use VC/VR in your own studies!
Globalpark. Manage what matters.April 19, 2011 Mobile Study III25
THANK YOU!
Globalpark. Manage what matters.April 19, 2011 Mobile Study III26
Appendix
Globalpark. Manage what matters.April 19, 2011 Mobile Study III27
DROP-OUT (WORK IN PROGRESS)Voice
CapturingVoice
RecognitionRegular
Test Drive Instructions
9 (9,3%) 5 (10,4%) -
Test Drive 1 (1,0%) 0 (0%) 0 (0%)
Single figure 0 (0%) 0 (0%) 0 (0%)
List 0 (0%) 0 (0%) 0 (0%)
Free text 0 (0%) 0 (0%) 0 (0%)Drop Out on other pages
0 (0%) 7 (14,6%) 2 (5%)
Drop Out Overall 10 (10,3%) 12 (25%) 2 (5%)
N= 97 (100%) 48 (100%) 40 (100%)
28 Globalpark. Manage what matters.
Comments and Data Samples
04/11/11 Mobile Study III
Comments Voicecapturing„Da ich nicht sprechen konnte, konnte ich den Fragebogen nicht zu Ende durchgehen.“
„Ich war krank, böse Erkältung und konnte mich nicht motivieren, zumal mir das Sprechen superschwer fiel.“ Comments Voicerecognition
„Spracherkennung ist bei meinem Handy für solche Felder nicht verfügbar, nur für die Google-Suche“
„Ich hatte nur die englische Spracheingabe. Habe im Internet nach Hilfestellungen gesucht um auf deutsche Spracheingabe umzustellen, allerdings nichts gefunden.“
„Spracheingabe funktionierte nicht richtig. Spracheingabe ging nur als englischer Text. Alles Einstellungen meines HTC waren auf Deutsch und ich habe keine Möglichkeit gefunden die Spracheingabe auf Deutsch zu berichtigen.“
Comments Touchscreen Input
„Gute Erhebungstechnik, gerne wieder über smartphone...“
„Würde mich sehr freuen weitere Umfragen mit dem Handy beantworten zu dürfen.“ Data Samples Voicerecognition – Testphrase „Diese Befragung ist sehr interessant“
„Die herbe frau geküsst interessant“
“These are The 5 sense sons”
“disappear.com sia into the sun”
„Die befragung sea of sand“
„wiese befragung ist sehr interessant“
29 Globalpark. Manage what matters.
Demo - Audio Samples iPhone
04/11/11 Mobile Study III
Apple Quicktime required
0: Testtrial
mingle61 (optimal)
0: Single Word
mingle 99 (Optimal)
mingle 166 (Background noise)
mingle 173 (Additional info)
2: List
mingle31 / 19 (optimal)
mingle16 (Background noise)
mingle8 (Real List)
3: Open answer
mingle198 (Slight saxon dialect)
mingle187 (Whole sentence)
mingle192 (slightly annoyed)