the next challenge · 2010. 4. 16. · the next challenge efficient and effective mixed- and...
Post on 02-Oct-2020
3 Views
Preview:
TRANSCRIPT
The next challenge
Efficient and Effective Mixed- and multi-mode research
Tim Macer, meaning limited, London, UK
Presented at the Dutch Market Research AssociationAnnual Conference, Rotterdam, Netherlands6 & 7 November 2003
Agenda
1. The Rise of Multiple modes
2. The Issues
3. Technical framework
4. Survival guide
1. The Rise of Multiple Modes
OMR scanning
Face-to-face
Telephone
CATI
TCASI (IVR)
MCAPI
CAPI
CASI
OCR scanning
WAP
Evolution of today’s survey modes
TechnologyindependentTechnology based
Disk bymail
1975 1980 1985 1990 1995 2000Time line
CAWI
The rise of multiple modes
In USA, Web surveys are the undisputedreplacement for paper-based mail surveys*
Response rates falling
‘One size fits all’ model does not work ininternational research
Case studies showing that mixing modescanAchieve a better response
Remain scientifically valid
*Source: RS Owen in Quirk’s magazine, Feb 2002, p.24-26
What do we mean by multi-mode?
Multi-mode• Surveys utilizing more than one research
channel to reach different sub-samples, butconfining each sub-sample to one channel
Mixed modeSerial
• Surveys that involve successive interviewingstages, each utilizing a different mode
Parallel• Surveys that allows participants to choose
the mode and even to switch modes
LEVEL OF DIFFICULTY
Mixing Modes: some examples
Multi-country studies Web in USA
CATI in EU countries
CAPI or paper in India
Let respondent choose Contact by phone
Continue by phone or web
In parallel
In serial
The multi-mode bandwagon
0
1
2
3
4
5
6
Modessupported
Product choice (42 packages)Product choice (42 packages)
Source: Research Guide to Software 2003
Multi-mode: the challenge
“Survey organizations, whether they arein universities like mine, in private-sector organizations or in governmentorganizations, are going to have tochange dramatically in some ways inorder to do effective surveys as we bringthese new technologies online and stilluse our other technologies where theywork.”
Don Dillman, Washington State University
2. The issues
What are the problems?
How can these be resolved?
The three types of modal issues
CalibrationThe risk of differential measurement error due to
modal effect on the respondent
CoverageSampling issues—risk of differential non-response
from sub-samples for each mode
ComplexityDuplication of operational and programming effort in
addressing more than one mode
Increased cost, delays and errors from thisduplication
Calibration issues
Don DillmanTotal Design Method in 1978 to achieve consistency
between phone and mail surveys
Revised in 1999 to take into account Internet surveys
Examined response rate measurement differences inexperimental trials
Dillman’s conclusionsThere are observable and systematic differences
Disadvantages outweighed by overall improvement insample coverage, response, time and cost
Source: Dillman et al, paper at AAPOR Conference, Montreal, 2001
Source: Paper at ESOMAR Technovate, Cannes, 2003
Modal influence: calibration or coverage?
Oosterveld and WillemsAnother experimental research design mixed
CATI/Web surveys
Aimed to separate modal effect from populationeffect
Source: Paper at ESOMAR Technovate, Cannes, 2003
Modal influence: calibration or coverage?
Oosterveld and WillemsAnother experimental research design mixed
CATI/Web surveys
Research design separated modal effect frompopulation effect
Their conclusionsThe majority of differences reported in previous
studies between Web and paper can be explained bypopulation difference, not intrinsic modal effects
Mixed mode studies can be designed to have noinfluence on the answers
Source: Quirk’s magazine, July/Aug 2002, p20
Mode switching to improve coverage
Allison & O’KonisMixed Web/CATI survey of online financial services
Initial approach by CATI or Web with option to switch
88% of CATI respondents agreed to a continue theirinterview on the web
54% of them went on to complete
Different modes gave highly similar responses
Their conclusionsSwitching modes does increase response rate
But, provided that the switch is done immediately:tomorrow is too late
Modal influences observed
Presentational influencesGanassali and Moscarola have measured increased
responses when relevant visual clues presented inweb interviews
Modal influences observed
The moderating effect of the interviewer Noted by Poynter and Comely amongst others
Can lead to under-reporting, especially of sociallyunacceptable responses
After: Poynter & Comely, Beyond Online Panels, ESOMAR Technovate 2003
With interviewerOnline
Using a mobile phone whilst driving: claimed level of usage
RarelySometimesOften
Open-ended responses
Oosterveld and WillemsObserved longer and more detailed verbatim
response on the web than phone
Allison and O’KonisObserved great similarity for for phone and web
However, population was one with high internetpenetration
Noted some content differences e.g on‘technographic’ subjects which they attributed topopulation effect
Scale questions
Humphrey Taylor (2000) Observed a tendency for respondents to answer scale
questions differently on the web
Dillman et al (2001) Characterised differences between CATI and CAWI on
anchored scale questions (1=strongly agree etc)
CATI respondents favors the extremes
CAWI significantly more likely to use the entire scale
Bäckström and Nilsson (2003) Observed the same tendency between self completion on
paper and web
More research required
Differences in ‘don’t knows’
HoggMore answers recorded as ‘Don’t know’ or
‘No answer’ in Web surveys than same surveywhen interviewer-led in CATI
Recommends omitting explicit DK/NA categoriesin version displayed on the Internet
Source: Quirk’s magazine, July/Aug 2002, p90
Population effects
Non-response (non-participation)Don Dillman and others observed greater tendency
for males not to participate in CATI and females inWeb surveys
Population effects are also influential in…Open-ended responses
Rating scales
Possibly more (Oosterveld & Willems)
Operational complexity issues
Different recruitment and screeningCan’t always approach by same mode
Duplication of the survey instrumentComplete duplication of effort may be required
Problems managing multiple versions
Data HandlingNeed data in one place in one format
Problems mixing online and offline modes
Mode switchingMust be fast if response rate to be improved
Mode-appropriate texts
3. Technical framework
How should technology be supporting mixedmode research?
What are the software developers doing toprovide this support?
1. Common survey authoring tool across all modes
2. Independence of design and execution
3. Mode specific texts (not through foreign languages)
4. One common, central database for all modes
5. Auto-determine contact mode from sample
6. Efficient mode switching
7. Concealment of previous data when switching to self-com.
8. Reminders and auto-revert to previous mode
9. Single view management & reporting tools across allmodes
10. Quotas that operate across all modes
11. Question constructs that recognise different modes
12. Recording of mode at datum not case level
Framework for the ideal MM system
Suppliers contacted
Askia Askia
Mercator snap
MI Pro MI Pro Research Studio
Nebu Dub Interviewer
Opinion One CAVI
Pulse Train Bellview Fusion
Sphinx Sphinx
SPSS MR Dimensions
Who supports what?
Full
Full
Full
Part
Part
SPSSMR
FullSoon
FullFullPartPaper
FullFullFullFullFullFullFullCAWI
FullPartFullSoonFullFullFullCAPI
FullFullPartFullFullFullFullCATIlight
FullFullFullCATI
SphinxPulseTrainCAVINebuMI ProsnapAskia
The issues—according to the developers
7
0
23
0
5
10
15
20
25
Calibration
Coverage
Complexity
Citations
4
3
6
11
0
0 5 10 15
Complexity
Sampling, screeningDuplicationData handlingMode switchingOthers
Innovation: Calibration issues
Reduction of modal influenceOpinion One CAVI
• Totally consistent appearance for Web, CASI &CAPI
• Novel method for unaided questions in self-completion modes
Sphinx
• Experimental approach
Measurement of modal differencesPulse Train
• collect paradata on mode for each question
Innovation: Complexity issues
Modal independent designSPSS MR
• Modal “players”
Askia, MI Pro, Pulse Train, Nebu, SPSS MR
• Modal templates applied to same surveyinstrument
Central databaseAll apart from snap
Wizards for importing offline data in Askia
Innovation: Complexity issues
Mode switchingHandled well in Askia, Pulse Train, Nebu and Opinion
One
Email despatched automatically in Opinion One
Nebu recognises ‘static’ and ‘dynamic’ swaps
Call me button in Pulse Train linked to dialler
Recall of interviews into CATI mode in Askia, Nebu,Pulse Train
Switching in and out of paper in MI Pro
Missing features
Ability to cross-tab data by mode at adatum level
Support for systematic removal of answersfrom modes, i.e. Don’t Know and Not Stated from self-completion
Up-stream sample management
Support to simplify parallel screening
Developers need to focus more on thecalibration and coverage issues!
4. Mixed mode survival guide
Metadata standards can help
MR slow to embrace standards to alloweasy data transfer from system to system
Most focus on the interchange of collecteddata, not survey instruments
Standards allows the metadata to betransferred along with the data
Examples of metadata include:Question typeUnique question nameQuestion texts and answer texts/codesPermitted ranges of valuesRouting or filtering context
Triple-s
www.triple-s.org
First published 1994
Originated in the UK but now implementedby 30 vendors worldwide
Exchange data and metadata via exportsand imports in a generalized formatVersion 1.1 introduced XML support
New version 1.2 adds filters, weighting and multi-language support
No metadata support for survey filtering orrouting logic
SPSS Dimensions Data Model
A new open (though proprietary)metadata model for survey data
Can be licensed independently of all SPSS MRproducts (don’t have to use SPSS software)
Comes with a developers’ library of tools forbuilding applications that will read or write data viathe SPSS Data Model
Many other software companies now providingsupport for the SPSS Data Model
Metadata for survey data not survey routing andlogic
QEDML
www.philology.com.au
New multi-platformsurvey authoring tool
Exports scriptinglanguages for severalpackages, includingQuancept, Surveycraftand In2form
XML based open system,allows other languagetranslators to be added
Tips for multi-mode survey design
Design your survey to be as mode neutralas possible
Pay attention to rating scales
Consider exclusion of Don’t know/Notstated answers on self-completion modes
Ensure you can identify the mode whenanalysing your data, at each question
Standardise on the software, or at least,the data format
In summary
Modal differences do exist, but can beovercome with careful design
Issues relate to: Calibration, Coverage andComplexity
Common survey authoring and a commonresults database improve MM efficiency
Software manufacturers are largelyfocusing resolving complexity issues
Better standards, especially for surveyinstrument metadata, are needed
BibliographyAllison J & O’Konis C (2002) If Given the Choice, Quirk’s Marketing Research Review,
July/August issue, p 20.
Bäckström, C & Nilsson, C (2002) Mixed mode: Handling method differences betweenpaper and web questionnaires,http://gathering.itm.mh.se/modsurvey/pdf/MixedMode-MethodDiff.pdf
Dillman D A (1978) Mail and Telephone Surveys: The Total Design Method, Wiley
Dillman D A, Phelps G, Tortora R, Swift K, Kohrell J & Berck J (2001) ResponseRate Measurement Differences in Mixed Mode Surveys Using Mail, Telephone,Interactive Voice Response and the Internet, AAPOR Annual Conference, Montreal
Ganassali S & Moscarola J (2002) Protocoles d’enquête et efficacité des sondages parInternet, Journées E-Marketing AFM/AIM Conference, Nantes, France
Macer, T (2003) Research Software Review, The Market Research Society, London.
Oosterveld, P & Williams P (2003) Two Modalities, One Answer. ESOMAR TechnovateConference, Cannes.
Owen R S (2002) A Matter of Trade-offs: Examining the advantages and disadvantagesof online surveys, Quirk.s Marketing Research Review, February, pp 24-26.
Poynter R and Comely P (2003) Beyond Online Panels. ESOMAR TechnovateConference, Cannes
Taylor H (2000) Does Internet Research Work? Comparing online survey results withtelephone survey, International Journal of the Market Research Society, 42.1
www.meaning.uk.com
top related