survey informatics: the future of survey methodology and ...€¦ · slide 12 ae2 do you have a...

Survey Research and MethodologyUNL-GALLUP Research Center


Survey Informatics:The Future of Survey Methodologyand Survey Statistics Training in

the Academy?

Survey Informatics:The Future of Survey Methodologyand Survey Statistics Training in

the Academy?

Allan L. McCutcheon

AAPOR Annual Conference

16 May 2015

11



Acknowledgement and Disclaimer

• This material is based upon work supported by theNational Science Foundation under grant No. SES1132015.

• Any opinions, findings, and conclusions orrecommendations expressed in thismaterial/presentation are those of the author and do notnecessarily reflect the views of the National ScienceFoundation.



With Special Thanks to

• Mark Schulman (AbtSRBI)

• LeenKiat Soh (Co-PI)

• Adam Eck (Post-doctoral Fellow)

33



Avoiding Obsolescence!Can We Adapt and Evolve?

• Our fate? (Thank you to Mark Schulman)



What is Survey Informatics?

55

ComputerScience

Mathematics& Statistics

Cognitive & SocialPsychology

SurveyInformatics



The Emergence of Web Surveysand Online Panels

• Challenges of existing (i.e., phone) technology– Caller id (Callegaro, McCutcheon & Ludwig 2009)– Falling response rates (Pew Research Center 2012)

• Resistance to online surveys has been similar to theemergence of telephone surveying– Skepticism on the part of professional practitioners– Resistance by many with vested interest in older technologies

• Still, the profession eventually adopts the newtechnology– AAPOR Report on Online Panels (POQ, Winter, 2010)

66



RISE OF TECHNOLOGY INSURVEY METHODOLOGY

77

• Survey methodologists are turning to the use oftechnology to– gather higher quality data– increase throughput and overall quantity in data collection– improve analysis efficiency and effectiveness



Four Types of Data for Online Surveys

• Substantive data (survey responses)• Paradata (process data)

– Time of day survey taken– Data device (operating system, version)– Response latency– Back-ups, changes– Complexity (question, page)

• Metadata– Codebook– Data collection agency/firm– Field dates

• Auxiliary data– From external (e.g., commercial data bases)– E.g., neighborhood characteristics

88



These Lead to Complex Data Structures

99

AE4

AE4 Do we need to redact any of this? Maybe as Dr. McCutcheon?Adam Eck, 3/19/2014




1010

GovernmentAgencies

andBureaus

GovernmentAgencies

andBureaus

CommercialCompanies;

SurveyResearch

Companies

CommercialCompanies;

SurveyResearch

Companies

Academic andResearchInstitutions

Academic andResearchInstitutions

InformationCommunication

Web 2.0

Mobile

Big Data

Digital Sensing

Cloud ComputingCrowd Sourcing

Social Network

SURVEYMETHODOLOGY

SURVEYMETHODOLOGY




1111

• Survey research companies have increased use ofcomputer-based (including web-based) response optionsto– improve responsiveness and flexibility

• Administration, configurability, adaptivity– better collect, organize, and store respondent data– better track sessions to generate paradata– reach more respondents (increase response rates)

AE1

AE1 I changed this from "increase coverage" to "increase response rates" becasue it could actually be hurting coverage -- you lose all peoplewithout technology (an ever decreasing subset of the population)Adam Eck, 3/19/2014




1212

• Survey methodology researchers are also exploring theuse of advanced Web 2.0 and information-communication technologies to– better elicit information from respondents

• smartphones for data collection (in self-administered and interview-based surveys)

• interactive online surveys– more effectively and efficiently visualize respondent data– model and directly track user behaviors

• Such as to survey daily activities by keeping track of steps taken byrespondents

AE2

AE2 Do you have a reference for this last bullet point? Is this done by survey methodologists or CS people? And is this for monitoring(unobtrusive surveys?). That is my impression from the bullet point, but I'm not sure what you were going for. If so, I like the passivemonitoring aspect.Adam Eck, 3/19/2014



A Case for a Survey Methodology that IncludesComputer Science and Statistical Modeling

1313

• Survey methodologists are embracing technology offersopportunities to– Expand the reach of products from computer science (CS) to

improve how we perceive and act in the world around us– Develop a synergistic relationship with CS to jointly improve our

understanding of both fields and their practices• That is, as survey methodologists employ technological

advances in their operations, new problems will arisethat will also be of interest to CS theoretically andpractically to a wide range of domains



IMPROVING SURVEY METHODOLOGY THROUGHCOMPUTER SCIENCE

1414

http://www.ypte.org.uk/environmental/meat-free-mondays/209http://prblog.typepad.com/strategic_public_relation/2007/06/top-10-reasons-.html

Data AnalysisRespondentSupport &Adaptation

DataAugmentation

SystemDesign

Enhancement

AE3

AE3 I like this slide a lotAdam Eck, 3/19/2014




1515

• Data Analysis: Use CS techniques to handle moreeffectively and efficiently large scale of data (includeautomation, meta-tagging, and management)– Pattern recognition, machine learning, and data mining



Future of Survey Research

• Declining response rates– CATI response rates at 10%

• Caller ID• Declining landline• Increasing refusal• Cell phone limitations (manual dialing)

– Web surveys have low response rates• Difficult to get reasonable sampling frames online

• Promise of panels– Must still deal with respondent engagement– Ease of response– Mobility and timing

• “Personal assistants” (Avatars)– Test-to-speach– Natural language recognition (Siri, Cortana, Simon)– Digitized assistants (Nvidia Face Works Tech Demo)

1616




1717

• Example: how to store multiple levels of survey data?– Flat files lack flexibility (large redundancy)– Paradata: variable length, sequential data

• Different per respondent

• Approach: relational database (MySQL)– Easier to link relationships– More compact representation overall– Faster querying to retrieve data– Easier to maintain (add, remove, modify)– Allow for more versatile data analyses



Challenges Within the Academy

• Traditional departments are relatively siloed– Often resistant to cooperation without additional resources– Interest in new technology most typical among younger faculty

• i.e., least powerful faculty

• New ‘kids on the block’ are likely to be resource deprived– Need for visionary and entrepreneurial administrators

• Likely will require private-sector partnering

• In the midst of ‘paradigmatic shift’– Will require ability to change at least part of training relatively

quickly

• Persisting need for substantial portion of current surveymethodology curricula

• Students will need to learn more than their predecessors

1818



Conclusions

• Technology will play an increasing role in surveyresearch

• Statistics will play an increasing role in survey research– Analysis of data and paradata– Predictive modeling (CASD, predictive analytics)– Linkage algorithms for administrative, auxiliary, geo-data

• New forms of data and paradata, such as social media,will need to be explored

• There will be obstacles to overcome within the academy

• The challenges will make for an exciting future– All of us will have lots to learn in the years ahead

1919



Thank youThank you

Allan L. [email protected]

2020



Something New: Smartphones Take Over

2121



Device Type Trends in Gallup Panel

2222

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

Apr

-12

May

-12

Jun-

12

Jul-1

2

Aug

-12

Sep

-12

Oct

-12

Nov

-12

Dec

-12

Jan-

13

Feb-

13M

ar-1

3

Apr

-13

May

-13

Jun-

13

Jul-1

3

Aug

-13

Sep

-13

Oct

-13

Nov

-13

Dec

-13

Jan-

14

Feb-

14M

ar-1

4

Apr

-14

May

-14

Jun-

14

Jul-1

4

Aug

-14

Phone

Tablet



Passive Data Collection

2323

survey informatics: the future of survey methodology and ...€¦ · slide 12 ae2 do you have a...

Documents