500 West 7th Street Cincinnati, OH 45203

burke.com phone|513.684.7663 fax|513.684.7633

2013 CASRO Online Research Conference: Burke, Inc. Submission for Presentation on Mobile Research Mobile Research Risk:

What Happens to Data Quality When Respondents Use a Mobile Device for a Survey Designed for a PC March 8, 2013 Background The growth in ownership of Internet-enabled mobile devices continues; 45% of US adults own a smartphone (Pew Research Center, December 2012), and 31% of US adults own a tablet or e-reader (Pew Research Center, January 2013). Accordingly, marketing researchers must determine whether and how to leverage these platforms to conduct research. Some researchers have benefitted from the near ubiquity of smartphone ownership within certain consumer populations of interest. Research use of mobile devices can involve directing participants to photograph products or product shelves, to respond to a handful of survey questions “in the moment” during shopping trips, or to use text messaging to capture impressions of customer experiences in real time. Such examples reflect intentional use of mobile devices for research purposes; researchers direct respondents to use the devices in particular ways for specific information needs. In contrast, mobile devices can be used by respondents to take surveys without the researcher directing them or expecting them to do so. This use of mobile devices for survey research is unintentional; the researcher does not necessarily intend for the survey to be taken on a small-screen device. Some respondents use their mobile devices to complete a survey in part because it is convenient. With declining survey response rates, researchers can no longer mandate that all surveys be completed using a personal computer, particularly for research involving client-supplied sample or for research involving otherwise hard to reach respondents. While intentional use of mobile devices for survey research involves testing and careful selection of the right technology for the research purpose, many researchers have yet to track, monitor, or plan for mobile data collection that is not the method intended by the researcher. This observation is empirically supported by survey results released in August 2012 indicating that 62% of the research companies surveyed either have no policy regarding respondent-driven use of mobile devices for surveys, or they permit the use of mobile devices but do not modify surveys for a mobile device (Macer and Wilson, 2012). This reality suggests the need for research uncovering the consequences of unintended use of mobile devices for data collection in survey research, especially when researchers have exerted no effort to optimize the survey for a device with a small screen. While Burke has published research demonstrating that personal and tablet computers generally produce comparable research results (Seal, 2012), analogous experimentation has not been done with smartphones. Accordingly, Burke conducted an experiment aimed at determining how data quality and other survey measures differ based upon a) which device is used to complete a survey, and b) whether steps were taken to optimize the survey for the data collection mode. The primary dependent variables to be compared across cells were the following:

Data quality measures such as dropout rates, fraudulence, mental cheating, straightlining, speeding, and missed survey item directives

Survey enjoyment, ease of responding, and likelihood to take another survey on the device used

Common marketing measures like category purchasing, brand awareness, and brand purchase

2

Research Design and Method This research utilized sample from a major online panel company. Qualifying respondents were willing and able to take the survey on all three of the following types of devices: a PC, a tablet computer, and a smartphone. Respondents were then randomly assigned to one of the following four research cells:

PC - Survey completion on a PC (247 respondents)

Tablet - Survey completion on tablet computer (249 respondents)

Smartphone adapted (SPA) - Survey completion on a smartphone; the survey was adapted for a small mobile device (251 respondents)

Smartphone un-adapted (SPUA) - Survey completion on a smartphone; the survey was not adapted for a small mobile device (242 respondents)

Examples of how survey items’ presentation differed by research cell appears in the Appendix. The online survey was about 10 minutes long. Results This research uncovered a number of difference and similarities between respondents in the four research cells. Respondent Profiles Despite random assignment of respondents to the research cells, some differences emerged in the profiles of respondents by cell (see Figure 1). Respondents taking the survey in the SPUA cell were slightly more likely to be female, and respondents in both smartphone cells tended to be younger than respondents in other cells. Figure 1: Demographic Variables

3

Examination of responses to early adopter survey items suggested no material differences in early product adoption and related behaviors across the research cells. Figure 2 shows agreement levels with 13 early adopter statements. Figure 2: Agreement with Early Adopter Items

Data Quality A primary area of focus for this research is to increase understanding of the effect of data collection device on data quality. Several data quality differences were found by research cell. Figure 3 displays the average survey duration by research cell; tablet and SPA respondents had the shortest survey times, while PC and SPUA respondents took considerably longer to complete the survey. Figure 3: Survey Length

4

Figure 4 shows that the device used for data collection has a significant impact on survey dropout rates, with nearly one in five respondents who started the survey in the SPUA cell not finishing the survey. That figure is about three and a half times the dropout rate among PC respondents. Figure 4: Survey Dropout Rate

Figure 5 shows further evidence of decreased data quality for respondents in the un-adapted smartphone cell; more than one in ten SPUA respondents failed to mark a 2 when directed to do so about 90% of the way through the survey. This failure rate is more than double the failure rate observed in the other three research cells. Figure 5: Failure to Mark a 2

5

Finally, the incidence of straightlining was measured within the block of 13 early adopter attributes located about 90% of the way into the survey. As Figure 6 shows, 5% of respondents in the un-adapted smartphone cell gave identical responses to all 13 items. Notably, three of the 13 items were negatively oriented toward innovation and early product adoption. In the smartphone adapted cell, only 1% of respondent straightlined, which is one third the incidence of straightlining found in the PC and tablet cells. This particularly low incidence of straightlining directly relates to the manner in which the multi-item grid was adapted for these respondents. Specifically, the grid was modified to increase the size of the text for the agreement items such that on a small mobile device, only one whole item’s text was visible at a time. (See the Appendix for some images of survey items as they were displayed for the difference cells.) Figure 6: Straightlining

Finally, the incidence of both failing to mark a 2 when directed and straightlining appears in Figure 7. The 4% incidence of both survey behaviors among SPUA respondents far exceeds the incidence found for the other three research cells. Figure 7: Failure to Mark a 2 and Straightlining

6

In summary, while PC and tablet respondents tend to have similar levels of data quality, quality can suffer among smartphone users taking a survey whose interface has not been adapted to accommodate a smaller screen. Quality of the Survey Experience In addition to data quality, another area of concern for researchers when unintentional mobile data collection occurs relates to respondent assessments of the quality of the survey experience. Such measures taken at the end of the survey give researchers a sense of whether the experience will positively or negatively impact a respondents’ propensity to take future surveys. With declining response rates industry wide, ensuring a favorable – or at least not an unfavorable – survey experience is important to the viability of the survey research industry. To assess the quality of the survey experience, respondents were first asked how favorable the survey-taking experience was. Figure 8 shows that the percent of respondents who deem the survey experience very favorable is greatest among PC respondents. The percent of respondents indicating the survey experience was very favorable is low and similar in the two smartphone cells. Figure 8: Survey Was “Very Favorable”

7

Similarly, the percent of respondents indicating that it was very easy to respond was highest among PC respondents. Nearly eight in ten PC respondents consider responding very easy, compared to slightly over half of tablet respondents. Among smartphone respondents, fewer than four in ten considered responding to the survey to be very easy. Figure 9: Responding Was “Very Easy”

Moreover, when asked how likely they would be to take another survey on the same device, two thirds of PC respondent indicate that they definitely would, compared to about half of tablet respondents. Thirty-two and 35% of respondents in the SPA and SPUA cells, respectively, definitely would take another survey on the same device. Figure 10: Definitely Would Take another Survey on the Same Device

8

These survey experience findings are corroborated by respondents’ preferred device on which to take a survey, shown in Figure 11. Even among respondents completing the survey on a tablet or taking either version of the smartphone, a majority indicated preference to take surveys on a PC. Not surprisingly, the percent of smartphone respondents preferring to take surveys on a smartphone is substantially higher among those who took an adapted rather than an un-adapted survey. Figure 11: Preferred Device for Surveys

Taken together, these findings evidence challenges associated with taking surveys on mobile devices, especially when the researcher dictates what device the respondent will use for the survey. It is important to note that respondents might judge the quality of the survey experience differently were they to choose to take a survey on a mobile device rather than being directed to do so as was done in this research. Category Purchasing Many marketing research surveys include questions about frequency of purchases in a particular category or the dollar amount spent in a category within a particular time frame. This research included category purchase questions about sports drinks and fast food or casual dining restaurants, as well as a question about the amount of respondents’ monthly cellular bill. Results for these survey items by research cell are shown in Figures 12, 13, and 14. For these product/service categories, no significant differences emerged based on the data collection device used for the survey. Figure 12: Frequency of Purchasing Sports Drinks

9

Figure 13: Frequency of Purchasing from Fast Food or Casual Dining Restaurants

Figure 14: Average Monthly Cellular Bill

10

Brand Awareness Another common set of marketing measures commonly included in surveys relate to brand awareness. In the current research, both unaided and aided awareness were measured for six brands in the sport drink, fast food, and cellular service categories, and results are shown in Figures 15, 16, and 17. As found with category purchasing, data collection device used for the survey does not significantly impact brand awareness for the brands studied. Figure 15: Brand Awareness for Gatorade and Powerade

Figure 16: Brand Awareness for McDonalds and Taco Bell

11

Figure 17: Brand Awareness for AT&T and Verizon

Conclusions Like any research, this study has limitations. They include the following:

Some non-response bias was introduced during screening via the requirement that respondents own and be willing to use any of three different devices to take the survey.

o Compared to younger panelists, older panelists are less likely to agree to take the survey on a mobile device.

As mentioned earlier, given that respondents were directed to take the survey on a particular type of device, less favorable ratings on items related to the survey experience could be affected, specifically through a phenomenon called psychological reactance.

o Psychological reactance is an aversive affective reaction in response to regulations or impositions that impinge on freedom and autonomy (Brehm, 1966, 1972, Brehm & Brehm, 1981; Wicklund, 1974). This reaction is especially common when individuals feel obliged to adopt a particular opinion or engage in a specific behavior.

Therefore, more favorable survey experience opinions might be found when respondents themselves choose the device used to complete a survey.

Findings could differ with other product categories or brands.

Finally, the affluence of respondents in this study might limit generalizability of the results.

12

Limitations notwithstanding, some conclusions from this research are as follows:

Data quality o In general, mental cheating is not rampant with a 10-minute survey. o Nevertheless, respondents in the SPUA cell tend to have higher dropout rates, greater

tendency to straightline, and more failures to mark a 2 than respondents in the SPA cell or in other cells.

o The higher dropout rates in the SPUA condition produce nonresponse bias, which slightly altered the demographic composition of the SPUA cell of respondents.

o Respondents taking the un-adapted survey on a smartphone (SPUA) took considerably longer to complete the survey, but survey duration in this cell is similar to that of respondents taking the survey on a PC.

Quality of the survey experience o The quality of the survey experience declines as the device gets smaller. o Survey enjoyment and ease of responding are markedly more favorable when the survey is

taken on a PC.

Category purchasing and brand awareness o Frequency of category purchasing does not differ by survey device for the categories

examined; nor does monthly cost of cellular service. o Neither unaided nor aided brand awareness differs by data collection device.

Implications Based on findings from this study, researchers should either direct known mobile survey takers to a survey adapted for a mobile device, or they should compel respondents to complete surveys on a personal computer. When making this decision, researchers should consider the following:

Survey length and complexity: For some projects with extensive or complex stimuli, e.g., many discrete choice studies, or for projects requiring long surveys, survey completion on a smartphone might be ill advised.

Sample source: When using client-supplied sample, especially if the sample elements are client customers, it might be advisable to offer a survey adapted for smartphones.

Sample scarcity and population size: The availability of an adapted smartphone version of the survey could increase participation rates when dealing with hard-to-reach or low incidence populations.

When adapting surveys for mobile respondents, researchers must be particularly vigilant in survey design in order to make responding less cumbersome on a mobile device. Researchers should...

Use fewer words in question introductions and question stems.

Ensure that questions and response options are visible.

Include a clear signal that a response has been registered. Importantly, pretesting on different device and software types needs to be part of the survey programming and quality checking process when mobile survey completion is permitted and likely. Smartphones in particular, as well as tablet computers enable an “always online” status for many people, and researchers must embrace the fact that respondents are using these devices for survey research, even when

13

such mobile device use is not mandated or driven by the researchers themselves. As such, the continued proliferation of technology for personal use requires that survey researchers adapt their expectations and methods to accommodate respondents’ device preferences.

14

Appendix Close-Ended Question Example

15

Appendix Attribute Grid Example

16

References Brehm, J. W. (1966). A theory of psychological reactance. New York, NY: Academic Press. Brehm, J. W. (1972). Responses to loss of freedom: A theory of psy- chological reactance. Morristown, NJ: General Learning Press. Brehm, S. S., & Brehm, J. W. (1981). Psychological reactance: A theory of freedom and control. New York: Academic Press. Macer, T. and Wilson, S. (2012). A report on the Confirmit Market Research Software Survey. Quirk’s Magazine, August. Pew Research Center. (2012). Internet and American Life Project: Smartphone Update for September. Seal, J. (2012). Tablet Data Collection: What Do You Need to Know? Paper presented at the 2012 CASRO Technology Conference in New York, NY, June. Wicklund, R. A. (1974). Freedom and reactance. Potomac, MD: Lawrence Erlbaum

17

Principal Authors and Presenters Jamie Baker-Prewitt, PhD Jeff Miller Senior Vice President, Director of Decision Sciences President and CEO Burke, Inc. Burke, Inc. 513.684.7663 513.684.7626 jamie.baker-prewitt@burke.com jeff.miller@burke.com Jamie is Senior Vice President, Director of Decision Sciences at Burke, Inc. Jamie’s experience as a research consultant spans a wide variety of industries and a broad range of business issues. Jamie is a seminar leader for the Burke Institute and a speaker at industry conferences, including events sponsored by CASRO, AMA, SPSS, IIR, and ASQ. Jamie has published papers in Quirk’s Marketing Research Review and the CASRO Journal. Jamie serves on Burke’s Board of Directors, Burke’s senior management committee, Burke’s R&D committee, and various other ad hoc management teams. She is a past president of the AMA Marketing Research Council, and she is Chair of the Advisory Board for the Masters of Science in Marketing Research program at the University of Texas at Arlington. Jamie is also a member of the American Psychological Association. Jamie started her marketing research career at Burke in 1992. She holds PhD and MS degrees in social psychology from the University of Kentucky and a BA in psychology from Ohio University.