Slide 1

Survey Nonresponse: A Decision- Making Approach Roger Tourangeau Joint Program in Survey Methodology, University of Maryland Survey Research Center, University of Michigan Slide 2 1 Outline Falling Response Rates Many surveys taking countermeasures Costs are rising, but response rates still falling Whats Behind It? Variety of theories Lack of civic engagement; value of polls and related activities Decrease in discretionary time Fending off unwanted intrusions now routinized At the individual level, how do people decide? The Salience-leverage model Belief-sampling as a model for quick judgments The heuristics approach Slide 3 2 Outline (Contd) Does the decline matter? The relation between nonresponse rates and nonresponse error The impact on surveys Rising use of incentives: Is there a cost? Surveys as opportunity/obligation vs. survey as transaction Many thanks to Bob Groves, from whom I stole many slides (and many ideas)!! Slide 4 Falling Response Rates Slide 5 4 What is Nonresponse? Unit nonresponse is the failure to obtain survey measures on a sample unit It occurs after the sampling step of survey It reflects total failure to obtain survey data (wont talk about item nonresponse, the failure to obtain an answer to a given item) Slide 6 5 Total Nonresponse and Refusal Rates Increasing over Time for BLS Consumer Expenditure Survey Census Bureau Slide 7 6 Until Recently, Current Population Surveys Rates Had Been Stable Census Bureau Slide 8 7 National Health Interview Survey Nonresponse Trends Slide 9 8 Also Apparent for Telephone Surveys: Survey of Consumer Attitudes Thanks to Rich Curtin for the data Slide 10 9 Multi-Country Studies of Cooperation (de Leeuw & de Heer, 2002) 16 countries (Western Europe and U.S.) As many as 10 ongoing surveys (mostly central government sponsored), mid-1980s-late 1990s Labor force Consumer expenditure Health Travel On average, 3 percentage point decline per year in cooperation rate Slide 11 10 Expected Proportion Noncontacted and Refused Under deLeeuw & deHeer Model by Year Slide 12 11 Impact on Costs per Case Costs have risen as surveys take countermeasures More extensive use of advance letters (especially in telephone surveys where they werent used before) More extensive use of incentives (revisit issue of cost impact) More callbacks Steeh et al. (2001) present evidence that Michigans Survey of Consumer Attitudes used to take around 6 calls per complete (mid-1990s) By 1999, it took 12 calls on average Seems clear in U.S. that survey costs rising far faster than inflation Slide 13 Theories of Response and Nonresponse: The Sociology of Nonresponse Slide 14 13 Three Forms of Nonresponse Noncontact Noncooperation Inability Unfavorable societal developments on all three fronts Slide 15 14 Noncontact: Face-to-Face Surveys Rise of doormen buildings, locked condos, and gated communities More than eight million American now live in gated communities and nearly 40 percent of newly built residential developments are gated (Blakely and Snyder, 1997) New residential arrangements featuring gatekeepers (assisted living, nursing homes, etc) Slide 16 15 Noncontact: Telephone Surveys Rise of answering machines, Caller-ID, cell phones By 1995, most U.S. households had answering machines and roughly 40 percent reported they used them to screen their calls (Tuckel and ONeill, 1995) By 1996, about 10 percent of all households nationally had Caller-ID. Slide 17 16 Inability To Provide Data Reflects both physical/mental limitations and language barriers Rising proportion of the population is 65 or older Concomitant increase in hearing problems, other disabilities Increase in immigrant populations 2002: 11.5 percent of the U.S. population was foreign-born According to Long Form data from Census 2000, 8.1 percent of the population over age five reported that they speak English less than very well. Many surveys now field both Spanish and English questionnaires, but only two-thirds of those who are less than completely fluent in English are Spanish speakers. Slide 18 17 General Theories Regarding (Non)- Cooperation Still, the big problem is non-cooperation Some theories are couched in terms of societal trends, others based on person-level characteristics Nonetheless, although level of analysis is different, the ultimate causal mechanisms in these theories are similar Three accounts widely cited People are too busy People are too self-absorbed People are erecting barriers to unwanted intrusions Both noncontact and non-cooperation may be the result Slide 19 18 Too Busy More people are labor force participants (e.g., 66.0% of all civilians, 16+ in U.S. were in the labor force in 2004 vs. 60.2% in 1970) The change is particularly dramatic for women (from whom respondents are disproportionately drawn): 59.2% of all women 16+ in the U.S. were in labor force in 2004 vs. 43.3% in 1970 60% of women who worked at all during 2003 were full- time (vs. 41% in 1970) Societal trend with individual-level impact: Opportunity costs of survey participation too high Slide 20 19 But Are People Really any Busier? Fewer adults are married than 25 years ago; also, fewer are parents More and more people are retired and they are retiring at younger ages; according to Robinson and Godbey, Americans aged 55-64 gained an average of ~10 hours of free time per week since 1965 Again, according to Robinson and Godbeys time diary studies, Americans have gained about 5 hours of free time per week on average since the 1960s Nonetheless, people feel busier, in part because of relentless multitasking Based on their perception, they may be more reluctant to give up free time Slide 21 20 Too Self-Absorbed Many survey researchers subscribe to one version or other of the social capital hypothesis Response rates falling for the same reason as declines in voting, other forms of civic participation; people feel less obligated, less interested in helping others Slide 22 21 Groves, Singer, and Corning (2000) Groves, Singer, and Corning (2000) assessed community participation in face-to-face survey: five items on joining organization to solve some community problem, writing to officials, doing volunteer work, etc. Civic duty: A feeling of obligation to provide help in the belief that the common good is thereby served. Apparently independent mail survey request of those completing a face to face survey, with reported community involvement attributes $5 incentive experiment, prepaid Slide 23 22 Results Overall, about 15% difference in response rates: 58.0% (262) vs. 43.1% (116); even bigger diff. with no incentive Slide 24 23 Implications Could explain why election polls, which traditionally get low response rates, nonetheless generally give accurate results Those most likely to vote overrepresented in polls; both surveys and elections overrepresent those high in involvement, social capital Slide 25 24 Too Many Unwanted Intrusions Modern life often seems to consist of continuous bombardment with unwanted information, intrusions via every medium Junk mail Telemarketing Spam Panhandling in big cities In response, people take countermeasures to limit access Spam filters Do Not Call lists, Caller-ID, answering machines Crackdown on panhandling in NYC and elsewhere Gated communities, locked apartment buildings, etc. Slide 26 25 Unwanted IntrusionsII May reflect diminished community involvement, sense of busyness Whatever the cause, contactability and willingness to cooperate may not be distinct phenomena but reflect effects to fend off unwanted contacts Slide 27 Theories of Response and Nonresponse: The Psychology of Nonresponse Slide 28 27 Leverage-Salience Theory of Survey Cooperation How do these societal trends play out at the individual level? Leverage-salience offers one important account Persons vary in the magnitude and direction (positive and negative) of influence of various psychological predispositions toward survey participation in general and toward various design features (topic, sponsor): Leverage The information about the survey request processed by the person varies due to interviewer variation in introductory scripts and their cognitive associations with information provided (Salience) Slide 29 28 Burden Incentive Authority of Sponsor Topic Person 1 Person 2 Authority of Sponsor Slide 30 29 Burden Incentive Authority of Sponsor Topic Person 1 Person 2 Slide 31 30 Implication of Leverage-Salience Theory for Nonresponse Bias People make decisions to cooperate or refuse on different bases If the salience of design attributes or focus of interviewer behavior systematically varies over contacts, then decisions can be based on different weightings of attributes over contacts Bias results when same survey attribute related to survey variable and survey participation decision (common cause model) Slide 32 31 Groves, Presser, and Dipko I Sample from five frames (four list samples plus RDD) teachers, parents of children under 6 months, people 65+, contributors to fringe candidates Two IVs: Topic (intro. Mentions topic twice) and letter plus incentive (half get letter plus $5) Done by Maryland SRC plus 12 Practicum students Response rate ~63.0 Slide 33 32 Groves, Presser, and Dipko II Key result: Outcome of first contact in which topic mentioned Incentives diminish topic effects for teachers and seniors, but increase them for parents Slide 34 33 The Belief-Sampling Model Model of attitude judgments made on the fly; four key components; salience-leverage a special case 1.Determine the issue (the pool of beliefs, values, impressions, existing judgments) from which sample will be drawn 2.Sample some of these considerations (i.e., think about the issue)probability of retrieving any given consideration related to its accessibility (salience may determine accessibility) 3.Scale (leverage) and integrate Anchor-and-adjustment Average/weighted average 4.Map onto response scale Slide 35 34 Formal model Integration Phase Reliability over two occasions n1: Number of considerations sampled at time 1 n2: Number of considerations sampled at time 2 1: Consistency in assigning scale values (scaling consistency) 2: Homogeniety in pool of considerations (homogeniety) q: Overlap between samples, expressed as proportion of n2 (overlap) Slide 36 35 An Alternative: Rules/Policies for Dealing with Unwanted Intrusions Both leverage-salience and belief- sampling assume that people make a decision at the time survey request is made Maybe their past decisions have hardened into a policya set of unconscious/semi-conscious procedures for dealing with unwanted intrusions Slide 37 36 Example Policies Throw out all mail, unless Its clearly a bill Its clearly a personal letter Its from a familiar organization to which you are sympathetic Hang up/dont answer telephone, unless Its a personal call from someone you know Its from a familiar organization to which you are sympathetic Its a business call that you are expecting Slide 38 37 Implications for Nonresponse Bias Rules could vary by subgroup, changing composition of samples in predictable ways Consistent with studies of telephone response, which indicate nonresponse happens very quickly Important to discover rules and learn 1.How to avoid quick judgment (by making policy seem inapplicable; Im not selling anything) 2.How to get people to suspend rule; people do make exceptions to policies or alter them Advance letters may help (moving call into different categoryexpected business call) Slide 39 Does Nonresponse Matter? Empirical Estimates of the Impact of Nonresponse Slide 40 39 Nonresponse Bias Classic formula for bias in uncorrected mean: Slide 41 40 Nonresponse Bias II Assumes nonresponse deterministic; two types of people Those who never respond (W N ) Those who always respond (1- W N ) Sampling error in mix, so nonresponse rate (and error) is not fixed Alternative: People have a response propensity (probability that theyll respond); Response propensity: Probability that theyll be contacted, agree to cooperate, etc. As a result, even if same people in the sample, outcome might be different Slide 42 41 Nonresponse Bias III Bias now depends on correlation between p (response propensity) and y (substantive variable) Slide 43 42 Empirical Estimates of Bias Three papers, taken together, suggest that as an empirical matter, nonresponse may not be that big of a problem Keeter et al. compare two surveys that vary in response rates Merkle and Edelman examine within-precinct error as a function of precinct response rate Curtin et al.: Look at what would have been result if SCA had stopped interviewing sooner Slide 44 43 Keeter, Miller, Kohut, Groves, & Presser Compared two surveysstandard vs. rigorouswith same questionnaire but different field procedures designed to yield different response rates Slide 45 44 Keeter et al.II Big weaknessconfounds many variables (R rule, advance letter, race and experience of interviewers, etc.) Big strengths Examine 91 items from a variety of domains Big differences in response rates 36.0% for standard vs. 60.6% for rigorous Mostly due to differences in contact rates (68.5% vs. 92.0%) rather than cooperation rates (58.1% vs. 73.7% cooperation) Slide 46 45 Keeter et al.III Key result: Differences in estimates 14/91 significant Mean over 91 items around 2% (Is this big or small? What if this were unemployment rate?) Largest difference (9 percent) involves interviewer rating of R interest (rigorous adds less interested cases) Other findings Standard survey doesnt seem to underrepresent Republicans or conservatives relative to rigorous Reluctant Rs (those classified as refusals) have higher item nonresponse rates Slide 47 46 Merkle and Edelman I Examine relation between within precinct error (relative to actual vote) and precinct response rate Strength: Know the truth and can a direct measure of bias Weakness: Purely correlational Key results: Little relation between nonresponse and error Slide 48 47 Merkle and Edelman II Slide 49 48 Curtin, Presser, and Singer Simulate effect of changing procedures in Survey of Consumer Attitudes (e.g., reducing number of callbacks) How would estimates change if we drop cases interviewed after call x? Converted cases? Percent Significant Differences Slide 50 49 Nonresponse Rates and Nonresponse Bias Overall picture: Little nonresponse bias Within a certain range (~25 to ~65%) Not looking for small effects (change in percentage of 0.2%) Two papers lead to different conclusions Groves, Presser, & Dipko (2004) Teitler, Reichman, & Sprachman (2003) Slide 51 50 Groves, Presser, and Dipko Samples teachers, parents of children under 6 months, people 65+, contributors to fringe candidates, general pop. Topic (intro. Mentions topic twice) and letter plus incentive (half get letter plus $5) Response rate ~63.0 Impact on estimates: 8 of 12 tests in line with hypotheses, 4 significantly so Part of the effect may be framing: people interpret questions differently based on stated topic Slide 52 51 Teitler, Reichman, and Sprachman I Fragile Families and Child Wellbeing Study Longitudinal study of new parents and their children Baseline survey: Births sampled from 75 hospitals in 20 large (200,000+) cities Total baseline sample size: 4,898 families; follow-ups at age 1 and 3 90% of mothers complete baseline; 70% of fathers (78% of those where mother completed baseline) Can characterize nonresponding fathers, since mothers provide some data about them Slide 53 52 Teitler, Reichman, and Sprachman II Baseline data collection Two survey houses (NORC, MPR); results based on 13 MPR cities Mothers interviewed in hospital Fathers in hospital (68%), by phone (10%), or face- to-face at home (2%); 80% interviewed in 13 cities Costs vary dramatically: Telephone cases 2x hospital cases, FTF cases 6x hospital cases Slide 54 53 Teitler, Reichman, and Sprachman III Estimates by cumulative rate, Respondents vs. All fathers Slide 55 Conclusions Slide 56 55 Transformation of Surveys Response rates lower Survey data collection costs are higher Incentives more widespread Small prepaid incentives most effective Less bang for the buck in telephone/face-to-face surveys: lower percentage point gains per dollar expended (Singer, 2002, page 165) Money more effective than gifts Slide 57 56 Transformation of Surveys II Image of surveys may be being fundamentally altered Formerly, surveys seen as an opportunity (to be heard) or an obligation Polls seen as an important and legitimate input to policy discussions Now, an economic transaction Incentives as payments rather than tokens invoking norm of reciprocity