CHIS Survey Methods Summary July 9, 2012 1

Overview The California Health Interview Survey (CHIS) has greatly improved the measurement of health

in California and has received attention from the survey methodology community for its

innovations. In this document we expand a discussion begun at the CHIS sponsor meeting in

Sacramento, CA on June 14, 2012. We discuss why CHIS uses the random digit dial (RDD)

sampling method, pros and cons of RDD, and active efforts to address noncoverage and

nonresponse threats to CHIS data. We then address several alternative survey methods that

CHIS has either considered in the past or is actively testing, including address-based sampling

(ABS). We close with a guiding approach to CHIS methodology going forward and a list of

selected CHIS methodology presentations and publications (see Appendix). Further details

about any of the research discussed here can be provided upon request.

Why is CHIS (Still) a Random Digit Dial (RDD) Survey? When considering an appropriate survey method for any project, the method must follow from

the project’s purpose and goals. CHIS was designed to collect policy-relevant health information

from a representative sample of California’s diverse population. With a large, geographically-

stratified sample, CHIS is able to meet the need for health data at state and county levels, as

well as for the state’s major racial and ethnic groups and subgroups. To represent California

fully, CHIS conducts interviews with adults and adolescents directly, and asks parents to report

about their children’s health. Interviews are conducted in English, Spanish, Chinese (Cantonese

and Mandarin), Korean, and Vietnamese.

After a long and deliberative planning process involving many stakeholders, CHIS decided that a

telephone survey using random digit dial (RDD) sampling best met its needs. CHIS has evaluated

and continues to explore and use other sampling strategies, including a surname list sample

and area-based sampling, but remains a telephone RDD survey because it is currently the best

survey method to achieve the project’s goals at a reasonable cost.

Why RDD Over Other Methods?

Many health surveys use telephone interviewing with RDD sampling because they are quick,

relatively inexpensive, and have desirable statistical properties. RDD sampling can be stratified

geographically by county to effectively cover the entire state of California. Telephone

interviews allow for smooth and fluid administration of complex questionnaires with efficient

data capture through Computer Assisted Telephone Interview (CATI). CATI software allows

streamlined within-household sampling (i.e., randomly selecting one adult among all eligible

CHIS Survey Methods Summary July 9, 2012 2

adults within a household) and simplifies questionnaire administration in multiple languages.

Automating the interview in this way also provides the ability to move easily among multiple

interviews completed in a single household. These features— randomized sampling within

households, interviewing multiple people in the household, and multilingual administration—

are critically important to generating high-quality data that accurately represent California’s

population and are not easily accomplished with alternative survey methods such as mail and

Web surveys.

In summary, RDD is a good fit for CHIS because it allows us to…

Efficiently contact households statewide, keeping CHIS affordable (in-person interviews

would cost much, much more)

Ensure high-quality, representative data using randomized within-household selection

of adult, adolescent, and child respondents

Ask many questions with complex eligibility criteria and skip patterns, and to fluidly

administer multiple surveys within a single household

Conduct interviews in multiple languages efficiently

Include questions on sensitive health topics that respondents may not answer honestly

in a face-to-face interview

Ask complicated health questions that respondents might find hard to answer without

an interviewer clarifying terminology

Taking content, length, cost, linguistic requirements, and other CHIS survey features into

account, a telephone survey with RDD sampling has been, and at present continues to be, the

best survey method for CHIS.

Threats to RDD

In recent years, CHIS and other surveys around the nation have been challenged by two trends:

declining response to surveys and the growth of cell phone-only households. Lower response

rates increase data collection costs and may portend nonresponse bias. There is evidence that

cell phone-only households differ systematically from households with landline telephone

service on important demographic and health variables. Thus, estimates from surveys that

sample only landline telephones are increasingly subject to potential noncoverage bias.

As with much of the telephone survey world, CHIS has been responding and adapting to a

rapidly changing survey landscape due to the dual threat of exclusive cell phone use

(noncoverage) and declining response rates (nonresponse).1

1 For more detailed descriptions of contemporary methodological terminology, findings, and design options, refer

to the book Survey Methodology, 2nd

Edition (2007, John Wiley & Sons) by Robert M. Groves, et al. Robert Groves is

CHIS Survey Methods Summary July 9, 2012 3

Noncoverage

The CHIS sample design has evolved to reflect the dramatic increase in people who have a cell

phone and no landline phone, dubbed the “cell phone-only” population. Survey statistics based

only on landline phones may misrepresent health statistics due to a lack of coverage of cell

phone-only households. Households that have only a cell phone as their method of

communication tend to be younger, non-white, and more urban than their landline-connect

contemporaries. They are also different on some important health indicators, such as health

insurance coverage and tobacco consumption.

RDD sampling can be used to produce a sample of cell phone numbers. In 2005, CHIS pilot

tested an RDD cell phone sample becoming the first large health survey to complete interviews

with this method. That successful pilot test led to the statewide cell phone sample in CHIS 2007.

A sample of cell phones has been included, and has grown, in every subsequent CHIS data

collection cycle. The cell phone sample accounts for 20% of the total sample in the current CHIS

2011-2012 cycle.

As part of CHIS 2007, with funding from The California Endowment, an area-based sample was

employed to test for potential noncoverage bias (due to cell phones) and nonresponse bias

(due to declining response rates). This sample was drawn in Los Angeles County, which had the

lowest response rate of all counties in the previous CHIS cycle. Recruiters equipped with cell

phones knocked on doors and attempted to recruit those in cell phone-only households as well

as those that did not respond to initial telephone contact attempts. In a preliminary analysis,

this test found large differences in some health estimates between landline and cell phone

respondents, demonstrating the importance of including a cell phone sample to address

potential noncoverage bias.

Having multiple sampling frames (e.g., landline and cell phone numbers) complicates statistical

weighting procedures. CHIS and its data collection partner Westat have led the complex

statistical process of appropriately generating sampling weights for dual-frame sample designs

that include households with landlines and cell phones. This work, led by renowned sampling

statistician Dr. J. Michael Brick at Westat, has been published in leading survey methods and

public health journals, and has been carefully documented in the five CHIS survey methods

reports that accompany each CHIS cycle. Many other survey organizations use the CHIS

methods reports to design and appropriately weight their surveys, which speaks to the

methodological influence CHIS has in the survey research field.

former director of the University of Michigan’s renowned Survey Research Center, current director of the U.S. Census Bureau, and has consulted with CHIS numerous times.

CHIS Survey Methods Summary July 9, 2012 4

The following innovations have been studied or implemented in response to coverage and

sample representation concerns:

CHIS 2005 cell phone pilot

Statewide cell phone RDD sample in 2007, increases in every subsequent CHIS cycle

Development of innovative weighting methods for non-overlapping and overlapping

dual-frames designs that are widely used throughout the industry

Area-based sample in CHIS 2007 to measure and detect noncoverage bias in CHIS

estimates

Evaluating CHIS sample weighting methods to address the distribution of poverty level

estimates for California

Nonresponse

Like all surveys, CHIS has witnessed a continual decline in response rates since 2001. This is

despite exploring ways to reduce or reverse the trend. If respondents differ systematically from

nonrespondents on important health dimensions, nonresponse bias may occur. Over the past

several years, CHIS nonresponse research has included:

Varying sponsorship on initial contact envelopes and letters

Focus groups to improve the messaging of contact materials

Adding monetary incentives and varying their amount

Testing methods of optimizing interviewers’ screening and interviewing tasks

Matching the surnames of respondents and interviewers to test whether mono-lingual

and bi-lingual interviewers can differentially impact response rates

Area-based sample in CHIS 2007 to measure and detect nonresponse bias

Focus groups with the parents of adolescents to improve adolescent participation rates

These efforts have had varied success at stemming the tide of declining response rates. All

things considered, we consider nonresponse to be a larger threat to CHIS than noncoverage,

primarily because noncoverage can be effectively addressed through RDD samples of cell phone

numbers. A critical part of maintaining high-quality data in this period of low response rates is

the application of nonresponse adjustments when the sampling weights are created. Westat is

a leader in this area and state-of-the-science weighting procedures are applied to CHIS to

maintain representative, high-quality data.

Our Ongoing Process of Evaluating CHIS Survey Methodology CHIS has a process for assessing and responding to challenges in survey data collection.

Participatory planning is fundamental to that process. The CHIS Advisory Board and Sample

Design and Survey Methodology Technical Advisory Committee (TAC) have consistently

CHIS Survey Methods Summary July 9, 2012 5

provided feedback and expertise on alternative survey methods. Additionally, the CHIS team

has directly consulted with leaders in the field of survey methodology to address the challenges

of nonresponse and noncoverage bias, leading to the studies implemented in CHIS 2007. In

2009, the National Cancer Institute sponsored a day-long conference focused on CHIS survey

methods that produced helpful methodological insights.

In 2005, CHIS added a survey methodologist to its staff. This is an important leadership position

on the CHIS team, responsible for assessing and exploring the methods best suited to meet

CHIS goals. The CHIS survey methodologist leads the Sample Design and Survey Methodology

TAC, is an active participant in the survey research methods community, and collaborates

extensively with our data collection vendor.

CHIS survey methods assessment and research has not only led to continual revision in the way

CHIS is conducted, but has had a broader impact on the field; CHIS methods have been the

focus of 7 peer-reviewed publications and 25 conference presentations (see Appendix).

The ongoing challenges to survey methodology, the speed of technological change, and an

environment of fiscal constraint provide an impetus to continue the review, evaluation and

discussion of CHIS survey methods to be sure that they are a) the best methods to meet the

original goals of the survey, b) keeping pace with or leading methodological innovations in

health surveys, and c) fiscally-optimal given available resources, survey methodologies, and

technologies.

Alternatives to RDD There are alternatives to the RDD method that may benefit CHIS. Every CHIS cycle has included

supplemental sampling methods to the primary RDD sample, such as surname list samples to

increase representation of some ethnic subgroups (e.g., Koreans and Vietnamese). As

previously noted, area-based sampling was implemented to test for nonresponse and

noncoverage bias in 2007, and we are conducting an address-based sample (ABS) pilot test in

the current CHIS cycle.

However, not all survey methods meet the original goals of CHIS. We briefly discuss some of the

pros and cons of three alternative survey methods below to illustrate the complex design

choices involved in advancing CHIS methodology.

Address-based samples (ABS): These designs use a list of addresses instead of phone numbers

to select households for participation. Some addresses have a phone number that can be

matched to the selected address, but matched phone numbers tend to be landline numbers,

thus not significantly expanding coverage over typical RDD samples. Addresses that cannot be

matched to a phone number must be requested by some other survey mode, such as by mail or

CHIS Survey Methods Summary July 9, 2012 6

in-person. Mailed requests remove the helpful role an interviewer plays in recruiting

respondents. Face-to-face recruitment significantly increases costs relative to RDD.

How ABS may help CHIS: ABS has become more popular in recent years primarily as a remedy

for coverage problems with RDD sample frames. CHIS implements a cell phone RDD sample to

address cell phone-only households, one RDD coverage problem. As a sample of addresses, ABS

is best suited for surveys with mailed questionnaires or face-to-face interviews. Converting CHIS

to ABS introduces challenges. Some components of CHIS will not work well as a mail survey,

specifically within-household sampling, collecting data from multiple household members, and

multi-lingual interviewing. These problems can be addressed by using ABS in combination with

other modes, such as telephone and Web. A successful mailed screener should reduce costs by

reducing interviewing labor costs needed to complete the screener by telephone. Other

surveys, with mixed success, have sent a mailed request for a telephone number to sampled

addresses and then collected data from responding household via a phone survey.

Another potential benefit of ABS for CHIS is improving coverage of households in small and

medium sized counties. Unlike with landline telephones, which are clearly tied to geography by

area codes, targeting cell phone sample to a particular geographic area is difficult and

inefficient. Difficulty matching cell phones to geography is particularly challenging in small and

medium-sized counties where those counties share cell phone area codes with large

metropolitan areas due to the way phone companies assign cell phone numbers. This often

leads to conducting too many cell phone interviews in larger counties and not enough in

smaller neighboring counties. For all but the largest counties, ABS would significantly improve

coverage because the connection between sampling unit (i.e., address) and geography is more

straightforward than with cell phone RDD.

CHIS will pilot test an ABS sample using a mailed screener questionnaire in the next several

months to explore potential benefits of the method. This test is part of TCE’s Building Healthy

Communities initiative focusing on 14 California communities. These communities are strictly

geographically-defined making RDD an unviable option for the reasons just mentioned. This test

will be closely evaluated to assess cost, efficiency, response rates, and comparability to RDD

estimates, producing an evaluation that will inform whether ABS could be used more broadly to

achieve CHIS project objectives.

Traditional panel surveys: Panel surveys follow a single group of people (cohort) over time

(e.g., Nurses’ Health Study, National Longitudinal Survey of Youth, Medical Expenditure Panel

Survey) and are appropriate for research that aims to estimate changes in individuals over time.

Limitations and opportunities of a panel design for CHIS: Restructuring CHIS as a panel survey

would fundamentally alter the nature of CHIS data and the questions it could answer. CHIS aims

CHIS Survey Methods Summary July 9, 2012 7

to measure changes in California, not individual Californians over time. Panel surveys also bring

challenges at least as great as those with RDD phone surveys. Panel attrition can lead to

nonresponse bias. Panel members relocate and people move into California, both of which

make it difficult to infer from a panel sample to a geographic area. The effort required to track

respondents who have moved drives up costs. Due to the myriad operational and statistical

difference between panel surveys and repeated cross-sectional surveys like CHIS, a panel design

would be a major operational and conceptual restructuring.

Web surveys: Having respondents complete the CHIS questionnaire on a computer or mobile

device is possible and has significant potential. This computerized, self-administered mode

allows the privacy of self-response, allows for multi-language administration, and can

accommodate complex skip patterns like interviewer-administered surveys. However,

population-based surveys have been cautious in adopting Web surveys for a number of

reasons.

Limitations and opportunities of Web Surveys: Accurately representing all of California’s

diverse population is of primary importance to CHIS. There is, however, no clear way to draw a

probability sample of “Internet users.” Web surveys generally use more traditional sampling

methods, such as RDD or ABS, to select respondents and then invite them to complete the

survey at a specific Web address. Some marketing firms use email invitations to recruit for Web

surveys, but this is not an appropriate probability sampling method for surveys of the general

population. Most California residents live at a physical address or are reachable by a phone

number, but large portions of the general public are still not connected to the Internet in any

formal way. That group is missed if email invitations are used for recruitment. Many Internet

users have multiple accounts and methods of connection, complicating the relationship

between potential sampling frames and individual people. Further, Internet access is not

ubiquitous; in late 2009 about 76% of people in California lived in a household that had access

the Internet2. Self-administration also requires literacy and could systematically reduce survey

participation among a vulnerable population group. Other large-scale surveys, like the

American Community Survey conducted by the U. S. Census Bureau, are currently testing Web

survey methods so there will soon be more evidence about how well they may work for surveys

like CHIS.

2 Internet Use in the United States: October 2009, U.S. Census Bureau, (see

http://www.census.gov/hhes/computer/publications/2009.html). The Pew Research Center’s Internet and American Life Project reports that, nationwide, Internet access and use is positively correlated with household income, making it undesirable as a single survey mode for CHIS (see http://pewinternet.org/Reports/2010/Better-off-households/Overview.aspx)

CHIS Survey Methods Summary July 9, 2012 8

CHIS Methodological Future Data are more ubiquitous than ever. This changes the world in which surveys are conducted,

and can make large-scale RDD surveys like CHIS seem irrelevant or at least “old fashioned.”

Responsible survey organizations are actively exploring new methods while maintaining the

scientific rigor of their data collection. The following guideposts will direct the methodological

future of CHIS.

Ongoing commitment to participatory planning: CHIS will continue to work with and

seek input from its broad community of stakeholders. Methodologically, CHIS is a

different survey than it was in 2001 and the collaborative and transparent nature of

methodological evolution over the past decade has served the project and its

stakeholders well.

Proactive but judicious exploration of new methods: CHIS has a responsibility to make

sure its data are as accurate as possible and not introduce additional complexity or bias.

This involves keeping our finger on the pulse of survey methodology, our survey goals,

and our own limitations, and using that information to design strategic studies that can

improve CHIS. It also involves acting on those study results to make CHIS a stronger

survey.

Responsibility to data users whose data gap CHIS originally filled: Changes to trusted

and familiar survey designs can frustrate a community of data users even if the data are

objectively more accurate as a result. If data are not used, a survey’s impact is lost. We

need to be sure we are making changes that improve CHIS data with as small of a

change to the survey as possible. We need to educate data users about the design

changes we make. We should also continually assess the needs of California’s public

health community to be sure we address emerging data needs (e.g., biomarker data

collection, new data on major health policy issues like the Affordable Care Act, surveys

of other components of the healthcare system like physicians and hospitals).

Fiscal responsibility: All of this must be done with a close eye on cost. By its nature,

survey data collection is more expensive than other forms of data collection, but its

inferential qualities usually outweigh the costs. There are numerous ways to reduce

survey costs, some of which would require major changes to the extant methodology

and some that do not. Some cost-saving measures would change CHIS goals, while

others may not. We consider cost management a key part of overall survey

methodology and are proactively seeking new ways to save costs and increase funding

for CHIS through new mechanisms.

CHIS Survey Methods Summary July 9, 2012 9

Appendix: CHIS Methodological References

Selected Publications

McClain, J., D. Grant, G. Willis, D. Berrigan 2012 Effect of temporal domain on self-reported walking behaviors in the California Health Interview

Survey. Journal of Physical Activity and Health, 9, 3: 344-51. Brick, J. M., I. Flores Cervantes, S. Lee, G. Norman 2011 Nonsampling errors in dual-frame telephone surveys. Survey Methodology, 37, 1-12. Grant, D., S. Scott, N. Breen, J. M. Brick, E. R. Brown 2011 Maintaining and enhancing representativeness of state health surveys. Survey Practice, April 25,

2-6. Lee S., J. Brick, E. R. Brown, D. Grant 2010 Growing cell phone population and noncoverage bias in traditional random digit dial telephone

Health Surveys. Health Services Research, 45, 4: 1121-39. Lee, S., E. R. Brown, D. Grant, T. Belin, J. M. Brick 2009 Exploring nonresponse bias in a random digit dialing telephone survey using neighborhood

characteristics. American Journal of Public Health, 99: 1811-16. Lee, S. and D. Grant 2009 The effect of question order on self-rated general health status in a multilingual survey context.

American Journal of Epidemiology, 169: 1525-1530. Brick, J. M., W. S. Edwards, S. Lee 2007 Sampling telephone numbers and adults, interview length, and weighting in the California

Health Interview Survey cell phone pilot study. Public Opinion Quarterly, 71: 793-813.

Selected Presentations and Conference Proceedings

Edwards, W. S., S. Dipko, R. Park, D. Grant 2012 Using a Hispanic surname list to tailor contacts in an RDD telephone survey. American

Association for Public Opinion Research, Annual Conference, Orlando, FL. Edwards, W. S., J.M. Brick, R. Park, D. Grant 2011 The validity of a question to identify cell-only households. American Association for Public

Opinion Research, Annual Conference, Phoenix, AZ.

CHIS Survey Methods Summary July 9, 2012 10

Presentations and Proceedings (cont’d)

Kennedy, C. 2011 An evaluation of popular weighting approaches in dual frame RDD surveys. American

Association for Public Opinion Research, Annual Conference, Phoenix, AZ. Stapleton, M., K. Levin, J. Newsome, S. Beauvais, S. Shariff-Marco, N. Breen, G. Willis, A. Hartman 2011 Does behavior coding capture cultural differences in survey response? American Association for

Public Opinion Research, Annual Conference, Phoenix, AZ. Willis, G., S. Shariff-Marco, T. Johnson 2011 Survey comprehension across multiple racial/ethnic groups: Evidence from the California Health

Interview Survey. American Association for Public Opinion Research, Annual Conference, Phoenix, AZ.

Zahnd, E., S. Holtby, D. Grant 2011 Increasing cultural sensitivity as a means of improving cross-cultural surveys: Methods utilized in

the California Health Interview Survey (CHIS) 2001 – 2011. American Association for Public Opinion Research, Annual Conference, Phoenix, AZ.

Edwards, W. S., J. M. Brick, R. Park, D. Grant 2010 A two-stage interviewing experiment in an RDD survey. American Association for Public Opinion

Research, Annual Conference, Chicago, IL. Grant, D., S. Lee, R. Park, J. M. Brick, W. Edwards 2010 Sampling children and teens in a cell phone health survey. Joint Statistical Meetings, Vancouver,

BC, Canada. Grant, D. M. 2009 Future of random digit dial telephone surveys. Joint Statistical Meetings, Washington, DC. Edwards, W. S., J. M. Brick, D. Grant 2008 Relative costs of a multi-frame, multi-mode enhancement to an RDD survey. American

Association for Public Opinion Research, Annual Conference, New Orleans, LA. Grant, D, S. Lee, J. M. Brick, R. Park, W. Edwards 2008 Design and Implementation of an area probability sample to explore nonresponse bias in an

RDD survey. American Assoc. for Public Opinion Research, Annual Conference, New Orleans, LA. Lee, S., D. Grant, R. Park, J. M. Brick 2008 Use of an RDD survey to explore nonresponse and noncoverage bias in an RDD survey. American

Association for Public Opinion Research, Annual Conference, New Orleans, LA.

CHIS Survey Methods Summary July 9, 2012 11

Presentations and Proceedings (cont’d)

Lee, S., and D. Grant 2007 Effect of translation on general health status across five interview languages. American

Association for Public Opinion Research, Annual Conference, Anaheim, CA. Nguyen, H.A., S. Lee, D. Grant, W. Edwards 2007 Impact of prepaid monetary incentives and sponsorship in survey advance letters: An

experiment in the 2005 California Health Interview Survey. American Association for Public Opinion Research, Annual Conference, Anaheim, CA.

Edwards, W., J. M. Brick, J. Kurata, D. Grant 2006 Effects on response rates of multiple sponsors on advance letters for an RDD survey. American

Association for Public Opinion Research, Annual Conference, Montreal, Quebec, Canada. Flores C. I., M.E. Jones, L. Alvarez-Rojas, J. M. Brick, J. Kurata, D. Grant 2006 A review of the sample design for the California Health Interview Survey. Proceedings of the

Survey Research Methods Section of the American Statistical Association. Grant, D.M. 2005 Developing a family history module—California Health Interview Survey. Centers for Disease

Control and Prevention, Coordinating Center for Health Promotion, Office of Genomics and Disease Prevention. Family History Tools to Improve the Public’s Health. Decatur, GA, Nov. 17-18.

Grant, D.M., A. Ramirez, W. Edwards, J. Rauch 2005 Collecting geographic location information in an RDD survey. American Association for Public

Opinion Research, Annual Conference, Miami, FL. Edwards, W. S., D. Martin, C. DiSogra, D. Grant 2004 Altering the hold period for refusal conversion cases in an RDD survey. Proceedings of the

Section on Survey Research Methods of the American Statistical Association. Edwards, W. S., S. Fry, E. Zahnd, N. Lordi, G. Willis, D. Grant 2004 Behavior coding across multiple languages: The 2003 California Health Interview Survey as a

case study. Proceedings of the Section on Survey Research Methods of the American Statistical Association.

Edwards, W. S., C. DiSogra, W. Yen 2003 Scheduling calls for refusal conversion in an RDD survey. American Association for Public

Opinion Research, Annual Meeting, Nashville, TN. Grant, D.M., C. DiSogra, N. Ponce, W. Yen, G. Willis 2002 Linguistic diversity in population based surveys: The California Health Interview Survey.

American Association for Public Opinion Research, Annual Conference, St. Pete Beach, FL.

CHIS Survey Methods Summary July 9, 2012 12

Presentations and Proceedings (cont’d)

Edwards, W. S., J. M. Brick, I. Flores Cervantes, C. DiSogra, W. Yen 2002 Sampling race and ethnic groups in RDD surveys. Proceedings of the Survey Research Methods

Section of the American Statistical Association.

Flores Cervantes, I., J. M. Brick, M. Jones 2002 Weighting for non-telephone households in the 2001 California Health Interview Survey.

Proceedings of the Survey Research Methods Section of the American Statistical Association.