technical report 05-022...mail-out package: the cover letter and the first page of the actual form....

TECHNICAL REPORT 05-022

Cognitive Evaluations of Three Census Form Design Features: The Internet Option Message,

Roster Instructions, and Identifying Person 1

April 2005

Prepared for

The U.S. Census Bureau

Submitted by

. Nicholas L. Parsons,

Taj Mahon-Haft, and Don A. Dillman

SESRC Social & Economic Sciences Research Center (SESRC)

P.O. Box 644014 Washington State University

Pullman, Washington 99164-4014 Telephone: (509) 335-1511

Fax: (509) 335-0116

Credits Nicholas L. Parsons and Taj Mahon-Haft are graduate research assistants in the Department of Sociology and the Social and Economic Sciences Research Center (SESRC). Don A. Dillman is Regents Professor and the Thomas S. Foley Distinguished Professor of Government and Public Policy in the Department of Community and Rural Sociology, Department of Sociology and the Social and Economic Sciences Research Center (SESRC) at Washington State University, Pullman, Washington 99164-4014. The research upon which this report is based was supported by Gunnison Consulting and The U.S. Bureau of the Census, and was conducted under Title 13 of the U.S. Code. Appreciation is expressed to Rita Koontz, Thom Allen and Mallory McBride for help with respondent recruitment and assistance in preparing this report. This is SESRC Technical Report 05-022. Any opinions expressed in this report are those of the authors and not to be attributed to Washington State University, Gunnison Consulting or the U.S. Bureau of the Census. Questions and comments should be addressed to [email protected].

Contents CHAPTER 1. INTRODUCTION ..............................................................................................1 Background And Purpose .............................................................................................................1 Issues Evaluated Here ...................................................................................................................1

Issue 1: Option to Complete the Form on the Internet..........................................................1 Issue 2: Making the Roster Box More Effective ..................................................................2 Issue 3: Who to List as Person 1 ...........................................................................................4

Organization Of This Report.........................................................................................................5 CHAPTER 2. PROCEDURES ..................................................................................................6 Recruitment Of Respondents ........................................................................................................6 Summary Of Protocol Procedures ................................................................................................7 Interview Site ................................................................................................................................8 Limitations Of The Interviews And This Report..........................................................................8 CHAPTER 3. RESULTS & DISCUSSION OF THREE PRIMARY OBJECTIVES........10 Introduction.................................................................................................................................10 Primary Test Objectives..............................................................................................................10

Option to Complete the Form on the Internet.....................................................................10 Which Message is More Effective? ....................................................................................12 Making the Roster Box Better ............................................................................................15 Question 5: Identifying Person 1 ........................................................................................20

CHAPTER 4. RESULTS AND DISCUSSION OF SECONDARY OBJECTIVES AND OTHER OBSERVATIONS .....................................................................................................26 Introduction.................................................................................................................................26 The Masthead ..............................................................................................................................26 “Form” is Misspelled as “From” on Form H..............................................................................26 Reaction to Question 2 ................................................................................................................26 Question 3: “Is this house, apartment or mobile home…”.........................................................27 Question 6: What is Person 1’s sex?”.........................................................................................27 Questions 8 and 9: Indentation of “Other” Response Options FOR “race” and “ethnicity” ......27 Question 10. “Does Person 1 sometimes live or stay somewhere else?” ...................................28 Page 2, Question 2: “How is this person related to Person 1?”..................................................28 Why Some People May Not Report Babies................................................................................29 Introduction of a Final Question.................................................................................................30 How Do These Forms Compare to Previous Forms With Regard to Ease of Completion? .......30 CHAPTER 5. RESULTS & DISCUSSION OF OVERALL FORM PREFERENCES .....31 Introduction.................................................................................................................................31 Form H or Form I? ......................................................................................................................31 Comparisions of Alternative Forms: H vs. K and I vs. J ............................................................32 Best of Four for Visibility of Internet Message ..........................................................................35 Best of Four for Getting Respondents to Read Roster Instructions ............................................36 All-around Best of Four Forms ...................................................................................................37

CHAPTER 6 - SUMMARY OF FINDINGS AND RECOMMENDATIONS.....................38 Introduction.................................................................................................................................38 Summary of Findings and Recommendations ............................................................................38

Internet Option Message .....................................................................................................38 Improving the Roster Box...................................................................................................39 Question 5 ...........................................................................................................................41

Summary of Secondary Findings ................................................................................................41 Other Findings about How the Form Works...............................................................................42 REFERENCES..........................................................................................................................43 APPENDIX A: FIGURES OF MATERIALS SHOWN TO RESPONDENTS ..................44 APPENDIX B: INTERVIEW PROTOCOL..........................................................................54

List of Tables and Figures Table 1. Respondent Characteristics ....................................................................................7 Figure 1: Census Form Cover Letter ................................................................................ 44 Figure 2: Form H (DC-2H) ............................................................................................... 45 Figure 3. Form I (DC-21).................................................................................................. 51 Figure 4. Form J (DC-2J).................................................................................................. 52 Figure 5. Form K (DC-2K).............................................................................................. 53

Report to the U.S. Census Bureau

1

Cognitive Evaluations of Three Census Form Design Features: The Internet Option Message, Roster Instructions, and Identifying Person 1

By:

Nicholas L. Parsons, Taj Mahon-Haft, and Don A. Dillman Washington State University

CHAPTER 1. INTRODUCTION BACKGROUND AND PURPOSE This is the third of three reports that analyze results from a series of cognitive interviews conducted at Washington State University between July 2004 and March 2005. The first analysis of 30 interviews compared the 2000 Census form with one that incorporated new optical features aimed at improving respondent comprehension of questions as well as machine processing of people’s answers (Dillman, Parsons and Mahon-Haft, 2004). The second analysis of another 30 interviews focused on respondent reactions to a proposed online replacement mail-out (Dillman, Mahon-Haft and Parsons, 2004). In both of these studies, respondents provided answers to one or more Census forms and suggestions were made for possible changes in the forms to improve respondent comprehension of Census questions. An additional 22 interviews were conducted in March, 2005 to follow up on an issue that arose in the earlier reports—the wording of the question that asks for the listing of information about Person 1 on the Census form. At the request of the Census Bureau, we have investigated two other issues in this report: 1) the wording and visual display of an option to respond to the Census form on the Internet; and 2) the wording and visual display of roster information stating who should or should not be reported for the responding household. ISSUES EVALUATED HERE Further analysis of each of the issues being evaluated yielded one or more test objectives. The analysis and resulting test objectives are detailed below. Issue 1: Option to Complete the Form on the Internet Previous Census tests have concluded that a number of respondents wanted to complete the form on-line. Providing respondents with the option to complete the Census form electronically might improve accessibility and completion time for many regular Internet users, potentially increasing the overall response rate. A primary issue for investigation in these interviews was to determine how respondents would react to inclusion of the Internet option on the Census form.


2

The opportunity to fill out the Census form online is announced in two locations in the mail-out package: the cover letter and the first page of the actual form. It first appears in bolded text in the second paragraph of the cover letter accompanying the form in the mailing package (see Appendix A, Figure 1). On the front of the form, it is located just below the masthead and just above the “Start Here” instruction and roster box. The test forms included two different visual presentations of the message on the form, but included the same wording:

“You may either answer this form on the Internet at www.Census.gov/Census2005 OR fill out this paper form. Use a blue or black pen and:”

This test was intended to determine whether the Internet option would cause any detrimental side effects that might offset the advantages, such as delays in response time or strong negative reactions from people who have an unfavorable impression of the Internet. We also asked people about their experience using computers and inquired about their likelihood of taking advantage of the option to respond electronically. If the message was well received, we intended to find out which form (H or I) had a more effective design for getting the message noticed. A clearly visible Internet message stands the greatest chance of improving response rate by making the offer known to more respondents. We set out to discover which visual design was best suited for making the message highly visible without interfering with completion of the form or conflicting with visual design aspects of other parts of the form. The difference in the forms was in the color scheme of the visual element that included the message. On Form H, the text was printed against a light blue background matching the background of the questions on the form (see Appendix A, Figure 2). In contrast, Form I had the Internet message printed against a dark blue background (see Appendix A, Figure 3). Form I’s color scheme resulted in the elimination of the boundary between the Internet message and the “Start Here” instructional arrow so that it would be perceived visually with the starting point instead of as a separate visual element. Thus, what appears as a simple change in color actually changes the visual interpretation of the beginning of the form, potentially amplifying any effects of the difference.

Test Objectives: To learn whether having the Internet option would hurt response rates or response times; to determine which Internet option message respondents were most likely to read; and to record any positive or negative reactions to its presence. We considered it important to obtain responses from people having a wide range of perspectives on and accessibility to the Internet. However, by agreement with the Census Bureau, this aspect of recruitment was assigned a somewhat lower priority than recruitment based on other criteria for fulfillment of the objectives described below. Issue 2: Making the Roster Box More Effective The roster box has undergone significant changes from the 2000 Census Form in both visual presentation and the wording of the roster instructions. In our previous studies,


3

respondents were sometimes confused about whom to include and exclude. Errors seemed more evident when respondents had to decide whether or not to include part-time residents, visitors, and people without permanent residences (Dillman, Parsons, and Mahon-Haft, 2004). We have modified the visual design on both forms — the “Include”/“Do Not Include” instructions are presented as a series of bulleted points in a single column to reduce the likelihood that one column will be skipped over. The bulleted points contain all of the instructions necessary to address potentially confusing roster decisions. The remaining text introduces and explains the roster rules. The main background of the roster box is a darker shade of blue than the light blue background that the inclusion/exclusion criteria are printed against, with the intention of drawing respondents’ attention to the important information. Three vs. Two Highlighted Areas of “Inclusion/Exclusion” Instructions Different ordering and organization of the instructions could lead to different interpretations of their meaning. Thus, a main goal of this test is to determine whether respondents better understood two or three highlighted areas. On both forms, five bulleted points explain the roster box rules for whom to include and exclude. On Form H (and alternative Form K), the criteria are grouped into two sections of bulleted points, first the “Include” instructions and then the “Do Not Include” instructions. On Form I (and alternative Form J), the bulleted instructions are divided into three sections. The main “Include” points precede two “Do Not Include” points, followed by a single point instructing how to count people with other institutional residences (e.g., those persons currently in the military). Form I also contains an additional explanation printed in dark bolded text that describes the rationale behind the fifth bullet. This explanation is not present on Form H. Applying different background colors behind the bulleted points is designed to draw attention to the most essential information, but doing so also adds complexity to the visual design. Too much visual complexity has bothered respondents during prior tests, so the addition of the attention-drawing backgrounds does not guarantee better respondent comprehension. On the alternative forms (J and K), all of the information in the roster box is set against a dark blue background, reducing the visual complexity created by the highlighted areas and possibly changing people’s overall reaction to the forms. The interviews included questions intended to determine which layout of the roster box is most visually pleasing and most effective for focusing respondent attention on the pertinent “Inclusion/Exclusion” criteria. Different Wordings in the Roster Box It is important that everyone living permanently in the United States be counted by the Census, even if they have no permanent address. At the same time, people who stay in multiple places must not be counted more than once. The new forms introduce revised instructions for counting people who had no other permanent residence.


4

The wording of the criterion for how to count people without another residence differs slightly between Forms H and I. Similar to past Census forms, in Form H, persons without a permanent residence are to be counted where they “sleep most of the time.” On Form H, the following instruction appears as the second bullet in the first highlighted area:

• “Count visitors here on February 15, 2005, only if they have

no other place they sleep most of the time. Otherwise, they may be missed in the Census.”

Since the terms “sleep” and “visitor” occasionally confused respondents interviewed in previous tests, Form I presents a somewhat different instruction that appeared as the fifth bullet in the roster box. It was also the only bullet in the third highlighted area on Form I. This bullet reads:

• “If someone who has no permanent place to stay is staying

here on February 15, 2005, count that person. Otherwise, he or she may be missed in the Census.”

Test Objective: To determine which version of the roster box has more effective wording, organization and visual appearance. To test the effects of the variations in wording, organization and visual appearance, we needed to obtain responses from persons living in complex or unusual households so that the roster instructions would be essential for them to understand whom to include or exclude. Thus, an effort was made to locate households with several unrelated adults, temporary visitors, and a householder away in the Armed Forces. Issue 3: Who to List as Person 1 Question 5 on the first page of the form instructs respondents regarding whose information should be filled in under Person 1. During previous interviews (Dillman, Parsons, and Mahon-Haft 2004; Dillman, Mahon-Haft, and Parsons 2004), some respondents were confused about whether or not to include their own name and information on the form. This difficulty appeared to result from the complicated third-person wording of the instructions prior to Question 5 (Dillman, Parsons, and Mahon-Haft, 2004). In the interviews, we tested two versions of Question 5 with revised wording to find out which one respondents understood more clearly. The bold sentence leading off each of the instructions is worded differently, while the instructions in italics after that are worded identically. Because the first sentences are different lengths, the line breaks in the instructions are also different.

On Form H, the instruction for Question 5 reads:

• “Next print the name of a person who lives here and in whose

name this house or apartment is owned or rented. If an owner or renter does not live here, print the name of any adult living


5

here. This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)”

On Form I, the instruction reads:

• “Next, if the owner or renter of this house or apartment lives

here, print the person’s name below. If an owner or renter does not live here, print the name of any adult living here. This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)”

Test Objective: To determine which question wording was easier for respondents to comprehend and answer correctly. Although all households surveyed are important to evaluating these questions, the wording is particularly important to residences occupied by persons who do not own or who pay cash rent. We are also concerned that this wording might be confusing to people who fill out Census forms for other people, e.g., a caretaker for elderly householders. Thus, we made an explicit effort to recruit respondents who occupy their residence without owning it or paying rent (e.g., apartment managers and pastors in parsonages) and caretakers who would fill out the Census form for others. ORGANIZATION OF THIS REPORT Chapter Two reports the recruitment of respondents, respondent characteristics, and limitations of this research. In Chapter Three, we discuss results regarding the three primary differences between the forms being evaluated. In Chapter Four, we cover comments about secondary issues arising from the fact that the forms used in these interviews are different than any used in previous Census tests. Chapter Five discusses the overall form preferences of respondents and provides information on their reactions to the alternative forms (J and K). Finally, in Chapter Six, we offer conclusions and recommendations based on our findings.


6

CHAPTER 2. PROCEDURES

RECRUITMENT OF RESPONDENTS

Some of the features being evaluated in these cognitive interviews required us to recruit persons from unusual housing situations. A summary of the main and sub-variations compared between test forms is as follows:

1. Visual display of the Internet option (main) and connectivity to the roster box because of line location (sub).

2. Three highlighted spaces in the roster instructions vs. two highlighted spaces (main), and differences in the location and wording of the temporary visitor instruction (sub).

3. Different lead- in sentences describing who should be listed as Person 1 (main) and differences in the line composition (i.e., line-breaks) of the remaining two sentences of this request (sub).

The composition of households needed for testing the roster box variations differs significantly from those needed for evaluating certain aspects of the Person 1 request. Even though it is possible to obtain reactions to the Internet option from all respondents, it is also important to obtain respondents with and without Internet access and who have varying attitudes towards use of the Internet. For practical reasons, we decided to focus our selection of respondents on complex households (to test the roster box) and households with unusual living situations (to test the Person 1 request). We believe that while focusing on these selection criteria,we will obtain the desired variation in Internet access and opinions. Because it was not our objective to test other features of the form (e.g., gender, age, ethnicity and race questions), we relaxed those criteria for selecting respondents. Our selection efforts resulted in the recruitment of 22 respondents, for which the recruitment criteria and special characteristics are listed in Table 1. All respondents were recruited from the Moscow, Idaho and Pullman, Washington areas, and were interviewed at Washington State University. Two employees of the Social and Economic Sciences Research Center (SESRC) who have lived in the Moscow/Pullman area for many years recruited the majority of respondents by asking acquaintances to identify people with particular characteristics. We also systematically called off-campus households and screened for several unrelated adults living together. Each invited respondent was told he or she would receive 35 dollars in appreciation for their time. During recruitment, the recruiter explained our research goals and respondent expectations, but tried to conceal most of our specific objectives (e.g., learning how persons who live in but do not pay for their residence react to the wording of Question 5). Those who agreed were asked to come to Wilson Hall on the Washington State University campus, where a large room (approximately 25 by 25 feet) near the building’s


7

ground level entrance was labeled with a sign saying, “Census Improvement Project.” Respondents were not required to go through a secured entry point of any kind. Like most college campuses, many buildings at the Washington State University campus can be freely entered and exited.

Table 1. Respondent Characteristics Int # Name Sex Age Race/Ethnicity Education Special Traits Int Date

1 Betty F 45 White SC NR, M 2-16-05 2 Tracy F 30 White/Hispanic SC NR 2-17-05 3 Nancee F 30 White SC NR 2-17-05 4 Norma F 53 White BA I 2-18-05 5 Eileen F 30’s White BA CG 2-18-05 6 Dorothy F 29 White BA NR, M, NM 2-18-05 7 Nadene F 20’s Asian Indian BA CH, NR 2-19-05 8 Isaac M 49 White PhD NR 2-19-05 9 Ursula F 42 White HS CH 2-21-05 10 Dianna F 51 White SC M 2-21-05 11 Nicole F 38 White BS NR 2-22-05 12 Olive F 60’s White MA CG 2-22-05 13 Beverly F 20 White SC CH 2-22-05 14 Laurie F 21 White BA NR 2-23-05 15 Bea F 34 White/Russian MA CH, NM 2-23-05 16 Evelyn F 19 White SC CH 2-23-05 17 Veronica F 50’s White SC CG 2-24-05 18 Tia-Maria F 34 White MA CH, NM 2-25-05 19 Esther F 73 White MA I 2-26-05 20 Todd M 40’s White SC CH 2-26-05 21 Terra F 42 White SC M 2-26-05 22 Kyle M 29 White HS CH 2-26-05

Notes: Highest Education Special Traits (i.e., reasons for recruiting) HS = High School NR = Housing provided with job; no rent SC = Some College I = Internet Concerns BA/BS = Bachelor’s M = Household member in Military MA = Master’s CH = Complex Household PhD = Doctorate CG = Caregiver (filled form for another person) NM = New Mother SUMMARY OF PROTOCOL PROCEDURES During each interview, respondents were assigned to complete one of the forms in full (odd numbered respondents received Form H, and even numbered respondents received Form I). Each respondent was then asked to complete the first page of the other form. Later in the interview, respondents were shown the alternative forms without the highlighted areas (Forms J and K) and asked to make comparisons. A 17-page protocol was created to guide the interview process, a copy of which is included in Appendix B.


8

Compared to the previous two series of cognitive interviews (Dillman, Parsons and Mahon-Haft 2004; Dillman, Mahon-Haft and Parsons 2004), these interviews were slightly less structured. For example, several of the questions on the protocol were only relevant if respondents answered a certain way to previous questions. When data from a single interview became saturated with respect to a specific test objective, additional questions that would have appeared redundant were not asked of the respondent. Don Dillman conducted all of the interviews. Nick Parsons and Taj Mahon-Haft took detailed notes for 11 interviews each. Each interview was summarized in written form (5-8 single space pages) within one day of the interview being conducted. These interview summaries aided in writing this analysis. INTERVIEW SITE The room used for interviews was set up with desks, wall hangings, books in book cases and other features around the periphery that made it seem like a multiple-purpose room. When respondents came into the room, they were asked to sit beside the interviewer at a table near the center of the room. A second note taker sat in a chair in front of and about ten feet to the side of the interview table, not directly facing the respondent. The location of the second person was designed to reduce potential anxiety that sometimes arises during interviews when a second person is present and cannot easily be seen. LIMITATIONS OF THE INTERVIEWS AND THIS REPORT In our view, completing only 22 interviews (as compared to 30 in past tests) was the most significant limitation of this investigation, reducing the breadth of our data. This reduction in number was the result of severe time constraints that required that all interviews be completed within a two-week period. In addition, the three main variations (Internet option, Roster Box and Identifying Person 1) appear on only two forms. This means that the same three variations are “blocked” with one another, making it possible that respondents’ reactions to one aspect of the form are influenced by reactions to another part. When we discussed this possibility with U.S. Census Bureau staff, we jointly concluded that the likelihood of such effects was fairly low, so we opted for limiting the number of different forms needed for this test to two. However, it is a potential limitation that merits recognition. A third limitation regards the demographic characteristics of our sample. Since the current set of 22 interviews addresses three specific issues requiring recruitment of households for special characteristics (e.g., persons ne ither renting nor owning their residence), we concluded that we could not recruit for variations in race, ethnicity, and education to the degree achieved in previous tests. Consequently, the racial background of the majority of our respondents is white and nineteen of our interviewees are female. A fourth and final limitation of these interviews is that the alternative replacement forms (J and K) sent by the Census are not printed in the same shade of blue as Forms I and H.


9

The backgrounds of Forms J and K are a very light blue due to printing differences. We presented respondents with Forms J and K to compare the un-highlighted roster instructions with the highlighted instructions on Forms H and I. To compensate for the discrepancy, we asked them to imagine that the background colors were identical, with the exception of the roster instructions. We do not know whether respondents would have reacted identically if the alternative forms had been of the same shade of blue as Forms H and I, but do not believe that this limitation was detrimental to interpreting respondents’ form preferences.


10

CHAPTER 3. RESULTS & DISCUSSION OF THREE PRIMARY OBJECTIVES

INTRODUCTION The overarching goal of this research is to identify means of improving response rates and accuracy on the Census form. Specifically, we are seeking the answer to certain questions relevant to the form designs for the 2005 National Field Test to be conducted in September. This is our first gauge of respondent reactions to the addition of the option to complete the form online, which is intended to improve response rate and possibly lower costs. We are also investigating the effects of reducing confusion and improving response accuracy through changes made to roster box instructions and the instructions on who to count as Person 1. Several of the changes made to these new forms are especially important because they address significant problems people experienced when completing forms in previous tests. PRIMARY TEST OBJECTIVES Option to Complete the Form on the Internet We recruited several respondents who were known, through informants, to be uncomfortable with submitting personal information over the Internet. We recorded all reactions to the messages as they completed the forms and asked a series of questions about their computer access and experience. We also asked respondents which visual layout of the Internet message was more likely to be noticed. The responses gathered during these interviews provide considerable insight into respondents’ views regarding the opportunity to complete the Census form online. We found no evidence that the inclusion of the Internet option message on the form and in the cover letter would damage response. As expected, a number of respondents strongly preferred the traditional paper format, most often because of security concerns. But the majority of respondents also indicated that, given the chance, they would be either “somewhat likely” or “very likely” to complete the form electronically.

How likely are you to complete the form on-line? (n=22)

Very Likely 32% Somewhat Likely 27% Somewhat Unlikely 14% Very Unlikely 27%

Respondents who preferred filling out the form online often mentioned convenience as the primary reason for their choice. Some respondents were quite pleased to be able to complete the process with fewer steps, have less paper to keep track of, and complete it while sitting at their computers doing something else. Comments made by some respondents who elaborated on their answer to this question include:


11

• Dorothy (6): “[I am] somewhat likely. It depends on how crazy my kids are being

at the time. It’s easier to start it on paper, put it aside to attend to my kids and then pick up the form again.”

• Ursula (9): “Very likely because I can just type answers in, hit send, and not have to deal with postage… don’t have to mail it.”

• Dianna (10): “Somewhat likely - just because of the convenience of online and not having to drop it in the mail; it’s just easier.”

• Veronica (17): “Me? Very likely. It’s absolute convenience for me. The ladies? [For whom she was responding as a caretaker.] Very unlikely; they don’t have a computer available. Our elderly people are stuck in their ways. They don’t trust the computer and would prefer to do it on paper.” Veronica filled out the form for an elderly woman she cares for.

Those who reported being “somewhat unlikely” or “very unlikely” generally either had security concerns or were not confident in their computer abilities.

• Norma (4): “Very Unlikely. It ain’t happening. It’s too easy for hackers.” • Evelyn (16): “Somewhat Unlikely - because it’s important info and you don’t

know if your info is out there.” Despite the preferences of some respondents to complete the paper version of the Census form, no respondents reacted harshly to the Internet option appearing on the form or in the letter. Many of those most opposed to filling out the form online were even supportive of providing that opportunity to others. Respondents who preferred to complete the paper version of the Census perceived themselves as part of a shrinking minority. Even though they personally preferred the paper form, they felt strongly that most people with Internet access would prefer to complete the form online.

Do you think most people with Internet access would prefer to fill out the Census form on the Internet if they had the chance, or would they prefer to return the paper form? (n=22)

On Line 82% Paper 18%

Comments made by some respondents who elaborated on their answer to this question include:

• Tracy (2) said she would prefer to fill out the paper version of the Census form but thought more people would prefer the Internet. “It’s an easier way to do it. For me, I adapt to change slowly. Maybe in the future I’d fill it out online, but not now.”

• Eileen (5): “Most would prefer the Internet. For people in my line of work [care-taker] especially, because everything’s computerized for many of them and it involves less paperwork. Generally, it depends on people’s age. People under 40


12

would like the Internet option better, but people fifty and older would be suspicious and like to have that paper.”

• Dorothy (6): “Most would prefer online; most people love the computer and do everything on it. They’re more computer literate than me.”

• Nadene (7): “[Most people would prefer] the Internet because it is just easier and more efficient, especially for people without a postal service outbox near their house.”

• Dianna (10): “People that I know would want to fill it out online. They use e-mail a lot, do a lot of things online like shopping, searching for info, etc. They’re very comfortable with the Internet.”

• Nicole (11): “Most people would prefer Internet. People are on their computers at work all the time. It’s easy. I wouldn’t though. I’ve lived overseas before and the Internet is not that secure yet. I’ve seen too many bank accounts wiped out.”

• Beverly (13): “The Internet—it’s convenient. You don’t have to worry about envelopes and mail or if it’s too late. I think more people would finish it on time.”

• Evelyn (16): “The Internet, because it seems like a lot more people use the Internet… because it’s faster, I guess.”

• Tia-Maria (18): “The Internet because it is just easier to get it done with… you don’t have to deal with any paperwork or mailing it in.”

• Todd (20): “The Internet. I expect most people are more proficient than me and assume that people with the computer skills to do so would.”

• Terra (21): “I think most would prefer the Internet, but it would depend. All those old people don’t want to use the Internet. You gotta think about them.” When the question was reread, emphasizing “most people with Internet” she said, “A lot more people are going on the Internet. It’s the way of the world.”

We also evaluated whether people might respond more slowly to the Internet version than to paper. Ten respondents explicitly preferred the idea of completing the form online, and only one of those ten felt that it would take them longer to complete the form that way. Overall, several people believed that completing the form on-line would take them more time, but these respondents reported that they would be more likely to complete the paper version.

Compared to the paper form, completing the form on-line would be done: (n=22)

Sooner 23% Same time 41% Later 36%

WHICH MESSAGE IS MORE EFFECTIVE? To maximize the benefits of the Internet option, respondents must see the message. Therefore, we also examined when and where they first read it and asked respondents for their opinions on which of two designs (on Forms H or I) was more effective in attracting attention. The Internet option message on Form H was set against a light blue background, separating the message from the darker background of the roster box and


13

creating more contrast between the black text and the background. Form I located the Internet option message in a darker blue background that tied it visually to the “Start Here” arrow of the same color background, located directly beneath it. The message on Form I provided more contrast between the Internet option message box and the form’s light blue background, but offered less of a contrast between the text and the darker blue background it was set against. From our observations, the Internet option messages on the forms were equally likely to be noticed, with each being read aloud only about half the time. However, several respondents who did not read the messages aloud later said they had noticed them but moved on quickly because they already knew the option existed from reading about it in the cover letter. At least eight respondents noticed the message and commented on it when looking over the cover letter, before they even examined the form itself. On the form, they caught a quick glance of the underscored URL and understood what it represented. It appears that Internet addresses are quickly becoming a readily recognized symbol in the U.S. The increase in the number of people who attribute the same meaning to this symbol could be used advantageously to better capture respondents’ attention. Veronica (17) even suggested that writ ing the font of the Internet instruction in blue would better signify that it is a web address (on e-mails and word documents, hyperlinks often appear in blue). Despite our observations that the Internet note was equally likely to be noticed on both forms, a preference for use of the dark blue background on Form I emerged during the discussions we had with respondents. A strong majority told us that having the Internet option message set against the dark blue background (on Form I) was more likely to be noticed. These respondents generally argued that the darker background attracts visual attention and highlights the message more effectively. Others added that the light blue background used in Form H makes the text of the Internet message fade into the regular text of the questions set against the same color. In addition, a minority (about 1 out of every 5 respondents) felt the text was more likely to be read against the light blue background of Form H, arguing that it provided greater contrast.

Which version of the Internet note has the best chance of being seen? (n=22)

Dark Background (Form I) 73% Light Background (Form H) 18% No Difference 9%

Some respondents who believed the Internet note was more noticeable on Form I (with the dark blue background), said :

• Eileen (5): “It’s highlighted up there… seems like important thing to read… in the ‘Start Here’ section.”

• Ursula (9) did not read either note, but said the invitation on Form I was “probably more noticeable because it says ‘Start Here.’”

• Dianna (10): “It seems more like I’d see it. I’d be less likely to read what’s before ‘Start Here’ on Form H.” When probed however, she realized she saw the Internet note in the letter, not on Form I.


14

• Nicole (11) “It sticks out, it’s highlighted. Also, it says ‘Start Here.’ It’s part of a highlighted area. With Form H, I saw ‘Start Here’ and I looked down [and consequently, did not notice the Internet instruction].”

• Tia-Maria (18): “The color of Form I draws you [to the Internet note]. The block of blue directs your eye downward into the instructions.”

Respondents who claimed the Internet option message was more noticeable on Form H (with the light blue background) said :

• Tracy (2): “‘Start Here’ draws my attention, then my attention was drawn up [to the note].”

• Dorothy (6): “I see the ‘Start Here’ on the I form; there’s so much dark text in the box [on Form I] that [the Internet note is] not as noticeable.”

• Nadene (7): “Form H is better because of the demarcation.” She did not read it on Form I mostly because she knew it was there already. She said that she skipped it on Form I because the dark blue background that matched the “Start Here” instruction “made the two run together.”

• Isaac (8): “Form H is better because [the Internet note] was not in the dark background. It’s harder to read on I. There’s not enough contrast and it’s not as friendly.”

• Evelyn (16): “On Form H, [the Internet note] seems more of a separate section. But, if Census put the ‘Start Here’ before the Internet box, people would read it more often.”

When asked solely about the overall visual appearance of the Internet instruction on both forms, most respondents also preferred the dark blue background used on Form I. Several specifically mentioned the pleasing visual combination of the Internet message, the “Start Here” arrow, and the roster box. Having the Internet message on Form I share the darker blue background color with the “Start Here” arrow makes for a single, larger visual figure located at the upper left-hand quadrant of the form. This change should bring added attention to the note, since the blue background is the most attention-getting visual piece on the page. It also increases the visual continuity between the note and the roster box, which many people found visually pleasing. Overall, the visual effects of the dark background on Form I improved the aesthetics and should increase the likelihood that the Internet message will be seen. Overall The dominant finding from these 22 interviews is that not a single respondent objected to the presence of the Internet option message. Although many preferred to respond by paper, offering that option for others was not something that people found objectionable. We also believe that, visually, the dark blue message works better with other elements of the form, particularly the roster box, with which it is visually integrated. Recommendation 1: The option to complete the Census over the Internet should be printed on the Census form. The message on the form should be presented in the dark blue format that integrates it with the roster box and draws visual attention to it. The


15

message should also appear in the letter, as in the test letters. The URL in the letter should be printed in a different color ink as well, preferably blue, so that it also draws attention and is visually associated with a link to the website. It is important to note that the bolded phrase in large font, “Start Here,” is located directly below the Internet instruction on both forms. In many interviews, respondents unconsciously processed “Start Here” before anything else. Since an arrow points down toward the roster instructions directly below this “Start” instruction, several respondents ignored the Internet message on both forms, and began reading the form below “Start Here.” The Census Bureau may consider locating the “Start Here” message above the Internet message on future forms to further increase its likelihood of being noticed. Making the Roster Box Better The roster box instructions on who to include on the form are critical to some respondents, but generally irrelevant to others (e.g., households with two adult partners or only one person). For these households, this section is often skimmed because respondents have no uncertainty over whose names should be included. Because of this concern, we recruited respondents to whom the roster information would be most importnat, and consequently, more likely to be read. All respondents were asked about the three critical issues involving the roster box. We analyzed their responses to find out if they preferred two or three sub-sections of bulleted instructions ; their reactions to the bullets being placed against an attention-drawing background color; and whether people preferred the wording of the second highlighted bullet on the H form or its equivalent, the fifth bullet on the I form. Two vs. Three Highlighted Areas We sought to determine whether it is better to organize the roster box into two or three areas of bulleted instructions. The differences in organization are primarily visual, but the additional instruction provided prior to the fifth bullet on Form I in combination with the alterations to the order of the bullets tie this issue to the way respondents interprete the roster box. This analysis takes into account all of those elements. To isolate the visual effect of the different organizations as much as possible, we asked for respondents’ visual preferences between the two roster box designs after asking specifically about the wording of each. However, we also realized that the look was likely to be influenced to some degree by the alternative ways of presenting the information about responding on the Internet located above the roster box. On Form I, the Internet information is integrated with the box, particularly the “Start Here” instruction, whereas on Form H, the Internet information is presented above and separate from the roster box. Thus, we expected that some respondent’s answer on the look of the roster box would take that information, as well as the number of highlighted boxes, into account. Visually, respondents had no clear preference when choosing between the two roster box styles. Form I (containing three highlighted sections) was preferred slightly more often.


16

Just in terms of looks, which box is better to you? (n=22)

Three highlighted areas (I) 54% Two highlighted areas (H) 46%

Some respondents who preferred the two highlighted areas on Form H said:

• Nancee (3): “Form H seems more organized, while [Form I] seems to run on.” • Dianna (10): “I prefer just the two boxes on Form H.” • Laurie (14) preferred the looks of Form H because “It is a lot more condensed, it

flows better and it looks like not as much text because there aren’t three different boxes, just two main ones.”

• Bea (15): “On Form H the instructions are easier to understand without jumping back and forth between ‘Include/Exclude.’”

Three respondents who preferred the three highlighted areas on Form I said :

• Olive (12): “Form I looks better. I do better with smaller divisions. It looks more official, I guess.”

• Evelyn (16) preferred Form I, saying it was “more enticing… not just block style [as on Form H].”

• Esther (19) preferred Form I because “It seems like there is less info to take in at one reading because of the smaller shaded regions [in Form I].”

Only a few people demonstrated a very clear preference for three highlighted sections. At least four people chose Form I not because of the visual appearance of the roster boxes alone, but because they preferred the integrated connection between the roster box and Internet option message.

• Ursula (9): “I just like the way [Form I is] more inclusive with the web address. I like the way the Internet message and the [roster box] go together visually [on Form I].”

• Beverly (13): “Form I looks better because the big dark blue box [of the Internet instruction] draws my attention to the roster box more than the dark little arrow [on Form H].”

• Veronica (17): “The dark blue box is included [on Form I] so it is more noticeable.”

• Terra (21): “The Internet note on Form I is drawn out better. They’re showing me that the whole [roster instruction] box has more importance. I really like the way the box is broken down and separated.”

In addition, respondents who had objections to the wording of the second bulleted “Inclusion” criterion of Form H were not able to separate this concern from its visual layout when asked solely about looks.


17

Looking more closely at the responses, it seems that people preferred having two highlighted areas (Form H) to three (Form I). Several respondents mentioned that having only two regions allowed them to easily consider first whom to include and then whom to exclude. Conversely, three highlighted regions broke up that simplicity and forced them to consider first whom to include, then whom to exclude, and then more about who should be included. The three-step process was confusing for some people, whose remarks included:

• Tracy (2): “The blocking [of the roster instructions] on Form H is better. With H, the first two bullets are ‘count,’ ‘count.’ The third, fourth and firth, say ‘do not,’ ‘do not,’ ‘leave.’ Grouping them like this seems more important.”

• Dorothy (6): “[The roster criteria on Form H are better] maybe because it looks simpler. ‘Count these, don’t count these.’ It sounds clearer. It’s more-clean cut. I like to finish one thing and move on to the other.”

• Beverly (13): “The way [the roster instructions on Form H] are grouped makes more sense. The last box on Form I kind of throws you off.”

Though direct respondent preferences slightly favored the visual design of the roster box on Form I, the comments made during the interviews suggest that respondents prefer the delineation of the highlighted sections in Form H. Because of the potential confusion from jumping back and forth between concepts with three sections, two highlighted sections seem preferable. However, respondents’ preferences are not strong and no one actually expressed extreme confusion or made any visible errors because of the three highlighted sections in Form I. We continue this discussion in Chapter 5, where we describe comparisons made between Forms H and K and Forms I and J. The alternative Forms K and J differed only with respect to the lack of highlighting used to illuminate the “Inclusion/Exclusion” criteria. We did not have respondents complete Forms J or K, but asked for their preferences regarding the use of highlighting the criteria in light blue or making the background color the same dark blue color background that the bolded instructions are printed on. The Use of Different Wording for “Inclusion/Exclusion” Criteria Forms H and I also differed with respect to the wording of the criterion regarding how to treat persons without a permanent place to live. By specifically recruiting respondents with living situations that required them to carefully consider the roster instructions, we acquired a deeper understanding of how the wording of this criterion can avoid future miscalculations. We asked multiple questions about their comprehension of the instructions. The fifth bullet on Form I read:

“If someone who has no permanent place to stay is staying here on February 15, 2005, count that person. Otherwise, he or she may be missed in the Census.”


18

The second bullet on Form H read:

“Count visitors here on February 15, 2005, only if they have no other place they sleep most of the time. Otherwise, they may be missed in the Census.”

Respondents demonstrated a strong preference for the wording of the fifth bullet on Form I.

Which wording of the bulleted items was preferred? (n=21)

Preferred 5 th bulleted item (I) 86% Preferred 2nd bulleted item (H) 14%

Some respondents explained that they preferred the wording of the fifth bullet because its use of the phrase “staying here” more clearly conveyed to them who should be considered when applying the rule. These respondents often referred to situations in which people they know sometimes stay with them, “sleeping” away from their official Census residence. In one interview, a woman briefly pondered whether or not her boyfriend, who often stayed at her apartment, should be counted, since he slept there on February 15, 2005 (the specific date listed). The phrase “no permanent place to stay” (on Form I) was much more informative for respondents than the use of the term “visitors” (on Form H), and seems to have made the largest difference in how people perceived the roster box and counted the number of persons in their household. Many people expressed confusion over the term “visitors” because it has so many possible connotations. It introduced the idea of guests without specifying a time frame. Furthermore, the respondent then had to interpret two additional clauses, “only if they have no other place they sleep,” and “most of the time,” in order to decide whether to report visitors. It was common for respondents to have had visitors to their household, e.g., people who came for a meal or stopped by for a short time. Thus, the wording of the second bullet of Form H required respondents to think about certain people and then eliminate them based upon the remaining part of the instruction. In contrast, the alternative bullet (Form I) about considering someone with no permanent place to stay made it clear at the beginning of the instruction that inclusion was provisional (if). This bullet also clarified what kind of visitor the respondent was being asked about (i.e., one with no permanent place to live).

• Betty (1) “I is clearer because the fifth bullet says, ‘if they have no permanent place to stay.’ On the other one (H), it makes me think I may have to include someone who is just staying the night.”

• Dianna (10) hesitated while filling out Form H when she read the second bullet and said, “There’s something about…when I think of visitors, I think of someone visiting here. This implies homeless people. I really have to stop and think what exactly they want here. A visitor is someone I know like a friend who lives someplace else.” When debriefed later regarding which bullet was clearer to her, she immediately said, “The fifth bullet on Form I. It is much clearer. All of my


19

questions and concerns [that I had filling out Form H] would have been answered.”

• Olive (12): “The leader (i.e., bold introductory statement) for this direction (the fifth bullet on Form I) was clearer. On form H, the second bullet made me stop. ‘Count visitors’ made me think, ‘Oh, if I visited, then I should count me,’ but then I read on and learned not to.”

• Veronica (17): “The fifth bullet on Form I is clearer. The second bullet (on H) uses ‘visitors.’ This is a common word. My lady [that I am a caretaker of] has visitors all the time. Do I count my nephew who came to visit me? On the fifth bullet in Form I, the bullet conveys it much clearer. ‘Count visitors here’ [on Form H] kind of muddies the waters.”

The use of the word “visitor” was confusing enough to one respondent to lead to an error.

• When filling out Form H, Bea (15) incorrectly included her parents who were visiting long-term from Russia and residing at her home because she was confused when interpreting “no other place to sleep most of the time” in the second bullet. She included them because she interpreted the second bulleted item as referring to visitors having any other place they sleep most of the time in the United States. When presented with Form I, she understood the wording in the fifth bullet much more clearly, and properly left the names of her parents off of the form.

According to the stated preferences and observed performances of respondents, the wording of the fifth bullet on Form I is superior to the wording of the second bullet on Form H. The roster box should be phrased with “no permanent place to stay” and “staying here” instead of “visitors” and “sleep,” which were often seriously confusing to respondents. Overall Neither version of the roster box was clearly superior to the other. The purpose of the roster box is to clarify to respondents who should be counted at their address and thus reduce the number of people not counted at all or counted twice. The mixed results must be weighed to favor accurate counts more than anything else. Visually, respondents seemed to mildly prefer two sub-sections (Form H) to three (Form I), but neither layout seemed to directly cause any errors in completion. The main source of negative performance came from the wording of the second bullet on Form H. Based on these observations, we make the following recommendations. Recommendation 2: Consideration should be given to organizing the roster box criteria so that there are only two highlighted sub-sections of bullets (as on Form H), instructing who to include in the first section and then who to exclude in the second. Having three sub-sections (as on Form I) could hinder respondents’ understanding of the instructions as they switch back and forth. However, a lack of strong evidence makes this a secondary recommendation. Further testing may be needed.


20

Recommendation 3: The wording of the second bullet in the roster box on Form H should be abandoned. Specifically, the terms “sleep” and “visitors” caused confusion and error and should be avoided in describing those people. The wording used in the corresponding fifth bullet on Form I is a much clearer explanation that helped reduce errors in counting from respondents living in non-traditional households. We offer a further recommendation that combines the positive aspects of the roster instructions from both forms. Recommendation 4: Consideration should be given to replacing the fifth bullet on Form I with the wording of the second bullet on Form H, by using the two-section layout present on Form H. The first section should contain the “Include” criteria. The second section should contain the “Do Not Include” criteria. A possible revision to the roster instructions that combines the better testing components of Forms H and I might read something like this: (Box 1)

• Count all people, including babies, who live and sleep here most of the time.

• Also, count anyone who has no permanent place to stay and is staying here on February 15, 2005. Otherwise, he or she may be missed in the Census.

(Box 2)

• Do not count anyone living away either at college or in the Armed Forces. • Do not count anyone in a nursing home, jail, prison, detention facility, etc. on

February 15, 2005.

• Leave these people off your form, even if they will return here after they leave college, the nursing home, the military, jail, etc. Otherwise, they may be counted twice.

Question 5: Identifying Person 1 In the past two sets of interviews (Dillman, Parsons and Mahon-Haft 2004; Dillman, Mahon-Haft and Parsons 2004), it was evident that some respondents were confused about who was supposed to be listed as Person 1. Some respondents even omitted themselves as a result. Thus, the third main test objective was to learn if respondents reacted differently to two revised versions of the instructions prior to Question 5 regarding who to count as Person 1. The simpler language of Question 5 used for both forms on this test explained that the owner or renter of the residence should be listed as Person 1 only if they live at that address. Many of the interviews were conducted with people who occupied a residence but did not own or pay rent. We recorded respondents’ questions, hesitations and comments when reading and comprehend ing these questions.


21

Question 5 on the front page of Form I reads:

• Next, if the owner or renter of this house or apartment lives here, print the person’s name below. If an owner or renter does not live here, print the name of any adult living here. This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)

Question 5 on Form H reads:

• Next print the name of a person who lives here and in whose name

this house or apartment is owned or rented. If an owner or renter does not live here, print the name of any adult living here. This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)

The wording of both of these revised instructions seemed somewhat difficult for respondents to understand at first. Many people exhibited some sort of initial confusion or hesitation before responding, evidence that the question was not particularly easy for respondents to process and answer in either format. This was especially true on the first form that they received and completed because they had not yet developed a sense of what the form was asking (as they had by the time they reached that question on the second form they completed). Odd-numbered respondents received Form H first; even-numbered respondents received Form I first.

Initial Confusion or Hesitation Experienced on Question 5 of: Form H (n=22) 37% (Interviews 1, 3, 7, 9, 11, 13, 15, 21) Form I (n=22) 32% (Interviews 2, 4, 8, 9, 10, 13, 16)

After respondents completed both forms, we specifically directed their attention to the two versions of Question 5 and asked them which version made more sense. Despite some degree of confusion experienced on both forms, respondents clearly preferred Form I.

Is one of these formats clearer than the other one? (n = 22)

Form H 23% Form I 73% Neither 4%

Respondents preferred Form I over Form H for a number of reasons, which are discussed below. “In whose name this residence is owned or rented” All eight respondents we recruited for occupying a residence without owning or paying rent reported that Question 5 on Form I was clearer than on Form H. The primary rationale was that the phrase on Form H, “in whose name this house or apartment is


22

owned or rented,” was confusing. Since these eight respondents did not pay rent or own the place where they lived, this phrase made them think of writing the name of the realty company, university, or church that afforded them housing.

• Isaac (8), a clergyman living in church-provided housing, reported that “in whose name” had “the connotation of a legal contract.”

• Nicole (11) read the bold instructions on Form H twice and said, “OK…owned or rented. We don’t own it, we rent it. Our situation is part of a package deal; the house comes with the job.” (Her Church provides housing for her and her family.) She said she eventually realized from the information in the parentheses that this question was simply asking her for the head of the household.

The response provided by Betty (1) merits a brief discussion.

• Betty’s living situation is particularly unusual. She and her husband are temporarily living in a motel due to a recent house fire, but own a home in the Puget Sound area of Western Washington. They plan to move to their home permanently when Betty’s husband retires later this year. She receives mail both at her home in Seattle and a PO Box in the Pullman/Moscow area. Betty wrote the name of the motel in for Person 1 on Question 5 of Form H. She then wrote the sex, age, and race of her husband to the remaining questions for Person 1 and did not realize she made a mistake until she turned the page and read, “What is Person 2’s name?” Afterwards, she claimed that if the Census form arrived in her mailbox at her Puget Sound Area home, she would have filled it out as if she were living there and indicated that Person 1 sometimes lives and stays somewhere else (in Question 10), by marking “To stay at a seasonal or second residence.” She concluded by noting that she may not have printed the motel’s name for Person 1 if she had received Form I first, since the wording of Question 5 is “clearer and ends with ‘print the person’s name below.’ The other one (Form H) was more getting me thinking about who owns the place.”

Other respondents who did not like the phrase “in whose name” said this:

• Nancee (3): “[Question 5 on] Form I is a lot more clear. On form H, I had to read it three times and the words are kind of jumbled. ‘In whose name’ is less clear wording.”

• Eileen (5): “The question on Form I is much clearer. I had to read through [Question 5 on Form H] a couple of times because of the ‘in whose name’ phrasing.”

• Beverly (13): “[With Question 5 on Form I], I automatically think of myself. On form H, the bold statement made me think of someone besides myself. All three of us roommates are on the lease.”

Owner and Renter Writing the Census questions in bolded font has the obvious advantage of making them more noticeable. However, one latent consequence is that attention is drawn away from


23

surrounding text that is not as visually prominent. On both forms, the words “own” and “rent” appear in bold. Respondents encountered some difficulties with these words on both forms either because they did not own or rent their residence (e.g., Dorothy (6), an apartment coordinator lives for free), or they did not read closely the un-bolded, italicized instructions following the first line of Question 5.

• When reading Form H, Dorothy (6) debated between putting the name of a university that owns her residence until she read, “If an owner or renter does not live here, print the name of any adult living here.”

• Tracy (3) hesitated right after she read the words “owned or rented” on Form H, and considered not even filling out the form since a university owns her apartment. She did not notice the less prominent instructions after the first line of Question 5.

• Terra (21) said that she wanted to include her husband as Person 1 because he owned the residence, though she realized she could not because he was overseas serving in the military. Had she noticed the portion of the instruction, “If an owner or renter does not live here…,” she would not have been as confused.

• On Form I, Evelyn (16) appeared to only internalize the word “owner” when trying to decide whom to list as Person 1. She lives in an apartment with two roommates and pays rent to a local realtor. Since she did not know the realtor’s name, she skipped Questions 6 through 8. When she got to Questions 9 and 10, she determined that she made a mistake and reread Question 5. She finally noticed the word “renter” and decided to list herself as Person 1.

The visual presentation of the four statements in Question 5 convey a successive demotion of importance as the font type switches from bold, to italicized, to italicized in parentheses. The fifth statement, “What is Person 1’s name?” printed in bold may have the effect of leading respondents to skip from one bolded statement to another, missing valuable information in between. Carry Over Effect From Question 3’s “Occupied without payment of cash rent” The respondents with unusual occupancy statuses seemed to find the wording of Question 5 in Form I clearer because it lacks the legal rhetoric of “in whose name” and ends with “print the person’s name below.” Although Betty (1) was the only respondent who made the mistake of printing the business name of her current residence, several others expressed this inclination. The response option for Question 3 of “occupied without payment of cash rent” helped two respondents interpret Question 5 on Form H correctly. This option informed Nancee (3) and Eileen (6) that the Census was asking for a member of the household, rather than the name of an apartment company or university providing their housing. “Next, Print” vs. “Next if…, print” Four respondents favored the instructions of Question 5 in Form I because of the word “if.” These respondents said that the phrase, “I f the owner…lives here, print the person’s name below,” conveyed to them that if the owner or renter did not physically inhabit their residence, they should not include the ir name. Nadene (7) and Nicole (11) have their


24

housing provided by their employers. “If” conveyed to them that Person 1 should be the name of someone in their household, rather than the name of their employer (since their employer does not live in their residence). “A” vs. “The” A common complaint respondents made of Form I was the use of the phrase “the owner or renter” in Question 5. Four respondents share ownership or rent with another person(s) they live with. Since Form I asks the respondent to print the name of the owner or renter, these respondents did not know whether to include themselves or the other person(s) who were co-owners or renters.

• Dianna (10) said, “My husband and I are co-owners. On [Form H], it asks for a person, so it doesn’t make me confused over whether they want me to list me or my husband.”

Dianna and three other respondents preferred Form H because it asks them to print the name of a person. From this, they implied that they could list their own name or another person in their residence with whom they share ownership or rent. Overall Forms H and Form I simplified the wording for Question 5 (compared to previous versions) in an attempt to clarify whose information belonged under Person 1. Both versions still caused a significant number of hesitations and many respondents needed to re-read at least a part of the instruction to fully understand it. The bold first sentence drew more attention than the italicized text that followed it, focusing respondents’ attention inadvertently on the terms “owner” and “renter.” This caused problems for many respondents who have their rent paid for by an employer or some other third party, and even led to some errors. One positive observation was that, overall, there appeared to be a reduction in major errors stemming from the wording of this question compared to versions tested in past sets of cognitive interviews (Dillman, Parsons and Mahon-Haft 2004; Dillman, Mahon-Haft and Parsons 2004). The majority (73%) of respondents found Question 5 on Form I to be clearer than Form H for a variety of reasons. Primarily, the difference lay in the confusing phrase “in whose name” on Form H and the addition of “if” to clarify the conditional nature of the first sentence on Form I. However, given the comments about the wording and visual layout of both forms, we believe a new question can be formulated that combines the strengths of both. Specifically, Question 5 on Form I has the advantage of avoiding legal jargon, ending with “print that person’s name below” and including the simple yet helpful conjunction “if.” Respondents who spoke positively of Question 5 on Form H specified that the improvement was its use of the phrase “a person” (rather than “the person” on Form I). We also believe that respondents would be more likely to read all instructions directly before Question 5 if more bold print were used. We recommend changing the third word from “the” to “an,” and to place the first three sentences in bold type.


25

Recommendation # 5: Question 5 should be reworded as follows:

Next, if an owner or renter of this house or apartment lives here, print the person’s name below. If an owner or renter does not live here, print the name of any adult living here . This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)

The wording of this question does not appear to present as much difficulty to those persons who have their housing provided free of charge by their employer or some other third party as the question wording used on Form H. Persons who share ownership or rent of their residence will not experience as much confusion over which person should be listed as Person 1. Finally, the graphical layout of this question may encourage respondents to read all of the statements before proceeding to, “What is Person 1’s name?” because the second and third statements are now in bold type.


26

CHAPTER 4. RESULTS AND DISCUSSION OF SECONDARY OBJECTIVES AND OTHER OBSERVATIONS

INTRODUCTION As a team, the three of us (Dillman, Parsons and Mahon-Haft) have conducted 82 cognitive interviews over the course of the last seven months. Thirty (30) interviews evaluated the 2000 Census Form and a series of optical design issues. An additional thirty (30) interviews evaluated replacement questionnaire considerations. Those interviews revealed a number of significant problems with the Census forms we tested. The Census Bureau responded to a number of the concerns raised in our first two sets of interviews by changing the forms tested during this third set of twenty-two interviews. Because the three primary objectives of these interviews were to observe how persons respond to variations in the Internet option message, the roster instructions, and Question 5, we focused most of our attention on these issues. We did not look as systematically at problems or trends that arose with other aspects of the questionnaires. However, two secondary objectives outlined in the original test objectives were to observe how respondents reacted to a change in Question 3 (where ellipses replaced a dash) and Questions 8 and 9 (where answer boxes were indented). We discuss the details of these changes and observations, in addition to other observations we made about aspects of the Census questionnaires not outlined in the original Test Objectives, in this chapter. THE MASTHEAD The great majority of people in our interviews did not read information printed in the masthead. We believe this occurred because of its use of reverse print. Two respondents who read the masthead noted that one form had the subordinate information on one long line rather than two short lines under “2005 National Census Test.” Both said it looked much better as two lines rather than one line, and made the heading less confusing. “FORM” IS MISSPELLED AS “FROM” ON FORM H Three respondents pointed out that the Internet option message on Form H reads, “You may either answer this from…” (bold added). These respondents were not confused, though one respondent said it “irritated” her. REACTION TO QUESTION 2 Question 2 asks the respondent to report any additional people not included in Question 1. Some people find this question annoying, if not offensive. It has the tenor of being a trick question. This occurs particularly among respondents who have done a conscientious job of reading the roster instructions. Several found it redundant and unnecessary.


27

QUESTION 3: “IS THIS HOUSE, APARTMENT OR MOBILE HOME…” It was our impression that the readability of this question was improved through the addition of the three dots. In our previous two sets of interviews, the question stem ended with a dash, rather than ellipses. In our last report (Dillman, Mahon-Haft, and Parsons 2004), we recommended that ellipses replace the dash because several respondents interpreted the stem as a question by itself. That is, they read it as, “Is this a house, apartment or mobile home?” In these twenty-two interviews, only one respondent indicated that she wanted to put an “a” into the sentence.

• Veronica (17): “I messed up on the questions. In my mind I want to say, ‘Is this A…’ but now I realize that I should have known the dots mean it’s a continuation.”

Only a few people read Question 3 twice or more, in contrast to our previous interviews. Overall, we believe the use of ellipses after this question stem is superior to the previously used dash. QUESTION 6: WHAT IS PERSON 1’S SEX?” One respondent’s comments about the word “sex” merit attention. Veronica (17), a 50-year old woman with some college education, filled out the form as a caregiver for an elderly woman. Veronica did not like using the word “sex” when asking for Person 1’s gender. She asked,

• “Would Census consider changing it to gender? There are a lot of negative connotations to the word ‘sex.’ Gender is clear; most people know what gender is. To these old ladies, sex is an act of procreation; something they did 60 years ago!”

This response may be suggestive of attitudes shared by a significant portion of the population. QUESTIONS 8 AND 9: INDENTATION OF “OTHER” RESPONSE OPTIONS FOR “RACE” AND “ETHNICITY” In previous interviews, we noted that some people read the “Race” question as two or three separate questions, and wondered if that effect stemmed from the character boxes for “American Indian” and “other Asian or Pacific Islander,” providing a visual barrier that suggested separate questions. In an effort to reduce the tendency for respondents to perceive Question 9 as multiple questions, the forms tested in these interviews presented the write- in “other” answer boxes as slightly indented, using an extra space. Compared to previous forms, fewer people read this question as multiple items, although three respondents were momentarily puzzled by whether it was one or multiple questions.


28

• Nancee (3) saw the “Race” question as multiple questions because of the location of the write- in answer boxes. When answering the race of Person 2, she specifically remarked, “It looks like I’m getting ready to answer another question.”

• Nadene (7) read Question 9 and began to scan the answers. She then asked, “Are those the only three choices?” She then noticed the other options that were listed below the write in box for “American Indian.”

• Isaac (8): “Why are there only three options? The ‘American Indian’ box kind of threw me off. It interrupts the flow.”

Despite the hesitations and comments of these three respondents, there is little reason to believe that the indentation of the write-in “Other” boxes increases the tendency to perceive Question 9 as multiple questions when compared to past versions where no indentation was present. None of the respondents in the current round of interviews experienced the substantial confusion observed in previous interviews. QUESTION 10. “DOES PERSON 1 SOMETIMES LIVE OR STAY SOMEWHERE ELSE?” For these interviews, we recruited two persons living in households composed of unrelated individuals attending college. This question was the source of brief puzzlement to both because the first response option following the “Yes” option is, “to attend college.”

• Beverly (20): “Right now Person 1 [her roommate] is living somewhere else to attend college, but she is living here. She has a home [her parents’] on the west side of the state that she goes away to once in a while. Right now, she is at college.” She marked, “For another reason.”

• The second college student, Evelyn (16) noted that she stays with her parents over the summers, but did not hesitate to mark, “staying at a seasonal or second residence.”

We have no suggestion for modification, but wonder whether this situation was taken into account in the formulation of the response alternatives for this question. Would it be more practical to list “to attend college” next to “staying at a seasonal or second residence?” Since this observation is based on only two interviews, it is difficult to conclude that a large proportion of college students receiving a Census form at their college address might be confused as to whether or not to include themselves. However, when coupled with the roster instruction in Question 1 that reads, “Do not count anyone living away…at college,” the “Yes” option of “to attend college” in Question 10 may introduce some confusion for persons who receive a Census form while at college. PAGE 2, QUESTION 2: “HOW IS THIS PERSON RELATED TO PERSON 1?” One of the answer choices for this question is “Roomer, boarder.” One respondent read these options and then flipped back to Page 1.


29

• Beverly (13): “That’s kind of confusing. What does ‘roomer/boarder’ mean?” The individual she listed as Person 2 was her roommate, yet the words “roomer” and “boarder” did not make sense to her. The Census may want to consider including the option of “roommate” with these options to reduce confusion in younger respondents or others who are not familiar with the terms, “roomer” and “boarder.” WHY SOME PEOPLE MAY NOT REPORT BABIES In past Censuses, respondents have mistakenly left the names and characteristics of their newborn babies off the form. We were able to interview three new mothers in this current set of interviews. Two of these three happened to have visitors temporarily staying with them, and the husband of the third was temporarily living in another city. Because of the addition of an explicit instruction in the roster boxes to include all babies, it seemed an appropriate area for additional focus. All three respondents were momentarily perplexed as to how to write the age of their babies into the three-digit box. One mother wrote “00.5” to indicate her child was six months of age. A second mother wrote “1 wk,” and the third person wrote “1” in the right box and “wk” beside it in the light blue area. We see no easy solution to their confusion regarding how to correctly enter a baby’s age. However, each person resolved it, and may even have felt some pride in coming up with a solution. Respondents’ reactions might best be described as amusement rather than frustration. We asked all three mothers if there were any conditions under which they might have left their babies off the form, which appears to have happened frequently in the last Census. All said no. However, the Russian mother (Bea (15)) explained that in Russia, different words are used for persons and babies. That is, one would never use the Russian word “person” to refer to a baby. She suggested that some people might not include a baby because the wording does not indicate that they should do that.

• Bea (15): “It’s a linguistic thing…when [Russians] think about ‘people,’ you don’t think about ‘babies.’”

In an interview we conducted last year (Dillman, Mahon-Haft, and Parsons 2004), a father from Malaysia did not include his baby son on the form, saying clearly that he did not belong. We now wonder if this situation may have involved a similar issue of translation. We also noted in the current interviews that all three mothers listed their baby last on the interview form as a fourth child, second child and only child. Each indicated that was the way they thought of family order in all forms they fill out. It seems plausible that the tendency to list babies last and the use of “Person #” to identify later spaces in the form might interact to produce the missing babies effect. We offer this only as an issue for possible follow-up and not as a conclusive explanation as to why some respondents leave babies off the Census form. However, we do agree that the introduction of “including


30

babies” in the roster instructions will help convey the necessity of inclusion early in the form. INTRODUCTION OF A FINAL QUESTION We have noticed over the course of these 82 interviews that many respondents find it difficult to know when they have actually finished the form. That is, they write the last person’s name on the form, thumb pages back and forth, and finally say something like, “Well it looks like I’m done. Is that it?” We are also intrigued with published work on “usability” of forms that mentions providing an opportunity to correct errors (Donald Norman, The Psychology of Every Day Things, 1988). The Census Bureau might consider including a final question in the Census Form for purposes of bringing closure and transitioning respondents to placing the form into an envelope for the return. If a closing statement were included, we suggest:

“Finally, before you send back this form, please check to make sure everyone you counted in your household in your answer to Question 1 on Page 1 has been listed on this form, and that information has been provided in the appropriate spaces for all persons, including any babies.”

Providing such a final check could both encourage people to look over their completed form to ensure better consistency between the answer they provide for Question 1 (“Number of people living in household”) and the number of persons for whom they provide information and make it clear that they have come to the end of the form. HOW DO THESE FORMS COMPARE TO PREVIOUS FORMS WITH REGARD TO EASE OF COMPLETION? Although respondents made a few errors, it is our conclusion that the current forms represent a substantial improvement over previous forms that we have tested. Overall, the changes in question wording and visual layout that have been made since we cognitively tested the ADI-2A and ADI-4 forms in 2004 (Dillman, Parsons, and Mahon-Haft) should produce significantly fewer errors in completion.


31

CHAPTER 5. RESULTS AND DISCUSSION OF OVERALL FORM

PREFERENCES INTRODUCTION After eliciting preferences and behaviors towards the three design features that were of focal concern in these interviews, we asked respondents their overall form preferences. In addition, we presented respondents with two alternative forms that differed from Forms H and I only with respect to the background color in the roster instructions. Forms K and J contrasted the roster box instructions with a solid light blue color. (Forms H and I highlighted the special criteria on whom to include and exclude in light blue, contrasted with a darker blue background). Below we tabulate respondents’ preferences among all four forms and elaborate on their rationales. FORM H OR FORM I? Respondents were asked:

Overall, if the Census had to choose between one of these two forms to use in the 2010 Census, which form should they use? (n = 22)

Form H 27% Form I 68% Blend Them 5%

As this table shows, two-thirds of the respondents preferred Form I. A large proportion of respondents justified their answers to this question by referring to one of the three main test objectives discussed in Chapter 3. Respondents preferred one or the other form because of the Internet option message, roster instructions, or wording of Question 5.

• Eileen (5): “[Form I] makes you look at the Internet note.” • Nicole (11): “[Form I] because the roster instructions, Question 5 and the Internet

note are all clearer.” • Beverly (13): “Form I because Question 5 and the Internet note are better.” • Terra (21): “Form I. ‘Only if they have no other place to sleep’ [in 2nd bullet of

Form H] is a bad way to say it. I think ‘stay’ means living. The ‘sleep’ thing bugs me.”

One respondent who felt Form H was overall a better form than Form I said this:

• Evelyn (16): “Form H, mainly because of the wording of Question 5. Also the bullets [in the roster instructions] are ‘spaced’ and ‘really clear.’”

Although only one person refused to choose one form over the other (by saying the Census should “blend them”), several more respondents who stated a form preference did


32

so conditionally. These persons generally wanted to combine aspects of one form with the other.

• Tracy (2) preferred Form H, but said, “I like Question 5 on the I form, but the ‘Start Here’ and blocking instructions on H.”

• Nancee (3) chose Form I because it helped her understand how to complete the form, but “visually, Form H is better.” She liked the two-box layout of the roster instructions of Form H.

• Eileen (5) preferred Form I because she found the Internet note more noticeable. However, she liked the layout of the roster instructions of Form H better because the use of two sections conveyed to her that “there are fewer sections to read.”

• Dorothy (6) chose Form H but preferred the version of Question 5 from Form I. • Isaac (8): “Blend them. I like H’s banner and roster instructions, but Question 5

on Form I is the best.” We also wanted input on which of the two forms was more appealing visually to respondents.

Which version looks better? (n = 22)

Form H 41% Form I 55% No Difference 5%

Overall, respondents tended to prefer the visual presentation of Form I. The majority of respondents who liked the looks of Form I said that the Internet message stood out more. Its dark blue color seemed to make the form more appealing and commanded their attention more than the upper left hand portion of Form H. Respondents who preferred the look of Form H generally referred to the layout of the roster instructions.

• Nancee (3): “Form H [looks better], because of the roster box and ‘Start’ instructions.”

• Nadene (7): “[The roster instructions make] Form H look less jumbled and more organized. Form I feels really compressed and the instructions feel longer.”

• Dianna (10): “Form H looks better because there are fewer shaded areas [in the roster instruction section].

• Bea (15): “The [roster] instruction box [in Form H] looks less busy, but the differences are not huge.”

COMPARISIONS OF ALTERNATIVE FORMS: H VS. K AND I VS. J Form H and Form K differed only in the background color of the roster instructions. On Form H, the bulleted “Inclusion/Exclusion” criteria in the roster instructions were contrasted against a light blue background while the surrounding text was contrasted against a darker blue background. On Form K, the “Inclusion/Exclusion” criteria was not highlighted in lighter blue, but contrasted in the darker blue used for the bolded


33

instructions. (See Appendix A, Figure 5.) Forms I and J differed from each other in the exact same way (see Appendix A, Figure 4). Since a main concern of the Census is to maximize the visual presentation of the “Inclusion” and “Exclusion” criteria in the roster instructions in order to increase the likelihood that they are read, we pointed out these small differences between forms to respondents and asked which versions they preferred.

DC-2H and DC-2K: Which of these two forms do you prefer? (n = 22)

Form H 77% Form K 23%

DC-2I and DC-2J: Which of these two forms do you prefer? (n = 22)

Form I 73% Form J 27%

The results in both tables above are virtually identical. Overall, approximately three-fourths of respondents preferred Forms H and I over Forms K and J. Since the only difference between the original and alternative forms was the visual presentation of the “Inclusion/Exclusion” criteria in the highlighted roster instructions, respondents’ preferences generally depended upon their like or dislike for the visually highlighted criteria. Most respondents preferred Form H and Form I because they found the contrast provided by the light blue background of the “Inclusion/Exclusion” criteria to be optimal.

• Tracy (2): “The roster instructions [on Form H] stand out better [than on Form K]; makes it easier to see.”

• Nancee (3): “[The criteria are] highlighted, so if you don’t read anything else, you can read who you need to count.”

• Dorothy (6): “Form I focuses me a little bit more on these [‘Include/Exclude’] instructions [compared to Form J]. I know I need to read these highlighted boxes; with only one color, this seems like a lot of work.”

• Olive (12): “One big one-colored box [on Form K] conveys ‘blah’ and a lot of text to read and get through. With lighter boxes [on Form H], it makes the bolder font stand out more.”

• Laurie (14): “It’s a lot easier to pick out the bulleted items [on Form I]… a lot easier to read and focus. On [Form J], you’re just going to swim in text.”

• Bea (15): “There is contrast between two colors [on Form H], so I might pay more attention to specific instructions [bullets].”

• Todd (20): “The boxes [on Form I] stop me and make me concentrate more on that category. It slows me down…helps me concentrate.”

• Terra (21): “I like Form H because of the highlighted boxes. It just pulls it out more. It jumps out more at you. They’re instructions and they are important to take into consideration.”


34

As the above two tables show, approximately one-fourth of respondents preferred the alternative forms to Forms H and I. Generally, these respondents thought the less visually complicated one-color layout of the roster instructions made them easier to read and made the form look better overall.

• Betty (1): “[Form K] is all in one color. It’s not separated. With the separated boxes, I’d be more likely to read it out of order.”

• Norma (4): “To me, [the roster instructions on Form K are] easier to read. The boxes [on Form H] are sort of a distraction.”

• Eileen (5): “Form K is a little less busy, or something. It might lead to a different reading sequence because the bold print is more noticeable [on Form K.] The appearance of the shaded regions make [Forms H and I] seem more complicated and make me feel intimidated.”

• Isaac (8): “The highlighted boxes don’t do anything for me on [Form] I. It’s too cut up. On Form I, the roster boxes look much longer because of the Internet note.”

• Dianna (10): “Form J is not as busy; I’d be more tempted to read more of this.” • Veronica (17): “[Form K] flows better and doesn’t break up the bullets. On [Form

K], I would just read it all. On H, I’d read it by section. Do this, then do this, then this. Maybe K has more continuous flow without the off-colored boxes.”

Though the majority of respondents favored the highlighted roster criteria on Forms H and I, the comments of those who preferred the layout of Forms J and K merit concern. To these persons, the highlighted roster section of Forms H and I made the “Inclusion/Exclusion” criteria look more jumbled. More importantly, these comments made by respondents suggest two reasons why Forms H and I could lead to potentially negative consequences in reading and comprehension. Firstly, these respondents report that the highlighted boxes may lead them to read the criteria out of order. Secondly, while the highlighted criteria are good at commanding respondents’ attention, they may reduce the likelihood that respondents will read the bolded text printed in the darker blue background. Since the criteria themselves are perhaps the most important thing in this section for respondents to read, overall we think that the highlighted format on Forms H and I are superior to the layout of the roster boxes on Forms J and K. While the highlighted boxes may draw respondents’ attention away from the bolded text in the darker areas, they appear to be more effective at getting respondents to read the “Inclusion/Exclusion” criteria. Recommendation 6: The roster box should include variations in the background color that visually highlight the bulleted points (on Forms H and I). Overall, most respondents preferred the variations and they seemed to be more effective at encouraging respondents to read all of the information necessary for responding. Any attention diverted from other text as a result did not produce confusion.


35

BEST OF FOUR FOR VISIBILITY OF INTERNET MESSAGE In Chapter 3, we discussed how noticeable respondents found the Internet message on Forms H and I. Here we briefly report the results from a comparison of all four forms in which respondents were asked to judge on which form the Internet message would be most likely to be noticed. The results shown in the table below are not surprising considering the comparisons made between Form H and I reported in Chapter 3. Sixteen of 22 respondents said that the Internet message was more likely to be noticed on Forms I or J — the forms in which the instruction is printed in a dark blue background.

On which form is the Internet instruction most likely to be noticed, or is there no difference? (n = 22)

Form H 5% Form I 55% Form J (alternative to I) 18% Form K (alternative to H) 9% Either H or K 5% No Difference 9%

Some respondents who thought Form I was the best at getting respondents to notice the Internet instruction said :

• Beverly (13): “Form I because it’s highlighted with ‘start here.’” • Laurie (14): “Form I because the blue is bolded and the top part is color

coordinated in different tones of blue.” • Todd (20): “The darker blue [on Form I] really helps.”

Some respondents who favored other forms for visibility of the Internet message said:

• Tracy (2) preferred Form K because of the “bigger contrast between the dark blue and light blue.”

• Nancee (3) said that either Form H or K was best at encouraging notice of the Internet message because “the instruction stands out more on lighter color.”

• Nadene (7) preferred Form H: “The problem with Form I is there is no separate delineation, which makes you go straight to the ‘Start.’ Form H makes it appear as two separate steps, while Form I looks like one step.”

• Bea (15) narrowed her choice down to Forms I and J, and eventually chose J, saying “Form I has too much information highlighted by different colors… too busy.”

Overall, this four-form comparison reinforces the findings reported in Chapter 3 that writing the Internet message in a darker blue background probably will make it more noticeable to respondents.


36

BEST OF FOUR FOR GETTING RESPONDENTS TO READ ROSTER INSTRUCTIONS In Chapter 3, we also discussed respondents’ reactions to the layout of roster instructions in Forms H and I. Near the end of the interview, we presented respondents with all four forms and asked which one they thought was best at encouraging people to read the “Inclusion/Exclusion” criteria.

Which form do you think is most effective in getting people to read the instructions for whom to count? (n = 22)

Form H 36% Form I 36% Form J (alternative to I) 27% Form K (alternative to H) 0%

As shown above, almost three-fourths of respondents (72%) said that either Form H or Form I was most likely to encourage people to read the instructions on who to include and exclude. These respondents said that highlighting the criteria in lighter blue made them much more noticeable, and consequently more likely to be read.

• Dorothy (6): “Form H. There’s just two boxes. It’s simpler and tells you who to count, who not to.”

• Nadene (7): “Form H because of the color differences.” • Isaac (8): “[Form H is] shorter and the lighter color boxes give separation.” • Nicole (11): “Form I. [Respondents will] go to the bold. By having it

separated it seems like there’s not as many as instructions as on [Form] J. It separates it into smaller pieces.”

• Olive (12): “Breaking up the text with not such big blocks [on Form I] makes it kind of step by step.”

• Laurie (14): “Form H because there’s only two boxes and it goes who to count to who not to count and not back and forth.”

• Terra (21): “I is better because of the highlighting. Just where the roster boxes are–it seems heavier.”

The few respondents who believed that Form J was best at getting respondents to read the roster criteria generally found them easier to read.

• Norma (4) eventually chose Form J and said, “Either one [Form J or K] without the [highlighted] boxes, which makes it easier to read.”

• Eileen (5) “[The criteria on Form J are] easy to read… not all hidden in different colors.”

• Veronica (17) “On Form J, ‘Start Here’ and the Internet instruction are emphasized and the bullets appear uncluttered.”

Despite the few respondents who argued that Form J was best at encouraging reading of the roster criteria, we believe that the use of highlighting provided by Forms H and I are


37

most effective. While the added variation in color provided by printing the “Inclusion/Exclusion” criteria against a different color background than the bolded instructions might lead to a less uniform Census form, based on our findings, it appears that highlighting these criteria is the better way to attract respondents’ attention. ALL-AROUND BEST OF FOUR FORMS The final question asked of respondents in these interviews was,

Of these 4 forms, which one should the Census use for the 2010 Census? (n = 22)

Form H 23% Form I 46% Form J (alternative to I) 27% Form K (alternative to H) 0% Combine H & I 5%

These final statements of preferences make it clear that the forms with the highlighted roster criteria were preferred over the forms without the highlighted areas. Three features of Form I (dark blue background for the Internet option message, wording of the fifth bullet in the roster box, and wording of Question 5 for Person 1) were clearly preferred by most respondents. The one feature of the H form that drew considerable favor was the use of two highlighted areas, rather than three. Those respondents who favored Form J generally liked most of the features of Form I, with the exception of the highlighted roster criteria.


38

CHAPTER 6 - SUMMARY OF FINDINGS AND RECOMMENDATIONS

INTRODUCTION Analysis of 22 cognitive interviews conducted in February of 2005 leads us to several conclusions and recommendations about how to best solve some earlier problems with response rate and errors about who to include on the form and where. The three main issues analyzed in the report are as follows:

• Internet Option Message : (1) Would the drawbacks of offering the Internet option to people not comfortable with using the Internet outweigh the potential benefits in response rate? (2) Which version of the Internet note printed on the cover page of the form was most effective at being noticed by respondents?

• Improving the Roster Box: (1) Is it better to divide the bulleted instructions into two highlighted sections or three? (2) Should the sub-sections of rules be highlighted visually by being set against a lighter shade of blue? (3) Which form was worded the most effectively for getting accurate counts of the number of residents at an address?

• Question 5: Do people better understand who to list as Person 1 from the wording of the instructions prior to Question 5 on Form H or I?

SUMMARY OF FINDINGS AND RECOMMENDATIONS In Chapter 3, we discussed our findings about these questions in detail and offered basic recommendations based upon our results and analysis. All three issues tested showed strong evidence that the latest alterations to these forms will significantly improve the response rate and accuracy of some of the most complex households. In this chapter, we briefly summarize these findings and reiterate our recommendations from previous chapters. Internet Option Message First, we found that the potential increase in response rate from offering the chance to complete the form online would not be offset by negative sentiments some people hold regarding transmitting confidential information electronically. Neither the bold URL on the cover letter nor the Internet option message on the cover page offended any respondents. In fact, even people who said they would be more likely to complete the paper version of the Census form embraced having the option of electronic completion. To maximize the effectiveness of the Internet option, the message has to be noticed. Testing two visual designs for the note on the cover page, we found that respondents selected the dark background integrated with the “Start Here” arrow (on Form I) as the optimal visual design for the Internet message. Based upon this analysis, we recommend:


39

Recommendation 1: The message offering the option to complete the Census over the Internet should be printed on the Census form. The note on the form should be presented in the dark blue format that integrates the message with the roster box and draws visual attention to the option. The Internet option message should also appear in the letter, as in the test letters. The URL in the letter should be printed in a different color ink as well, preferably blue, so that it also draws attention and is visually associated with a link to the website. Improving the Roster Box Next, we sought to determine which roster box was easier to understand and would be more likely to reduce respondent errors when counting residents. Our subjects were recruited to test for errors that were likely to arise for complex households. Studying the visual design, organization, and wording, we found that the most confusion and errors stemmed specifically from the wording of the second instructional bullet on Form H. Otherwise, the revisions tested were successful, with very few errors made compared to previous test forms. We found a soft preference (with little performance evidence) for organizing the instructional bullets into two sub-sections (Form H). Several respondents said that it was easier to mentally manage two sections than three, specifically when the first box lists the inclusion criteria (i.e., “Count,” “Count”), and the second box lists the exclusion criteria (i.e., “Do not count,” “Do not count,” “Leave off”). They also tended to interpret the two-box layout has having a simpler visual impression. Recommendation 2: Consideration should be given to organizing the roster box criteria so that there are only two highlighted sub-sections of bullets (as on Form H), instructing who to include in the first section and then who not to include in the second. Having three sub-sections (as on Form I) could hinder respondents’ understanding of the instructions as they switch back and forth. However, a lack of strong evidence makes this a secondary recommendation. Further testing may be needed. Visually, the addition of lighter backgrounds beneath the bullets was a very positive change (on Forms H and I, and not on Forms J and K). Their distinction from the rest of the box drew the respondents’ attention to the essential points and to the roster box as a whole. Because of this, we think the use of highlighting roster criteria should help more people to notice and read them in the future. Since we did not have respondents complete the alternative forms (J and K), there were no observable performance benefits to the visually highlighted areas from this test. However, they did not appear to be distracting or cause problems. The recommendation below is listed as Recommendation 6 because it was included in the four-form comparison discussion in Chapter 5. Recommendation 6: The roster box should include variations in the background color that visually highlight the bulleted points (on Forms H and I). Overall, most respondents preferred the variations and they seemed to be more effective at encouraging respondents to read all of the information necessary for responding. Any attention diverted from other text as a result did not produce confusion.


40

The variation in the wording of the roster box showed a crucial difference between Forms H and I in the way in which respondents understood them. The criterion explaining how to count people without another permanent residence who are staying at the residence where the form is sent on the Census date had previously caused errors in roster counts. On Form H, this criterion was listed as the second bullet. It closely resembled the wording from previous versions in its use of the terms “sleep” and “visitors.” On Form I, this criterion was listed as the fifth bullet. It was revised to refer to where people were “staying” who had “no other permanent place to live” on the date of the Census mailing. Many respondents were still confused and even committed some errors due to the wording of the second bullet on Form H, while the wording of the fifth bullet on Form I was obviously superior at clarifying Census expectations. Recommendation 3: The wording of the second bullet in the roster box on Form H should be abandoned. Specifically, the terms “sleep” and “visitors” caused confusion and error and should be avoided in describing those people. The wording used in the corresponding fifth bullet on Form I is a much clearer explanation that helped reduce errors in counting from respondents living in non-traditional households. Our recommendation for the roster box is a juxtaposed combination of the three differences between forms. It was not completely developed at the time of our preliminary report, so it was not applied to the design of the National Test Form. However, the recommendation remains for possible application at a later date. Our version would maintain the general wording from the fifth bullet on Form I and the highlighted sub-sections of bullets. However, we believe this criterion should be listed as the second bullet when the roster criteria are organized into two sub-sections, “Include” and “Do Not Include.” Recommendation 4: Consideration should be given to replacing the fifth bullet on Form I with the wording of the second bullet on Form H, by using the two-section layout present on Form H. The first section should contain the “Include” criteria. The second section should contain the “Do Not Include” criteria. A possible revision to the roster instructions that combines the better testing components of Forms H and I might read something like this: (Box 1)

• Count all people, including babies, who live and sleep here most of the time.

• Also, count anyone who has no permanent place to stay and is staying here on February 15, 2005. Otherwise, he or she may be missed in the Census.

(Box 2)

• Do not count anyone living away either at college or in the Armed Forces. • Do not count anyone in a nursing home, jail, prison, detention facility, etc. on

February 15, 2005.


41

• Leave these people off your form, even if they will return here after they leave

college, the nursing home, the military, jail, etc. Otherwise, they may be counted twice.

Question 5 Finally, we examined two variations in wording for Question 5 designed to clarify who should be listed as Person 1 and reduce the major mistakes resulting from the third-person language used to word this question on previous forms. Both versions tested were improvements because they both used more “everyday” language than earlier forms and errors were less frequent. However, the mixture of bold and italicized formatting on both forms focused respondents’ attention on the terms “owner” and “renter,” making it difficult for people to whom neither of these terms applied to complete the form. Between the two new forms tested here, Question 5 on Form I was by far the easier one for respondents to understand. Respondents preferred it and performed better on Form I because its wording is less complicated in a number of ways. For instance, the first sentence begins with, “Next, if…”. This seemed to help clarify the conditionality of filling in the owner/renter. As a result, people understood the long instruction more often and showed less hesitancy and confusion. Despite the improvement of the wording of Question 5 on Form I, we believe that some relatively minor alterations could make it even more effective. A few minor adjustments in its wording will make the instructions read more like typical conversation that people would better recognize and understand. Printing more of the instruction in bold should encourage people to read more of it as well. Recommendation # 5: Question 5 should be reworded as follows:

Next, if an owner or renter of this house or apartment lives here, print the person’s name below. If an owner or renter does not live here, print the name of any adult living here . This is Person 1. (On the next page, we will ask about the other people you counted in Question 1.)

SUMMARY OF SECONDARY FINDINGS In Chapter 4, we discussed two secondary test objectives, as well as additional issues and patterns that arose during these interviews. Specifically, we examined whether changes made to Question 3 and Question 9 from past forms reduced hesitation and confusion. The stem of Question 3 used to end with a dash. On the forms tested here, the question stem ended with three dots. This change was made in an attempt to reduce the tendency of respondents to misread the question as, “Is this a house, apartment, or mobile home” rather than, “Is this house, apartment or mobile home --” Though one respondent misread this question, we believe that the use of ellipses following this question stem works much better than a dash. Also, while the write- in “other” boxes for Question 9,


42

asking for Person 1’s race still led some respondents to perceive it as multiple questions, the indentation of these write-in boxes appears to reduce this tendency. The Census Bureau might want to consider focusing more attention on how college students interpret Census instructions. Specifically, the “Exclusion” criterion that reads, “Do not count anyone living away…at college” seems to conflict with the “Yes” option of “To attend college” offered by Question 10. Will college students understand correctly that they are to fill out a Census form? The Census Bureau should consider examining forms completed in past Decennial Censuses to determine if households who omitted babies tended to speak a language other than English. Two people we interviewed who did not speak English as their first language may have been inclined not to count babies because some foreign languages attach different meanings to the words “people” and “babies.” Finally, the Census Bureau might want to consider including a final question that encourages respondents to re-read the completed form and check their answers. One latent consequence of this would be providing closure to those respondents who tend to be unsure about when they are done filling out the form. OTHER FINDINGS ABOUT HOW THE FORM WORKS Since the tested versions of the form are soon to be implemented in the 2005 National Census Test, the form’s overall effectiveness was also an important consideration. Thus, any additional findings that resulted from how they all work as a unit are noteworthy. Overall, the forms tested in these interviews appear to be a significant improvement to those we tested in previous two sets of interviews (Dillman, Parsons, and Mahon-Haft 2004; Dillman, Mahon-Haft, and Parsons 2004). However, we also believe positive changes that incorporate the aforementioned findings and recommendations could be made that would further improve the form’s utility. While some of our findings are rather clear-cut, future tests of potential forms may unearth a greater understanding of how to create the best possible Census form for the 2010 U.S. Census.


43

REFERENCES Dillman, Don A., Nicholas L. Parsons and Taj Mahon-Haft. 2004. “Connections

Between Optical Features and Respondent Friendly Design: Cognitive Interview Comparisons of the Census 2000 Form and New Possibilities.” Social and Economic Sciences Research Center Technical Report 04-030. Washington State University: Pullman. 87pp.

Dillman, Don A., Taj Mahon-Haft, and Nicholas L. Parsons. 2004. “Will a Change in

Appearance Influence How People Respond to a Decennial Census Replacement Form Mailing Package?” Social and Economic Sciences Research Center Technical Report 04-040. Washington State University: Pullman. 120pp.

Norman, A. Donald. 1988. The Design of Everyday Things. New York: Basic Books.


44

APPENDIX A: FIGURES OF MATERIALS SHOWN TO RESPONDENTS

Figure 1: Census Form Cover Letter


45

Figure 2: Form H (DC-2H)


46

Figure 2. Form H (DC-2H) (continued)


47



48



49



50

Figure 2. Form H (DC-2H) (conclusion)


51

Figure 3. Form I (DC-21)


52

Figure 4. Form J (DC-2J)


53

Figure 5. Form K (DC-2K)


54

APPENDIX B: INTERVIEW PROTOCOL

Census Questionnaire Evaluations

A. INTRODUCTION Thank you for coming here today to help us out. The reason we have asked for your help is that every ten years the United States conducts a complete count of everyone who is living in the United States. The Census form comes in the mail to all residences, requesting that people who live there complete and return it. Today I am going to ask you to look at Census forms that are being evaluated for possible use in the 2010 Census. Your reactions to these forms will provide us with information that will help make the form as easy to complete as possible. Okay? B. HAND RESPONDENT CONFIDENTIALITY FORM The first thing I need to do is to ask you to read and sign this consent form. But first let me explain what it is about. This interview is voluntary. It is being conducted by us for the U.S. Census Bureau in Washington D.C. Everything you write on the Census forms is confidential. The only people who can see the information you provide are employees of the Census Bureau and those of us at Washington State University who will be conducting the interviews. We have been sworn by the Census Bureau to keep individual answers confidential, and can be fined if we reveal people’s specific answers in any way that the person can be identified. The statement we are asking you to sign indicates that you have volunteered for this interview. I will also sign it as well since I am the person conducting the interview and want to assure you in writing of my promise to keep all of your information confidential. C. EXPLAIN PROCEDURE In a couple of minutes, I am going to hand you a Census form in an envelope. When I do, I would like for you to talk out loud about your reactions to the form as you read questions and fill it out. I would like to know everything you think about it. Talking out loud about these sorts of things may seem a little unusual, so before I give you the Census mailing, I have a really short PRACTICE mailing. When I give it to you please tell me everything you are thinking as you look at the envelope, and start thinking about what to do with it and the form that is inside of it. I would like to know any thoughts you have about whether it strikes you in a favorable or unfavorable way, whether its clear what to do or not to do, and so forth.


55

D. HAND RESPONDENT PRACTICE MAIL-OUT Okay, please read the questions out loud and tell me everything you are thinking about while you fill it out. (Provide positive reinforcement, e.g., “Good, that’s what we need to know.”) (Encouragement for other information, e.g., “When you do the real Census form just be sure that you tell us about your reactions to everything, the envelope, the way the whole things looks, whether its clear what to do or not do, anything you don’t understand, or anything that seems strange. E. HAND FIRST MAIL-OUT TO RESPONDENT. Now, here is the mail-out that might arrive in the mailbox at the address for which you are completing the Census form in 2010. Please take your time and tell me any reactions you have to everything that you see in front of you. (Note: If person is responding for someone else, e.g., an elderly friend, mark here ? and make sure respondent understands our expectations.)

1. Any reactions to the mailing package:

2. Did they read the cover letter?

? Fully ? Partially ? Not at all

3. Did they react at all to the opportunity to fill out the form electronically (from the

cover letter)? If so, how?

F. ASK RESPONDENT TO FILL OUT THE FORM CONTAINED INSIDE Now, please fill out the census form and talk out- loud about your impressions of it. We would like for you to read whatever you would read at home while filling it out; however, if there is anything you wouldn’t read, don’t read it here. We’d like for you to fill it out just like you would at home, except that you should talk out- loud about it, and anything you read to yourself should be read out- loud. Please go ahead.


56

Probes that might be used: • What are you thinking right now?

• Remember to read aloud for me—it’s up to you what you read, but

whatever you decide to read please do out-loud so I know what you re looking at it.

• Can you tell me more about that?

• Could you describe that for me?

• Don’t forget to tell me what you are thinking as you do that.

RECORD RELEVANT COMMENTS, ERRORS, HESITATIONS AND OTHER INDICATORS OF POTENTIAL PROBLEMS DURING COMPLETION (TO BE USED TO FRAME FOLLOW-QUESTIONS).

1. Which form did respondent receive first? (Odd #s receive H, even #s I) ? H: Light blue internet note ? I: Dark blue internet note

2. Did they read the note about filling out the form electronically?

? Yes ? No 3. What reactions did they volunteer, if any?

4. Did they read the roster instructions? ? Fully ? Partially ? Skimmed ? Not at all

5. Any reactions/hesitations/questions to the roster instructions?


57

6. How did they read Q3 about the type of payment?

? Correctly ? As if it asked what type of dwelling (i.e., “is this a house, apartment…”)

7. Did they read the instructions for whom to fill in as P1? ? Fully ? Partially ? Skimmed

? Not at all

8. Any reactions/hesitations/questions to the instructions for P1?

9. Was the respondent confused at all regarding whether or not to include himself or herself on the form?

? Yes

? No

10. How did they initially read Q9 (the race question)?

? As a single question with multiple write-in spaces ? As multiple questions ending with the write-in spaces

11. Any other significant observations about the first page?

12. Any significant observations about the other P’s?

13. Did they fill in the same number of P’s as the number of people they indicated were residents?

? Yes ? No


58

G. DEBRIEFING QUESTIONS (FIRST FORM)

1. Overall how easy or difficult was the form to complete?

? Very Easy ? Somewhat Easy ? Somewhat Difficult ? Very Difficult

2. Was there anything unclear or confusing about how to fill out this Census form?

? Yes ? No

(If yes) Please explain:

3. If this form arrived at your residence in the mail, how soon do you think you

would respond?

? The same day ? In 1—2 days ? In a week or so ? Two weeks or more ? Not at all

H. ASK RESPONDENT TO FILL-OUT PART OF SECOND FORM Okay, now imagine that this version of the form, which is a little bit different than the other one, came in the same mailing package. Please fill out the first page of this Census form just like you did the first from while you talk out- loud about your impressions of it. Like before, we would like for you to fill it out just like you would at home, except that you talk out- loud about it, and anything you read to yourself should be read out- loud. (Respondent needs to fill out only the first page.) Record observations here:

1. Which form did respondent receive second?


59

? H: Light blue Internet note ? I: Dark blue Internet note

2. Did they read the note about filling out the form electronically?

? Yes ? No 3. Did they notice it was different?

? Yes ? No Record comments:

4. Did they read the roster instructions?

? Fully ? Partially ? Skimmed

? Not at all

5. Did they notice it was different from the first form?

? Yes ? No

6. Any reactions/hesitations/questions to the roster instructions?

7. Did they read the instructions for whom to fill in as P1? \ ? Fully ? Partially ? Skimmed

? Not at all


60

8. Did they notice it was different?

? Yes ? No

9. Was the respondent confused at all regarding whether or not to include himself or

herself on the form?

? Yes ? No

10. How did they initially read Q9 (the race question)?

? As a single question with multiple write-in spaces ? As multiple questions ending with the write-in spaces

11. Any other significant observations about the first page? I. DEBRIEFING QUESTIONS (SECOND FORM)

1. Overall how easy or difficult was the form to complete?

? Very Easy ? Somewhat Easy ? Somewhat Difficult ? Very Difficult

2. Was there anything unclear or confusing about how to fill out this census form?

? Yes ? No

(If yes) Please explain:

3. Did you notice any differences between the two forms you just completed? If so, which ones?


61

4. Which form was easier to complete—the first one, the second one, or was there

no difference?

? H: Light blue Internet note ? I: Dark blue Internet note ? No difference

If respondent chose H or I, ask, “Why did you find that form easier than the other?”

J. COMPARISON OF COMPLETED FORMS

So far, you’ve filled out two different possible forms for the 2010 Census. There are three differences between the forms, some of which you may have already noticed. Now we would like to ask some specific questions about those differences.

Internet Note-

1. The Census wants to make it possible for people to fill out the form on line in the future. To do that, people have to know about it. On which form is the note about completing the form on-line more likely to be read?

? H: Light blue Internet note ? I: Dark blue Internet note

For what reasons did you select that answer?

2. Given the choice, would you rather fill out the form on paper or on-line? ? Paper ? On-line For what reasons did you select that answer?

3. How likely would you be to complete this census form online?


62

? Very likely ? Somewhat likely ? Somewhat unlikely ? Very unlikely For what reasons did you select that answer?

4. If you were to complete this form online, do you think you are likely to do it

sooner, later, or in about the same amount of time as it would take to fill out the paper form?

? Sooner ? Later ? About the same time

5. Do you have access to the Internet from home, work and/or somewhere else or

not at all? (If yes) From where?

? Home ? Work ? Somewhere else ? Not at all

6. (If yes) How many days/week are you on the computer?

7. How much experience do you have with computers?

? A lot of experience ? Some experience ? A little experience ? No experience

8. Do you think most people with Internet access would prefer to fill out the Census

form on the Internet if they had the chance, or would they prefer to return the paper form?


63

? Yes, most people would ? No, most people would not For what reasons did you select that answer?

Roster Box-

9. Which version of these instructions is easier to understand? ? H: Two highlighted sections ? I: Three highlighted sections ? No difference For what reasons did you select that answer?

10. The Census wants to include people who have no permanent place to stay. These

two forms state this in slightly different ways. (Point out statements to respondent - 2nd bullet on Form H; 5th bullet on Form I). Which of these two statements is clearer?

? H: 2nd bullet ? I: 5th bullet


11. Which instruction layout looks better to you?

? H: two highlighted sections ? I: three highlighted sections ? Neither For what reasons did you select that answer?


64

P1 Instructions-

12. Question 5 is a little different on one form that it is on the other. Is one of these formats clearer than the other on what you should do to complete it?

? H: Light blue Internet note (Next, print…) ? I: Dark blue Internet note (Next if the owner…) ? Neither

(If yes) For what reasons did you select that answer?

13. Is either version of the wording confusing? ? H: Light blue internet note ? I: Dark blue internet note

? Neither is confusing


Overall-

14. Overall, if the Census had to choose between one of these two forms to use in the 2010 Census, which form should they use?

? H: Light blue Internet note ? I: Dark blue Internet note For what reasons did you select that answer?

15. Which version looks better? ? H: Light blue internet note ? I: Dark blue internet note



65

K. COMPARISON OF ALTERNATIVE FORMS

Now I am going to show you two alternative versions of the two forms you have already worked on. These versions are identical except for the background coloring of the box with instructions for whom to count as living at your address. Please take a look at these alternative versions of the forms, but there is no need to fill them out at all. Once you have done that, I would like to ask you just a few more questions about all of the forms. Have respondent compare DC-2H and DC-2K

1. Which of these two forms do you prefer? ? ? H: Light blue internet note (DC-2H) ? Alternative H: single roster box background color (DC-

2K)


Have respondent compare DC-2J and DC-2I

2. Which of these two forms do you prefer?

? I: Dark blue internet note (DC-2I) ? Alternative I: single roster box (DC-2J)


Now I would like you to look at all 4 of these forms while I ask you three more questions.

3. On which form is the Internet instruction most likely to be noticed, or is there no difference?

? H: Light blue Internet note ? I: Dark blue Internet note ? Alternative H: Single roster box background color ? Alternative I: Single roster box background color



66

4. Which form do you think is most effective in getting people to read the instructions for whom to count?

? H: Light blue Internet note ? I: Dark blue Internet note ? Alternative H: single roster box background color ? Alternative I: single roster box background color

For what reasons did you select that answer? 5. Of these 4 forms, which one should the Census use for the 2010 Census?

? H: Light blue internet note ? I: Dark blue internet note ? Alternative H: single roster box background color

? Alternative I: single roster box background color


L. That’s all of the questions I have about the forms . Is there anything else you would like to tell us about anything we have discussed today? M. IF EDUCATION AND OCCUPATION OF RESPONDENT ARE NOT KNOWN AT THIS POINT, ASK:

What is the highest level of education you have completed? What is your current occupation?

N. We appreciate your coming in to talk with us. Your help will bring us closer to designing a survey that is easily understood and completed by everyone . Thank you very much for your time and thoughts.


67

O. SUMMARY COMMENTS OF IMPORTANT OBSERVATIONS TO BE RECORDED IMMEDIATELY AFTER COMPLETION OF THE INTERVIEW TO HELP GUIDE THE ANALYSIS AND REVIEW.

technical report 05-022...mail-out package: the cover letter and the first page of the actual form....

Documents