Online Resources

Online Resource 1: Knowledge Network Sampling Methodology

GOVERNMENT & ACADEMIC RESEARCH

KNOWLEDGEPANEL® METHODOLOGY

Knowledge Networks has recruited the first online research panel – KnowledgePanel®– that is representative of the U.S. population. Panel Members are randomly recruited by telephone and mail surveys, and households are provided with access to the Internet and hardware if needed. Unlike other Internet research panels sampling only individuals with Internet access volunteering for research, KnowledgePanel is based on a sampling frame which includes both listed and unlisted numbers, those without a landline telephone and is not limited to current Internet users or computer owners, and does not accept self-selected volunteers.

Knowledge Networks selects households using random-digit dialing (RDD) or by using address-based sampling. Once a person is recruited to the panel, they are contacted primarily by e-mail (instead of by phone or mail). This permits surveys to be fielded very quickly and economically. In addition, this approach reduces the burden placed on respondents, since e-mail notification is less obtrusive than telephone calls. Most respondents also find answering Web questionnaires to be more interesting and engaging than being questioned by a telephone interviewer.

Knowledge Networks’ panel recruitment methodology uses the same or similar quality standards established by selected RDD surveys and area probability surveys conducted for the Federal Government (such as the CDC-sponsored National Immunization Survey).

Until recently, KnowledgePanel’s probability-based recruitment had been based exclusively on a national random-digit dial (RDD) frame. In 2009, Knowledge Networks added address-based sample (ABS) frame (to supplement the RDD frame) in response to the growing number of cell-phone only households that are outside of the RDD frame. ABS involves probability-based sampling of addresses from the U.S. Postal Service’s Delivery Sequence File. Randomly sampled addresses are invited to join KnowledgePanel through a series of mailings and in some cases telephone refusal conversion calls when a telephone number can be matched to the sampled address. Invited households can join the panel by one of several means: by completing and mailing back a paper form in a postage-paid envelope; by calling a toll-free hotline maintained by Knowledge Networks; or by going to a designated KN Web site and completing the recruitment form at the website. After initially accepting the invitation to join the panel, respondents are then profiled for demographics and maintained on the panel using the same

procedures established for the RDD-recruited research subjects. Respondents sampled from the RDD and ABS frames are provided the same privacy terms and confidentiality protections.For the RDD-based sampling, Knowledge Networks utilizes list-assisted RDD sampling techniques on the sample frame consisting of the entire United States residential telephone population. Knowledge Networks excludes only those banks of telephone numbers (each consisting of 100 telephone numbers) that have zero or one directory-listed phone numbers. Two strata are defined using 2000 Census Decennial Census data that has been appended to all telephone exchanges. The first stratum has a higher concentration of Black and Hispanic households and the second stratum has a lower concentration of these groups relative to the national estimates. Knowledge Networks’ telephone numbers are selected with equal probability of selection for each number within each of the 2 strata, with the higher concentration Black and Hispanic stratum being sampled at approximately twice the rate of the other stratum. Note that the sampling is done without replacement to ensure that numbers already fielded by Knowledge Networks do not get fielded again.

Knowledge Networks is able to recover a valid postal address for about 60%-70% of all telephone numbers in its samples. The telephone numbers for which an address is recovered are selected with certainty; between one-half and one-third of the remainder was subsampled randomly depending on the recruitment period up until July 2005. In May 2007, sub-sampling at a rate of 0.75 of non-address households was implemented. The households for which we have an address-matched telephone number are sent an advance mailing informing them that they have been selected to participate in KnowledgePanel 7 to 9 days prior to the recruitment telephone call.

Following the advance letter, the telephone recruitment process begins for all sampled phone numbers. Cases sent to telephone interviewers are dialed up to 90 days, with at least 10 dial attempts on cases where no one answers the phone, and on phone numbers known to be associated with households. Extensive refusal conversion is also performed. Experienced interviewers conduct all recruitment interviews. The recruitment interview, which typically requires about 10 minutes, begins with the interviewer informing the household member that they have been selected to join KnowledgePanel®.

In addition to the above-documented English-based panel recruitment, in 2008 we constructed KnowledgePanel LatinoSM to provide researchers a capability to conduct representative online surveys with U.S. Hispanic community. Prior to the advent of KnowledgePanel Latino, there did not exist in the U.S. an online panel that represents both the internet and non-Internet Hispanics, and that was representative of that part of the U.S. population able to participate in Spanish-only surveys. The sample for the KnowledgePanel Latino is recruited by a hybrid telephone recruitment design, based on a random-digit dialing sample of U.S. Latinos and Hispanic-surname sample. It is a geographically balanced sample that covers areas that, when aggregated, encompasses approximately 93% of the nation’s 45.5 million Latinos.

The first RDD recruitment to KnowledgePanel was conducted in 1999. At that time, all households recruited were given a WebTV to use for answering surveys. In August 2002, KN

began allowing households to use their own computers connected to the Internet for taking surveys. At this time, recruited households were no longer sent a WebTV if they reported having access to the Internet. Currently, if a household does not have a computer and/or access to the Internet from home, work or school, they are told that in return for completing a short survey weekly, the household will be given a device (a WebTV or Laptop) and free monthly Internet access. All members in the household are then enumerated, and some initial demographic variables and background information of prior computer and Internet usage are collected.

Knowledge Networks attempts to recruit to KnowledgePanel every household member who is 13 years of age or older. For household members aged 13 to 17, we collect consent from the parents or the legal guardian during the initial recruitment interview. If no consent is given, no further direct communication to the teenagers will be attempted.

Prior to shipment, each WebTV or Laptop is custom configured with individual email accounts so that it is ready for immediate use by the household members. Most households are able to install the hardware without additional assistance, though Knowledge Networks maintains a telephone technical support line and will, when needed, provide on-site installation. Panel members may contact the Panel Member Support Department for questions using a toll-free number. The Panel Member Support Department is available Monday-Friday 9AM-12AM EST and Sat-Sun 12PM-8PM EST. The Support Department also contacts household members who do not respond to survey invitations and attempts to restore contact and cooperation. Panel members who have Internet access provide Knowledge Networks with their email accounts and their weekly surveys are sent to that email account.

All new non-Internet panel members are sent an initial survey to confirm equipment installation. For all new panel members, demographic information such as gender, age, race/ethnicity, income, and education are collected in a follow-up survey. This information can be used to determine eligibility for specific studies. This information also eliminates the need for gathering basic demographic information on each panel survey. Once this survey is completed, the panel member is regarded as active and ready to be sampled for other surveys.

Knowledge Networks Key Company Information, Past External Review, Confidentiality and Privacy Protections for Panelists: Documentation for Human Subject Review Committees

© 2009

Online Resource 2: IPCC Items

Instructions (Verbal – Numerical Group) :

The IPCC has been established by the World Meteorological Organization and the

United Nations Environmental Program to assess scientific, technical and socio- economic information relevant for the understanding of climate change, its potential impacts and options for adaptation and mitigation. The IPCC gathers and assesses scientific information that could potentially lead to better understanding of climate change. The conclusions reached by the IPCC are condensed into "Assessment Reports" and distributed publicly.

You will be presented with sentences from reports published by the Inter Governmental Panel on Climate Change (IPCC). Each sentence includes phrases that represent the likelihood of occurrence of some event. The goal of this study is to see how people interpret these sentences. We ask you to give your best estimate of the probability conveyed by each sentence. In other words, we are asking for your best understanding of the probabilities that the authors intended to communicate. There is no right or wrong answer, so we encourage you to give honest estimates of the information represented by each sentence.

Probabilities range from 0% to 100%. The response of 0% indicates that an event is impossible.

On the other hand, a response of 100% indicates that an event is certain to happen. The response

scale provides a continuum for you to provide probabilities between 0% and 100%.

1. Continued greenhouse gas emissions at or above current rates would cause further warming

and induce many changes in the global climate system during the 21st century that would

very likely (greater than 90%) be larger than those observed during the 20th century.

2. It is very likely (greater than 90%) that hot extremes, heat waves, and heavy precipitation

events will continue to become more frequent.

3. The Greenland ice sheet and other Arctic ice fields likely (greater than 66%) contributed no

more than 4 m of the observed sea level rise.

4. Temperatures of the most extreme hot nights, cold nights and cold days are likely (greater

than 66%) to have increased due to anthropogenic forcing.

5. Over the past 3,000 to 5,000 years, oscillations in global sea level on time-scales of 100 to

1,000 years are unlikely (less than 33%) to have exceeded 0.3 to 0.5 m.

6. Reconstructions of climate data for the past 1,000 years also indicate that this warming was

unusual and is unlikely (less than 33%) to be entirely natural in origin.

7. It is very unlikely (less than 10%) that the MOC will undergo a large abrupt transition during

the 21st century.

8. It is very unlikely (less than 10%) that climate changes of at least the seven centuries prior

to 1950 were due to variability generated within the climate system alone.

NOTE: The odd-numbered items are specific prediction items and the even-numbered items are

general items.

Online Resource 3: Items to Measure Attitudes toward climate change

Instructions:

Please rate the following on a 5 point scale, with “1” = Strongly Disagree to “5” = Strong Agree.

Belief in Global Climate Change (BGCC) 1. Global warming is occurring now.

2. I am quite sure that global warming is occurring now.

Personal Experience with Global Climate Change (PE)3. I have already noticed some signs of global warming.

4. It seems to me that temperature is warmer now than in years before.

5. It seems to me that weather patterns have changed compared to when I was a child.

Perception of Causes (PCA)

6. Global warming is mainly due to natural causes, not human activity.*1

7. The main causes of global warming are human activities.

8. Global warming is merely a natural fluctuation, not caused by human activity.*

9. I am quite sure that human activities are responsible for global warming.

Perception of Consequences (PCO)10. There will be some positive consequences of global warming for the environment.*

11. The consequences of global warming will be harmful for the environment.

12. Global warming will bring about some serious negative consequences.

13. The consequences of global warming will be more positive than negative overall.*

1 The items with * are reversed scored.

Online Resource 4: Numeracy Items

Instructions: Please provide your best response to the questions below. You may not use a

calculator for any of these questions.

1. Imagine that we roll a fair, six-sided die 1,000 times. (That would mean that we roll one

die from a pair of dice.) Out of 1,000 rolls, how many times do you think the die would

come up as an even number?

Answer: _____________________

2. In the BIG BUCKS LOTTERY, the chances of winning a $10.00 prize are 1%. What is

your best guess about how many people would win a $10.00 prize if 1,000 people each buy

a single ticket from BIG BUCKS?

Answer: _____________________ people

3. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much

does the ball cost?

Answer: ______ cents

4. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines

to make 100 widgets?

____ minutes

5. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48

days for the patch to cover the entire lake, how long would it take for the patch to cover

half of the lake?

Answer: ______ days

Online Resource 5: Coding for Covariates and Frequencies

Variable Coding Frequency PercentEducation 1='Less than HS' 77 13.85

2='HS' 190 34.173='Some college' 143 25.72

  4='Bachelors degree or higher' 146 26.26

Ethnicity 1='White, Non-Hispanic' 395 71.042=' Black, Non-Hispanic' 52 9.353=' Other, Non-Hispanic' 21 3.784=' Hispanic' 71 12.775=' 2+ Races, Non-Hispanic' 17 3.06

Gender 1='Male' 262 47.122='Female' 294 52.88

Marital Status

1='Married' 285 51.262='Widowed' 42 7.553='Divorced' 69 12.414='Separated' 11 1.985='Never Married' 119 21.46='Living with Partner' 30 5.4

MSA 1=' Metro' 98 17.63  0='Non-Metro' 458 82.37

Region 1='Northeast' 89 16.012='Midwest' 120 21.583='South' 221 39.754='West' 126 22.66

Party 1='Strong Republican' 104 18.712='Not Strong Republican' 52 9.353='Leans Republican' 87 15.654='Undecided/Independent/Other' 23 4.145='Leans Democrat' 114 20.56='Not Strong Democrat' 67 12.057='Strong Democrat' 109 19.6

Religion -2='Not asked' 0 0-1='Refused' 2 0.361='Baptist' 102 18.352='Protestant' 126 22.663='Catholic' 117 21.044='Mormon' 10 1.85='Jewish' 14 2.526='Muslim' 0 0

Variable Coding Frequency Percent7='Hindu' 1 0.188='Buddhist' 4 0.729='Pentecostal' 18 3.2410='Eastern Orthodox' 0 011='Other Christian' 65 11.6912='Non-Christian' 15 2.713='None' 82 14.75

Religiosity -2='Not asked' 0 0-1='Refused' 6 1.271='More than once a week' 59 12.452='Once a week' 118 24.893='Once or twice a month' 49 10.344='A few times a year' 103 21.735='Once a year or less' 73 15.46='Never' 66 13.92

Online Resource 6: Detailed Findings from Analysis of Complete Data

Using the complete data, we re-examined the numerical estimates provided as a function of the

(a) word (Very Likely, Likely, Unlikely, Very Unlikely), (b) presentation format (control,

translation, VN), (c) order of presentation (IPCC items first then attitude survey or vice-versa)

and (d) item content (specific prediction, general statement) using a 4-way ANOVA. The format

and order are between respondent factors and words and content are repeated within-respondents

factors. The order of the tasks and the content of the item did not affect the numerical assignment

significantly.

We found significant differences between words (F3,1641 = 157.54, p < .001). As expected, the

mean probability estimates were the highest for Very Likely, followed by Likely, and then

Unlikely, and lowest for Very Unlikely and all differences between adjacent words are

significant, indicating that the respondents’ answers were meaningful and sensible. We also

found significant differences between presentation formats, (F2,547=3.12, p = .045). Most

importantly, we found an interaction between presentation format and words (F6,1641 = 2.86, p =

0.009). As the plots in Figure 1 show, the differentiation between the words is most (least)

pronounced in the VN (Control) group. We found significant differences between VN and the

average of the other two groups for Unlikely (F1,551 = 10.63, p = .001) and Very Unlikely (F1,551 =

10.45, p = .001).

Very Unlikely (<10%) Unlikely (<33%) Likely (>66%) Very Likely (>90%)0

10

20

30

40

50

60

70

80

90

100

Control (n=193)Translation (n=175)Verbal-Numerical (n=188)Very Likely (IPCC: >90%)Likely (IPCC: >66%)Unlikely (IPCC: <33%)Very Unlikely (IPCC: <10%)

Figure 1 Mean Probability Estimates for the Four Words (and SE bars) as a Function of the Presentation Format and Probability Phrase (Original Data)

Using the complete data, we also re-examined the numerical estimates provided as a function of

the (a) word (Very Likely, Likely, Unlikely, Very Unlikely), (b) presentation format (control,

translation, verbal-numerical), and (c) order (IPCC items first then attitude survey or vice-versa)

and (d) item content (specific prediction, general statement) using a 4-way ANCOVA. The

covariates used are those variables which had significant correlations with the estimates (see

Table 4) – the four attitude scales, numeracy scores, education and party affiliation.

We replicated our key results – We found significant differences between words (F3,1620 = 26.52,

p < .001): as expected, the mean probability estimates were the highest for Very Likely, followed

by Likely, and then Unlikely, and lowest for Very Unlikely. The differences between presentation

formats, (F2,540=1.41, p > .05) is not significant overall but, most importantly, the interaction

between presentation format and words (F6,1620 = 6.89, p < 0.001) is significant. There are

significant differences between the VN format and the other two formats (Control, Translation)

for Very Likely (F1,542=7.52, p =0.006); Unlikely (F1,542=4.11, p =0.043) and Very Unlikely

(F1,542=8.47, p =0.004). There are also significant differences between the Control and

Translation groups for Very Likely (F1,542=5.85, p =0.016) and Likely (F1,542=3.97, p =0.047).

Figure 2 plots the least squares estimates of the mean estimates (adjusted for all the covariates),

and shows that the differentiation between the words is most pronounced in the VN group.

Very Unlikely (<10%) Unlikely (<33%) Likely (>66%) Very Likely (>90%)0

10

20

30

40

50

60

70

80

90

100

Control (n=193)Translation (n=175)Verbal-Numerical (n=188)Very Likely (IPCC: >90%)Likely (IPCC: >66%)Unlikely (IPCC: <33%)Very Unlikely (IPCC: <10%)

Figure 2 Adjusted Mean Probability Estimates (and SE bars) as a Function of Presentation Format and Probability Phrase (Original Data).

Online Resource 7: Why are the Probability Estimates so Regressive?

A surprising result in the current paper is that the mean values of the four terms used are more regressive than in most previous papers. For example Table 1 compares the mean values with those obtained for the same words and in the same context in a sample of over 450 individuals in the UIUC community (Budescu, et al, 2009).

Table 1: Mean and Standard Deviations of Estimates for Each Word in the UIUC sample (Budescu et al. 2009) and the current sample

UIUC Sample Current SamplePhrase Mean

(No context)Mean

(IPCC Context)Mean

(IPCC Context)Very Likely 82 83 62Likely 71 67 54Unlikely 28 34 44Very Unlikely 18 30 41

There are several differences between this sample (and administration) and most others. One important difference is related to the elicitation methodology. In most published studies on quantification of terms (including the Budescu, et al, 2009 study) the respondents came to a laboratory and were ran (individually or in small groups) under the direct supervision of research assistants. The current study was done online at home and, it is possible that under these circumstances people are a little less careful and attentive to the details. Another difference between the samples, which we study more carefully here, is the composition of the study population.

Compared to the UIUC study (Budescu, et al. 2009) the participants in the present study are older (Mean age 48 compared to 27 at UIUC) and there were slightly more men (41% vs. 38%). Most importantly there are significant differences in their views of GCC. Table 2 summarizes the means and standard deviations of 4 scales measuring perceptions of GCC in the two samples. The scores on all scales are on 5-point scales. The present national sample has more variability and (significantly) lower scores (between 0.64 SDs and 0.90 SDs on the various scale). Thus, they are, on average, more skeptical about GCC.

Table 2: Means and Standard Deviations of Scales of Perceptions in GCC in the UIUC sample (Budescu et al. 2009) and the current sampleScale UIUC Sample (n = 456) Present sample (n=553) t

(d.f. >700)Cohen’s d

M SD M SDBCGG 4.32 0.92 3.32 1.25 14.3* 0.90PE 3.87 0.98 3.16 1.08 10.8* 0.68PCA 3.97 1.02 3.11 1.13 13.6* 0.86PCO 3.98 0.76 3.46 0.86 10.2* 0.64

Another source of possible differences is self-selection of respondents. We compared the key demographic characteristics of the 556 respondents to the 285 people who were contacted but declined to participate. The two groups are not different in terms of (a) their gender composition

, (b) the region of the country where they reside , (c) their

stated political affiliation , and (d) their religion . However, they differ on other dimensions: First, they are older (mean age = 47.8) then the non-respondents (mean

age = 39.6) . The ethnic composition is also different with a higher proportion of whites among the respondents (71% vs. 55% among non-respondents) and a higher proportion of blacks and Hispanics among the non-respondents (39% vs. 22% among the respondents).

The respondents are better educated then the non-respondents with more college graduates (26% vs. 20% among non-respondents) and fewer people with less than high school education (14% vs. 21% among non-respondents). Still the proportion of people with some college education is under 50% which is lower than the typical samples in previous studies that have often used students (e.g. in Budescu, et al. 2009, 60% of the respondents were students and a portion of the non-students were college educated professionals).

To examine the possibility that education (and the correlated level of numeracy) affect the nature of the responses we contrasted two extreme groups of respondents. Respondents in the “Low” group, respondents have high school or lower education and have a numeracy score of 2, or less, out of 5 (n=230, 41.37% of the respondents in the analysis). In the “High” group the respondents have college (or higher) education and numeracy scores of 3 or higher (n=126, 22.68% of the respondents in the analysis). Table 3 presents the mean estimates for each word in the two groups, across the three experimental conditions

Table 3: Mean and Standard Deviations of Estimates for Each Word in the Low and High GroupsGroup N Word Mean Standard Deviation

Low numeracy & low education

230 Very LikelyLikelyUnlikelyVery Unlikely

59.552.544.841.6

25.320.419.220.4

High numeracy & high education

126 Very LikelyLikelyUnlikelyVery Unlikely

68.257.040.931.1

26.617.620.325.9

It is apparent that the estimates in the High group are less regressive than the Low group and are much closer to the regular estimates.

presents the mean probability estimates for each word for the “High” and “Low” groups separately for each presentation format. The probability estimates are more differentiated for the “High” group compared to the “Low” group for all presentation formats. And, consistent with our hypothesis the benefit is most pronounced in the dual (verbal-numerical) format.

Very Un-likely

Unlikely Likely Very Likely0

10

20

30

40

50

60

70

80

43.842.2

50.2

55.9

34.5

48.3

53.7

63.5

Control Low

Control High

Mea

n Pr

obab

ility

Esti

mat

es

Very Un-likely

Unlikely Likely Very Likely0

10

20

30

40

50

60

70

80

4749.6

55.9

63.7

34.2

39.5

59.6

70.8

Translation Low

Translation High

Very Un-likely

Unlikely Likely Very Likely0

10

20

30

40

50

60

70

80

34.6

42.7

51.4

58.7

23.1

33.7

58

70.4

Verbal-Numerical LowVerbal-Numerical High

Figure 3 Probability Estimates of the High and Low Numeracy Groups for the Control, Translation and Verbal-Numerical Formats.