incorrect knowledge or miseducated guesses? evaluating
TRANSCRIPT
Incorrect Knowledge or Miseducated Guesses?
Evaluating Survey Measures of Political Misperceptions
Matthew H. Graham
July 15, 2020
PRELIMINARY DRAFT
Are survey respondents who provide the incorrect answer to factual questions misinformed? Acrossa wide range of political topics, I show that the average respondent who answers incorrectly issubstantially less certain about their answer than the average respondent who answers correctly.“Don’t know” response options do little to close this gap. Even among those who claim to beabsolutely certain, a latent belief difference exists. Using panel data and a revealed preferencemeasure, I show that when beliefs are elicited a second time, the average respondent who initiallyclaims to be absolutely certain of the correct answer often indicates 90 percent certainty on a 50 to100 scale. For those who claim to be certain of the incorrect answer, the second measure rarely tops75 percent certainty. Claims to be certain of the wrong answer to factual questions about politicizedcontroversies are no more robust than claims to be certain that the incorrect party controls the U.S.House of Representatives. Incorrect answers represent a mix of blind and “miseducated” guesses,but even those who claim to be certain of the wrong answer are not generally misinformed about thefacts.
Political misinformation is on the rise. Political leaders appear to feel increasingly untethered to any
standard of truth, and malicious actors frequently promote outright false information, especially in online
media. For research in this area, a basic descriptive question concerns the prevalence of misperceptions
or the frequency with which members of the public are misinformed. Researchers frequently use survey
data to examine this question, but rarely attempt to distinguish respondents who have been misinformed
from those who hazard an uncertain guess. For the minority of studies that draw such a distinction
(e.g., Kuklinski et al. 2000; Pasek, Sood and Krosnick 2015), survey methodological research provides
little insight as to whether claims to be absolutely certain about the wrong answer represent “incorrect
knowledge” (Hochschild and Einstein 2015, 10), “miseducated guesses,” or mental coin flips.
This paper evaluates the degree to which respondents who offer incorrect answers to factual ques-
tions about politics are misinformed. The evaluation begins by providing new evidence on the predomi-
nant analytic posture in research on the prevalance of political misperceptions, which does not distinguish
highly certain beliefs from educated guesses or random responding. The results show that the typical
incorrect response is stated with substantially less confidence than the typical correct response, and is
closer to a coin flip than to incorrect knowledge. This is true even in categories in which misinformation
and misperceptions are widely expected — politicized controversies (e.g., Barack Obama’s birthplace),
social and economic conditions (e.g., the unemployment rate), and the government’s fiscal situation
(e.g., foreign aid spending). Neither splitting the results by party nor providing a “don’t know” option
meaningfully alters these results.
In light of concerns that “respondents frequently answer factual questions even when they do not
know the answers,” researchers on misperceptions and misinformation have shown increasing interest in
“distinguishing the genuinely misinformed from the guessing uninformed” by “ask[ing] about people’s
confidence in their answers to factual questions” (Kuklinski et al. 2000, 793). To check that certainty
scales capture meaningful variation in certainty, the paper presents a three-part validation exercise that
shows that certainty scales reliably predict accurate responses, response stability, and a measure that
reveals respondents’ probabilisic beliefs through costly choices.
Despite the general promise of certainty scales, the validation exercise also highlights a previously
unrecognized feature: they are much better at identifying respondents who are informed than they are
at identifying respondents who have been misinformed. When beliefs are measured a second time, the
average respondent who claimed to be absolutely certain of the correct answer often assigns a probability
of 0.90 or more to their original choice (on an 0.5 to 1 scale). But for the average respondent who claimed
to be absolutely certain of the incorrect answer, this figure rarely rises above 0.75. The average respondent
who claims to be certain that the wrong party controls the House of Representatives displays a stronger
tendency to stick with their original response than the average respondent who claims to be certain that
Barack Obama has not released his birth certificate. Even those who claim to be certain of incorrect
1
answers are largely making spontaneous, miseducated guesses, not revealing incorrect knowledge.
These results highlight a fundamental gap between misinformation in the information environment
and research that classifies respondents who answer incorrectly as “misinformed.” Anecdotal evidence
makes clear that some people genuinely believe outright false, apparently-factual claims. Yet as a tool
for generalizing from these anecdotes, surveys are a remarkably unreliable instrument, inflating both the
apparent prevalence of misperceptions and the apparent depth at which respondents hold them. These
inconvenient features of survey technology have broad implications not just for research that attempts
to document the degree to which the public has been misinformed, but for other research that leans on
strong interpretations of incorrect answers, including research on the predictors of misperceptions and
conspiracy beliefs, efforts to correct pre-existing misperceptions, and studies of the downstream effects
of corrective information on attitudes.
Survey Measures of Misinformation
Survey-based studies of political misinformation often treat all correct and incorrect answers to
survey questions as evidence that the respondent has been informed or misinformed (e.g., Delli Carpini
and Keeter 1996; Hochschild and Einstein 2015). A critique of this approach holds that it fails to
distinguish the misinformed from the uninformed. This perspective originates with Kuklinski et al.
(1998, 2000) and Hofstetter et al. (1999), who wrote at a time when misinformation was only beginning
to be recognized as a problem in American politics. To distinguish the “guessing uninformed” (who “do
not hold factual beliefs at all”) from the “genuinely misinformed” (who “firmly hold beliefs that happen
to be wrong”), Kuklinski and colleagues recommend asking respondents how certain they are about their
answers (Kuklinski et al. 2000, 793). As concern over political misperceptions has grown, scholars have
increasingly taken interest in respondents’ confidence in their answers to factual questions (e.g., Flynn
2016; Graham 2020a; Jerit et al. 2020; Lee and Matsuo 2018; Marietta and Barker 2019; Pasek et al.
2015; Peterson and Iyengar 2020). Luskin et al. (2018) call methods that focus on those who claim to
be certain a “24 carat gold standard” for identifying misinformed respondents.
In what sense have respondents who claim to be confident about the wrong answer been misin-
formed? One possibility is that they have been misinformed about the facts themselves. This is consistent
with the bulk of research on misinformation in the information environment, which defines misinforma-
tion in terms of outright false, purportedly-factual claims and showcases such claims in its motivating
examples (e.g., Lewandowsky et al. 2005, 2012; Thorson 2015a; Swire et al. 2017).1 Consistent with
research on misinformation in the information environment, respondents who are certain and wrong are
said to “firmly hold the wrong information” (Kuklinski et al. 2000, 792) or “hold a false or unsupported
belief about the answer” (Flynn et al. 2017, 127). At face value, these accounts — as well as the respon-
1These definitions often encompass claims that are presented as true, but later turn out to be false.
2
dents’ own claims to be absolutely certain about their answer — would seem to indicate that certainty
scales can bridge the fundamental divide in survey research between respondents’ real-world beliefs and
their answers to survey questions. Under this interpretation, respondents who answer incorrectly“hold
misinformation” as a sort of “incorrect knowledge” (Hochschild and Einstein 2015, 10-11) that they later
reveal in response to a survey question
Notwithstanding the substantial degree of focus on outright false claims, authoritative definitions
of misinformation also encompass information that is misleading (e.g., Lazer et al. 2018). Accounts
that describe respondents who claim to be certain about incorrect answers as “misinformed” leave a
role for “inferential reasoning” (Hofstetter et al. 1999, 353), “mistaken inferences” (Flynn et al. 2017,
128) and “translat[ing] ... general notions into more specific ones” (Kuklinski et al. 2000, 795). This
raises the possibility that respondents who claim to be certain of the incorrect answer may be making
“miseducated guesses” that are not actually a product of false information about the fact in question. No
prior research provides an evidence-based evaluation of the difference between these two interpretations,
either for incorrect answers in general or for incorrect answers about which the respondent claims to be
certain.
How could mistaken inferential reasoning lead respondents to claim absolute certainty about factual
beliefs that they did not hold before entering the survey environment? It is widely recognized that
surveys do not generally elicit “decided,” “fixed,” or “true” beliefs that existed before the survey (Flynn,
Nyhan and Reifler 2017, 140; Bullock and Lenz 2019, 326; Berinsky 2017, 317). Instead, surveys induce
respondents to construct beliefs on the spot by “sampling” their perceptions and integrating them into
a summary judgment (Nadeau and Niemi 1995; Tourangeau et al. 2000; Strack and Martin 1987). A
little-recognized implication of such accounts is that the process of “sampling” considerations can affect
the extent to which respondents appear to be certain about their answers. Zaller and Feldman (1992)
describe such a case:
Consider respondent A. His first reaction to a guaranteed standard of living was that it was incon-sistent with American ideals; he was also bothered by the unfairness of supporting those who refuseto work. Yet he worried about letting individuals get ahead on their own, saying that some peopleneed special help and that society has an obligation to help the needy. In the second interview,however, there was no sign of this ambivalence. Respondent A gave six reasons why individualsought to get ahead on their own, including a restatement of his feeling that job guarantees areun-American, without raising any opposing considerations. Respondent A ... went from being anambivalent conservative on this issue to being a confident conservative. (594-596)
A similar phenomenon may affect the extent to which respondents appear certain of the wrong
answers to factual questions. Alongside a set of questions concerning matters on which researchers expect
partisan-biased misperceptions, Graham (2020a) measures respondents’ confidence in their answers to
general knowledge questions of the type that have traditionally been used to measure political awareness.
Despite the face implausibility that many respondents would be exposed to outright false information
about such facts, about one in every ten respondents selected an incorrect answer and indicated that they
3
were “very” or “absolutely” certain about it. Graham (2020a) attributes this to respondents falling into
“traps” set by the response options, e.g., “Nancy Pelosi as the Senate Minority Leader (instead of Chuck
Schumer)” and “the filibuster as the Senate procedure to make budget changes via a simple majority
(instead of reconciliation)” (318). Even as this provides some evidence that claims to be certain can reflect
mistaken inferences, this interpretation ultimately relies on the face implausibility that respondents are
misinformed about these facts. It could be that in other cases, respondents hold false beliefs about the
facts themselves. Examining questions about facts on more contested issues, e.g. the economy, social
conditions like health care and gun violence, and the fight against international terrorism, Flynn (2016)
and Graham (2020a) each find that claims to be certain of the incorrect answer are more prevalent.
The lack of concrete evidence as to whether respondents who answer incorrectly have been mis-
informed leaves a potentially fundamental gap between research on misinformation itself and research
that identifies members of the public as having been misinformed. Accordingly, this paper tackles three
under-studied empirical questions. First, do certainty scales actually measure certainty? Second, does
the average incorrect answer reflect misperceptions or misinformation to the same degree that correct
answers reflect accurate perceptions or knowledge? Today, assumptions about whether respondents who
answer incorrectly are ignorant or misinformed are largely made ad hoc. Third, how successful is the
approach of focusing on respondents who claim to be certain of the incorrect answer? To the extent it
succeeds, does it identify respondents who actually feel they possess knowledge, or does it simply identify
respondents who are a bit more confident in their miseducated guesses?
All of these questions are distinct from another, more-studied sense in which respondents may not
believe their answer, that respondents may choose a response other than their sincere best guess as a way
of expressing partisan sentiments (Berinsky 2018; Bullock et al. 2015; Prior et al. 2015). Evidence that
respondents are, or are not, sincerely reporting their best guess does not provide any direct information
as to their level of certainty.2
Research Design
To examine these questions, I draw upon nine online surveys conducted between 2017 and 2020.
Respondents were recruited through Amazon Mechanical Turk (MTurk), Lucid, and YouGov. Each
survey included a series of factual questions about politics. Questions were drawn from five categories:
• General knowledge. These questions cover facts that have traditionally been used to measurepolitical awareness, including knowledge about institutional rules (e.g., the length of a Senate term),individual officeholders (e.g., what is John Roberts’ job), and party control of political institutions(e.g., which party controls the U.S. House of Representatives). As no extant scholarship argues infavor of misperceptions or misinformation about these topics, they serve as a useful benchmark.
2Although it is sometimes speculated that expressive responding is concentrated among respondents who are uncertain(e.g., Bullock et al. 2015; Bullock and Lenz 2019), the results in Graham (2020b) and Peterson and Iyengar (2020) castdoubt on this conjecture, at least in its extreme form.
4
Table 1: Survey information.
Questions Responseoptions
Certaintyscale
Follow-upmeasure
Anti-cheatingstrategyDate N Vendor G E S F C
Sep. 2017 898 Lucid 8 8 6 2 - 3 Subjective None PledgeMay 2018 619 MTurk - - - - 1 2 Probabilistic None Pledge, catchNov. 2018 988 YouGov - 6 5 2 1 3 Subjective None PledgeJune 2019 1244 MTurk - 3 1 - 4 2 Probabilistic Stability Pledge, catchJuly 2019 2438 Lucid 9 3 1 - 4 2 Probabilistic None PledgeOct. 2019 631 MTurk 3 - - 1 - 2 Probabilistic Stability Pledge, detectDec. 2019 4125 Lucid - 4 1 1 1 2 Probabilistic None Pledge, detectFeb. 2020 532 Lucid 4 4 1 2 - 2 Probabilistic Revealed Pledge, detectMar. 2020 939 MTurk 4 3 - 1 4 2 Probabilistic Revealed Pledge, detect
Total 12414 28 31 15 9 15
• Economic conditions. These questions cover facts that are used to measure economic perceptions,with a focus on statistics that make frequent appearances in the literature (e.g., unemploymentand inflation).
• Social conditions. These questions cover facts about social conditions that are not focused oneconomic conditions (e.g., the crime rate, the abortion rate, and levels of immigration). Thisincludes topic areas that have a clear economic component, but also involve social programs orsocial identities (e.g., the health insurance rate, food stamp use, and gender- and race-based incomegaps).3
• Fiscal situation. These questions cover facts about the federal budget. All of the questions areabout three topics: the federal budget deficit, the national debt, and federal spending on foreignaid.
• Controversies. These questions cover facts about politicized controversies. In most cases, claimshave been made in the public sphere about the specific fact in question that were demonstrablyfalse when they were made (e.g., that Barack Obama never released his birth certificate) or werepresented true but later turned out to be false (e.g., that Robert Mueller had evidence that PresidentTrump personally colluded with Russian agents during the 2016 election). In a few cases, theyconcern controversies whose factual basis is not disputed (e.g., Hillary Clinton’s use of a privateemail server when she was Secretary of State).
A breakdown of the question categories represented in each survey appears in Table 1. Although re-
search on these various categories of questions often proceeds separately, presenting them side-by-side
aids interpretation by showcasing important similarities and differences between them. The individual
questions are introduced in more detail as the results are presented, and the appendix provides the full
text of each question.
After respondents chose their best guess on each question, a certainty scale appeared just below
the question on the same screen. In seven of the surveys, each question had two response options; for
these surveys, the certainty scales were given a probabilistic interpretations using both numerical labels
(e.g., 50 to 100 percent certain) and subjective statements (e.g., “pure guess” to “absolutely certain”).
In the other two surveys, each question had three response options, and the certainty scales were labelled
only in terms of subjective statements.
3Classifying the questions this way helps keep the economic category more clearly comparable to previous research. Inpractice, the two categories are quite similar, and will often be combined below.
5
As the results are intended to be applicable to surveys that do not include explicit measures of
certainty, it was important to provide some assurance that the act of stating one’s certainty level did not
systematically alter respondents’ best guesses. Consequently, in two of the surveys, a group of randomly-
selected respondents was assigned to answer the questions without a certainty scale, as they would on
a standard survey. In neither case did the certainty scale appear to systematically affect responses (see
the appendix).
To further probe the trustworthiness of respondents’ claims to be certain or uncertain, four of the
surveys featured second measurements of the respondent’s beliefs about the same questions. Two, the
MTurk June 2019 and MTurk October 2019 surveys, were panel surveys that recontacted respondents and
asked the same questions at a later date. The second measurement permits the computation of measures
of response stability, the canonical metric for the degree to which survey responses measure attitudes or
beliefs (e.g., Converse 1964, 1970; Ansolabehere et al. 2008). To the extent that respondents’ claims to
be certain or uncertain reflect their underlying perceptions, they should, on average, make a similarly
certain inference when asked to do so at some later point in time. To the extent that respondents’ true
underlying perceptions are less dispositive than they claim, they should display a tendency to “back
off” their initial claims of certainty, much as the largest outliers in a probability distribution have a
well-known tendency to regress to their mean.
The other two surveys with second measurements, both conducted in 2020, use a different validation
strategy with the same basic interpretation. In these surveys, respondents completed a discrete choice
task that revealed their beliefs through a series of costly choices. To make this concrete, consider a
question about whether unemployment increased or decreased over the past year. Respondents were
first asked to choose whether they would rather win a bonus if unemployment increased or win the same
bonus if unemployment decreased. Next, respondents chose between winning $100 if their answer was
correct and a 6 in 10, 7 in 10, 8 in 10, 9 in 10, and 99 in 100 chance of winning the same bonus. The
point at which the respondent “crosses over” from preferring payment for a correct answer to preferring
the lottery is an indicator of how likely they think their answer is to be correct (Hill 2017; Holt and
Smith 2016). As the reward is held constant, tasks of this type have an important theoretical advantage:
the only difference in the expected payoff is the respondent’s self-assessed probability that their answer
is correct, making it robust to risk aversion. At the end of each survey, one respondent was randomly
selected to be eligible for the bonus based on their choices.
The revealed belief task offers two advantages over traditional measures of response stability. First,
as a distinct task that involves costly decisions, it has a great deal of validity as a measure of beliefs.
Second, because it is elicited in the same survey, the revealed belief task mitigates the concern that
response instability could be caused by learning between the two surveys.
Even in a single-shot survey, respondents could still learn the answer by looking it up. To mitigate
6
this concern, the four most recent surveys included a snippet of Javascript that monitored whether the
survey remained visible on the respondent’s screen. All cases of possible information search are dropped
from the analysis. The remaining surveys also took steps to discourage and detect information search.
Prior to the beginning of the factual question, every survey included a pledge not to look up the answers.
Two of the surveys also included “catch” questions that respondents would be unlikely to answer correctly
without looking up the answer. Clifford and Jerit (2016) show that these methods successfuly deter and
detect information search.
Do Certainty Scales Measure Certainty?
To be useful in research on misperceptions, certainty scales must capture meaningful variation in
the degree to which respondents have been exposed to information, as well as the degree to which they
are genuinely certain or uncertain about their answer. This section presents the results of a three-part
validation exercise that confirms that certainty scales accomplish both goals. All of the analysis in this
section pools across the question categories listed above.
Does Certainty Predict Accuracy?
To examine the degree to which certainty scales capture meaningful variation in information expo-
sure, I examine the relationship between certainty and accuracy on general knowledge questions, defined
above as the trivia-style questions about institutions and political figures that have traditionally been
used to measure political awareness. The advantage of these facts is that respondents are unlikely to have
encountered false information about them during the course of their daily lives. Consequently, to the
extent that certainty scales capture meaningful variation in information exposure, it should be the case
that respondents who are absolutely certain almost always report the correct answer, while respondents
who are completely uncertain are no more accurate than chance.
The results of this test show that certainty is a good predictor of information about general knowl-
edge questions. For each of the five surveys that included general knowledge questions, Figure 1 displays
the conditional relationship between certainty and accuracy. The September 2017 Lucid survey used a
five-point subjective scale similar to that used in most extant misinformation research (e.g., Kuklinski
et al. 2000; Pasek et al. 2015). Respondents stating the minimum level of certainty were not much more
accurate than chance, while respondents who claimed to be absolutely certain were correct 85 percent
of the time.
The other four surveys used labelled 50 to 100 scales that are interpretable in probability units.
In all four surveys, respondents who stated probabilities between 0.5 and 0.59 chose the correct answer
between 49 and 54 percent of the time, while respondents who stated a probability of 1 chose the correct
answer between 89 and 96 percent of the time. MTurk respondents seem to report their certainty
7
Figure 1: Probability of correct answer by certainty level, general knowledge questions.
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●
●
● ● ●
●
●
●
● ●
● ●
●
●
Lucid, Sep. 2017 Lucid, July 2019 MTurk, Oct. 2019 Lucid, Feb. 2020 MTurk, Mar. 2020
1 2 3 4 5 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 10.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Certainty
Pr(
corr
ect)
Note: For each of the five surveys that included general knowledge questions, this figure displays the probability of answeringcorrectly (y-axis) conditional on the respondent’s stated certainty level (x-axis). The September 2017 and July 2019 surveysused discrete certainty scales; respondents could only choose the values displayed on the x-axis. The other surveys usedquasi-continuous scales that allowed respondents to select any integer between 50 and 100. These measures are binnedusing the following groups: 0.5; [0.51, 0.59]; [0.6, 0.69]; [0.7, 0.79]; [0.8, 0.89]; [0.9, 0.99]; 1. The appendix presents allestimates in tabular form.
level slightly more reliably than Lucid respondents: while the October 2019 and March 2020 surveys
saw respondents who claimed to be absolutely certain answer correctly 96 and 95 percent of the time,
respectively, equally certain respondents in the July 2019 and February 2020 suveys were correct 92 and
89 percent of the time.
To summarize the relationship displayed in each figure, I used OLS to estimate
Correctij = α+ βcij + εij , (1)
where i indexes respondents, j indexes questions, Correctij is an indicator variable for choosing the
correct answer, and cik is the respondent’s certainty level. c is scaled to range [1/k, 1], where k is the
number of response options on the question. Estimates of the slope parameter, β, check the visual
impression that the certainty scale measures the extent to which the respondent is informed about the
fact. The closer β is to 1, the better the measure. The slopes for the three Lucid surveys were 0.72, 0.79,
and 0.68, while the slopes for the two MTurk surveys were 0.99 and 0.92 (for regression tables, see the
appendix).
For this set of questions, the strong relationship between certainty and accuracy suggests that
certainty scales capture meaningful variation in the respondent’s level of information. However, because
the expectation that accuracy should correspond depends on an assumption that respondents were not
exposed to false information, this strategy cannot be used to evaluate certainty scales on questions where
misinformation or misperceptions are expected. To evaluate the scales on more politically controversial
topics, I turn to two strategies that tap certainty more directly.
Does Certainty Predict Response Stability?
The idea that survey respondents who really believe something should choose the same response
in a second survey has a long history in public opinion research (e.g., Converse 1964, 1970). In this
8
context, we care not just about whether respondents make the same best guess when asked a second
time, but whether they indicate an equal level of certainty. To the extent that respondents genuinely
assign probablility p to their answer, they should do so again when asked a second time, allowing for
some attenuation due to measurement error.
Formally, let ait ∈ {0, 1} be respondent i’s answer choice at time t, and cit ∈ [0.5, 1] be the
probabilistic certainty level that they assigned to this response at time t. In the initial measurment
taken at t = 1, the respondent’s certainty level is equivalent to the probability they assign to their
answer: ci = pinitiali1 . In the second measurement, taken at t = 2, the probability the respondent assigns
to their initial t = 1 response, pinitiali2 , is
ci2 if ai1 = ai2
1− ci2 if ai1 6= ai2. (2)
If beliefs were measured without error, it would always be the case that ci1 = pinitiali2 (see the appendix).
To summarize the relationship between certainty and stability, I compute a similar slope statistic
by using OLS to estimate
pinitiali2k = α+ βci1k + εik, (3)
where values of β closer to 1 indicate greater correspondence, on average, between the beliefs stated in
wave 1 and wave 2.
Two of the surveys included follow-up surveys with identical measures of the respondent’s beliefs.
Respondents to the October 2019 survey were recontacted after one month to answer the same four
questions, three general knowledge items (party control of the Senate, the country Angela Merkel leads,
and the country Viktor Orban leads) and one fiscal item whose correct answer has been the same for
the past five-plus decades (whether more money is spent on Medicare or Foreign Aid). Respondents to
the June 2019 survey were recontacted after one year to provide a second response to four questions
about political controversies (Barack Obama’s release of his birth certificate, Donald Trump’s ‘grab
them’ comment, Hillary Clinton’s use of a private email server while Secretary of State, and Robert
Mueller’s conclusions about Trump’s personal collusion with Russia). Public figures have made outright
false claims about three of these facts, while Clinton’s use of a private email server is not disputed.
Figure 2 displays the results separately for these three categories of questions. The x-axis displays
the certainty level stated in wave 1, while the y-axis displays the average probability assigned to the initial
(wave 1) response when asked again in wave 2. On the general knowledge questions, certainty was highly
predictive of response stability. Among those who assigned a probability of 0.50 to 0.59 to their best
guess in the wave 1 survey, the average respondent assigned a probability of 0.54 to this same response
when recontacted a month later. Among those who had initially stated a probability of 1, the average
9
Figure 2: Response stability by certainty level.
Slope = 0.92 (0.04)
● ●
●
●
●
●
●
Slope = 0.60 (0.09)
●
●● ●
●
●
●
Slope = 0.53 (0.06)
●●
●
●
●
●
●
General knowledge(MTurk, Oct.−Nov. 2019)
Fiscal(MTurk, Oct.−Nov. 2019)
Controversies(MTurk, June 2019−2020)
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Certainty, wave 1
Belief ininitial
response,wave 2
Note: Conditional on the certainty level the respondent stated in wave 1 (x-axis), this figure displays the average probabilityrespondents assigned to their wave 1 response when asked again in wave 2 (y-axis). The dashed grey line indicates therelationship that would realize in the absence of measurement error. These measures are binned using the following groups:0.5; [0.51, 0.59]; [0.6, 0.69]; [0.7, 0.79]; [0.8, 0.89]; [0.9, 0.99]; 1.
probability in the second wave was 0.97, suggesting minimal exaggeration of the degree to which these
respondents really thought they knew the answer. The slope estimate, β̂ = 0.92 (s.e. = 0.03), indicates
a tight correspondence between certainty and response stability on general knowledge questions.
Certainty was also a reliable predictor of stability in the other two categories, though to a lesser
degree. On the fiscal question, respondents who claim to be more certain in the wave 1 survey consis-
tently assigned greater probability to that same response when asked again in wave 2. Relative to the
general knowledge questions, one distinctly different result emerges: the average respondent who stated
a probability of 1 in the first wave only assigned a probability of 0.8 to this same response when asked
again in wave 2. The slope, β̂ = 0.60 (s.e. = 0.09), similarly captures the solid but weaker correspondence
between certainty and stability on these questions.
A similar pattern emerges on the questions about political controversies. Certainty is a strong
predictor of response stability, but the average respondent who stated probabilities of 1 in wave 1 regressed
to 0.87 in wave 2. Further, when contacted a year later, respondents who initially said that they were
almost completely uncertain (probabilities between 0.5 and 0.59) also display a modest tendency to
put more stock in their answer than they initially claimed. This pattern is driven by the questions
about Clinton’s emails and Trump’s “grab them” comments, which respondents overwhelmingly answer
correctly. The average initially-uncertain respondents assigned a probability of 0.76 and 0.66 to their
initial response on these questions, compared with 0.60 and 0.57 on the two questions that required
respondents to reject the false claims that Obama refused to release his birth certificate and that Mueller
reported personal collusion between Trump and Russia. The slope statistic for the controversy questions,
β̂ = 0.52 (s.e. = 0.06), is somewhat attenuated by this pattern.
These results provide assurance that respondents’ claims to be certain capture a substantial amount
of genuine variation in certainty. Even so, response stability has two key shortcomings. First, because
10
response stability takes the same measure twice, one must take for granted that the construct that is
stable is, in fact, certainty. To the degree that a measure of certainty also captures systematic variation
in some other construct, stability may not be evidence that the scales measure certainty itself. Second,
because the surveys are conducted at two different points in time, instability can occur due to genuine
changes in beliefs. This is especially worrisome for questions about social and economic conditions, whose
correct answers can change frequently and on which respondents make frequent use of their day-to-day
experiences in formulating their responses (Graham 2020a).
Does Certainty Predict Revealed Belief?
To address the shortcomings of response stability, I now turn to an alternative strategy for exam-
ining the extent to which certainty scales measure certainty. In the February and March 2020 surveys,
all respondents’ probabilistic beliefs were measured a second time using a series of costly choices. This
strategy adds evidence of construct validity: while talk is cheap, costly choices give respondents an in-
centives to accurately reveal their beliefs. Moreover, because both measures were collected in surveys
that included a technology that detects likely cases of information search (see above), there is minimal
concern that genuine changes in underlying beliefs, due either to learning or information search, distort
the results.
The February and March 2020 surveys had considerable, but not perfect, overlap in terms of
their questions in three categories: general knowledge, economic conditions, and the fiscal situation.
To facilitate more direct comparison between them, analysis of these categories restricts attention to
questions that were included in both surveys: in the general knowledge category, the job or political
office held by two political figures (John Roberts and Jerome Powell), party control of the House of
Representatives, and the difference between reconciliation and the filibuster; in the economic conditions
category, the change in unemployment, inflation, and GDP in the year prior to the survey; and in the
fiscal category, foreign aid spending and change in the federal budget deficit. The March 2020 survey
also featured four questions about politicized controversies: Trump’s “grab them” comment, Robert
Mueller’s findings on Trump’s collusion with Russia, Trump’s claim of unlimited power under Article II,
and Obama’s preemptive admission that the Deferred Action for Parents of Americans (DAPA) program
ignored the law. Figures for each individual question appear later (Figures 6-9).
Figure 3 displays the results by question category and survey. Its interpretation is the same as
the interpretation of Figure 2, with the exception that revealed belief has been substituted for response
stability. The results are also broadly similar. The correspondence between stated certainty and revealed
beliefs is strongest on the general knowledge questions, and is modestly stronger for the MTurk sample
than in the Lucid sample: MTurk respondents who stated a probability of 1 revealed a belief of 0.94 in
this same response (top, middle-left), compared with 0.85 for Lucid responses (top, far-left). The slope
11
Figure 3: Revealed belief by certainty level.
Slope = 0.53 (0.04)
●●
●
●●
●
●
Slope = 0.42 (0.06)
●
● ●
●●
● ●
Slope = 0.75 (0.03)
●
● ●
●●
●
●
Slope = 0.51 (0.03)
●●
● ●
●●
●
Slope = 0.42 (0.05)
●
●●
●● ●
●
Slope = 0.55 (0.03)
●●
●● ●
●
●
Slope = 0.45 (0.03)
●
●● ●
●
●
●
Fiscal(Lucid, Feb. 2020)
Fiscal(MTurk, Mar. 2020)
Controversies(MTurk, Mar. 2020)
General knowledge(Lucid, Feb. 2020)
General knowledge(MTurk, Mar. 2020)
Economic conditions(Lucid, Feb. 2020)
Economic conditions(MTurk, Mar. 2020)
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Certainty
Revealedbelief in
initialresponse
Note: Conditional on the certainty level the respondent stated (x-axis), this figure displays the average probability re-spondents assigned to their response using the measure of revealed beliefs (y-axis). The dashed grey line indicates therelationship that would realize in the absence of measurement error. These measures are binned using the following groups:0.5; [0.51, 0.59]; [0.6, 0.69]; [0.7, 0.79]; [0.8, 0.89]; [0.9, 0.99]; 1.
statistics are printed above each panel.The existence of a reasonably strong correspondence between
respondents’ stated levels of certainty and the beliefs they revealed through costly choices suggests that
certainty scales capture meaningful variation in certainty. Relative to the certainty-accuracy relationship
depicted in the two rightmost panels of Figure 1, the certainty-stability relationship has a steadier upward
slope. The combination of these findings indicates that middling levels of certainty capture meaningful
variation in respondents’ confidence in misleading heuristics.
On the fiscal questions, stated certainty also predicts a substantial amount of variation in revealed
belief (bottom left). The results are again similar to general knowledge questions, with the notable
exception that respondents who claimed to be absolutely certain display a greater tendency to “back
off.” The average MTurk respondent who initially stated a probability of 1 revealed a belief of 0.85
in this same response, about 0.09 lower than the corresponding figure for general knowledge questions.
For the average Lucid respondent, the corresponding figure was 0.76, also about 0.09 lower than the
figure for general knowledge questions. Similar patterns are observed for the other two categories, with
approximately the same difference between the Lucid and MTurk samples.
12
How Certain Are Respondents Who Answer Incorrectly?
Having established that certainty scales measure certainty, the analysis now interrogates the stan-
dard analytic posture in survey-based research on misperceptions or misinformation, which treats correct
and incorrect answers as equally reflective of the respondents’ perceptions. This portion of the analysis
pools all 100,668 unique responses provided by the 12,414 respondents who participated in the nine
surveys,4 permitting statistically precise comparisons between respondents who answer correctly and
incorrectly within each of the five categories of questions considered in the analysis.5 To accommodate
differences between the certainty scales, all measures of certainty are rescaled to range between 0 and 100,
where 0 is the lowest point on the scale and 100 is the highest point. This measure can be interpreted
as a percentage of the width of the certainty scale.
For each of the five question categories, Figure 4 presents the average level of certainty for correct
and incorrect answers, as well as the difference between the two types of responses. General knowledge
questions serve as a useful benchmark for the other four categories. On average, respondents who
answered correctly had a confidence scale score of 70, equivalent to 0.85 on a 0.5 to 1 probability scale.
This means that on average, these respondents were about two-thirds of the way between a pure guess
and absolute certainty. By contrast, respondents who answered incorrectly had an average score of 40,
equivalent to 0.7 on a 0.5 to 1 scale. Incorrect answers were closer to pure guesses than to “incorrect
knowledge.” As a proportion of the scale, incorrect answers are stated with 30 percent greater uncertainty
than correct answers. Relative to one another, respondents who answered correctly were about 1.75
times as certain. In a rough sense, this validates researchers’ traditional assumption that correct answers
represent a mix of knowledge and ignorance, while incorrect answers tend to reflect uncertain guesses.
At the same time, it suggests that mental coin flips are a poor analogy for incorrect responses. 6
As noted above, researchers who suspect misperceptions or misinformation treat incorrect answers
as being fundamentally different than incorrect answers to general knowledge questions. To what extent
is this justified? On questions about social and economic statistics, respondents who answer incorrectly
had an average score of 45, about 5 percent of the scale more confident than incorrect answers to general
knowledge questions. Respondents who answered these questions correctly had an average score of
about 56 — 11 percent more confident than respondents who answer incorrectly, but about 14 percent
of the scale less confident than correct answers to general knowledge questions. Relative to general
knowledge questions, the most distinguishing feature of questions about economic and social statistics
is that respondents who answer correctly are relying more heavily on educated guesses, and less heavily
4These totals exclude respondents in the Sep. 2017 and July 2019 surveys who either did not have a certainty scale orwere allowed to say “don’t know.”
5The appendix shows that the same patterns obtain when the results are split by survey.6This is consistent with Schuman and Presser (1981), who find that even respondents who “provide an opinion on a
proposed law they know nothing about” are not “merely flipping mental coins . . . Respondents make an educated (thoughoften wrong) guess as to what the obscure acts represent, then answer reasonably in their own terms about the constructedobject” (158-59).
13
Figure 4: Average certainty by correct/incorrect answer and partisan valence.
Correct answersIncorrect answers
All respondents
47.5
49.1
45.4
45.1
39.2
70.2
63.9
55.7
56.6
69.9
−22.7 (0.7)
−14.7 (0.7)
−10.3 (0.5)
−11.6 (0.4)
−30.8 (0.5)
Difference (s.e.)
●●
●●
●●
●●Correct answer is congenialIncorrect answer is congenial
46.850.3
45.850.2
45.947.0
45.746.8
75.166.3
68.767.2
58.353.6
60.353.8
−28.3 (1.0)−16.0 (0.9)
−22.8 (1.3)−17.0 (1.3)
−12.4 (0.8) −6.7 (0.8)
−14.6 (0.7) −7.0 (0.6)
0 100
Lowestscalepoint
Midpoint Highestscalepoint
Controversies
Fiscal situation
Social conditions
Economic conditions
General knowledge
Controversies
Fiscal situation
Social conditions
Economic conditions
Average certainty
Cat
egor
y
Incorrect Correct
Note: Pooling across all surveys, this table displays average certainty among respondents who answered correctly andincorrectly. To standardize differences between the surveys, all of the certainty scales are rescaled to range from 0 to 100,where 0 is the lowest point and 100 is the highest point (see text). Triangles are the average for all respondents; squares,for correct answers; circles, for incorect answers. Hollow shapes are incorrect answers; solid shapes are correct answers.Differences calculated using OLS with standard errors clustered by respondent.
on knowledge.
Responses to questions about the government’s fiscal situation follow a similar pattern, but with
modestly higher confidence in both categories. The higher level of confidence is driven by the debt and
deficit questions, on which respondents’ open-ended descriptions of their answers suggest the availability
of fairly stable heuristics. The foreign aid questions perform more similarly to the questions about social
and economic conditions.
When it comes to questions about politicized controversies, the average respondent who answered
correctly had a certainty scale score of 70 — statistically indistinguishable from general knowledge ques-
tions but substantially higher than economic, social, and fiscal questions. Relative to these question
types, correct answers to question about politicized controversies are closer to knowledge. By contrast,
incorrect answers to questions about politicized controversies are offered with only slightly more confi-
dence than incorrect answers to other types of questions. The average scale score of 47.5 indicates 2 to
3 percent greater certainty than repondents who incorrectly answer social and economic questions, and
7 percent greater certainty than respondents who incorrectly answer general knowledge questions. As a
rule, there is nothing all that special about incorrect answers to questions about politicized controversies.
They are delievered with slightly higher confidence than incorrect answers to other question types, but
come nowhere close to representing “incorrect knowledge.”
14
Accounting for partisanship.
Researchers generally expect the strongest misperceptions among those who have a partisan moti-
vation to believe false or misleading claims. It could be that there is something systematically different
about incorrect answers that are congenial to the respondent’s partisanship. To examine this possibility,
the bottom portion of Figure 4 splits respondents according to the partisan valence of the correct answer.
during this period, unemployment was steadily declining under a Republican president. This is congenial
to Republicans and uncongenial to Democrats.
To the extent that partisan congeniality predicts differences in certainty about one’s answer, those
differences are more driven by convenient knowledge than by belief in convenient falsehoods. On questions
about politicized controversies, respondents who have a partisan incentive to answer incorrectly are about
3.5 scale points more confident in their incorrect answers than respondents who have an incentive to
answer incorrectly (50.3 versus 46.8). Among respondents who answered correctly, the corresponding
difference is 8.8 points (75.1 versus 66.3). For the social and economic questions, there is only about a
1 point difference among respondents who answer incorrectly, but a 5 point difference among those who
answer correctly.
Incorporating “don’t know” response options.
The foregoing analysis examined questions without a “don’t know” (DK) response option. Though
this is the standard practice in research on political misinformation (Luskin et al. 2018), it is worth
evaluating whether the usual approach to handling respondent uncertainty would change this picture.
To examine this question, the July 2019 survey included a split-ballot experiment designed to examine
the properties of DK response options.
The results of this experiment demonstrate that providing a DK option does little to narrow the
certainty gap between respondents who answer correctly and those who answer incorrectly (Figure 5).
Without a DK option, the difference in certainty between respondents who answer correctly and incor-
rectly is about 25 percent of the confidence scale. This shrinks to 22 percent of the scale when a DK
option is provided, about one eighth of the difference without a DK option.
The failure of DK response options to close much of the certainty gap between correct and in-
correct answers owes to DK options’ tendency to discourage only those respondents who are very
uncertain from answering the question. This has long been apparent to survey researchers, but in
this context, it aids interpretation to have a more precise sense of how confident DK respondents
would have been if they had provided an answer. Let E[C | ·] denote the conditional average cer-
tainty level among some group of respondents. Although E[C | said DK] is unobservable,7 it can be
7In principle, it would also be possible to ask DK respondents for their best guess, then how certain they were. This wasavoided for two reasons. First, it would require DK respondents to expend more effort than non-DK respondents, whichcould discourage DK responding. Second, the act of saying DK could affect responses to questions about confidence, justas Bishop et al. (1984) find that it affects self-reported political interest.
15
Figure 5: Average certainty by correct/incorrect answer and availability of DK option.
Correct answersIncorrect answers
DK allowed
DK not allowed
57
38
55
50
49
35
52
40
83
76
62
55
75
65
75
66
25.1 (1.7)
37.8 (1.5)
6.6 (1.1)
5.1 (1.0)
26.6 (1.0)
29.6 (0.9)
22.2 (0.8)
25.3 (0.7)
Difference (s.e.)Difference (s.e.)
0 10 20 30 40 50 60 70 80 90 100
Controversies
Economic/social
General knowledge
All questions
Lowestscalepoint
Midpoint Highestscalepoint
Average certainty
Note: Using data from the July 2019 survey, this figure displays average certainty among respondents who answeredcorrectly and incorrectly. For comparability with Figure 4, the certainty scale was rescaled to range from 0 to 100, where 0is the lowest point and 100 is the highest point. The original probabilistic scale ranged from 50 to 100. Hollow shapes areincorrect answers; solid shapes are correct answers. Squares represent the average among respondents who were randomlyassigned not to have a DK option; triangles, the average among respondents who had a DK option and chose to answerthe question. Differences calculated using OLS regression with standard errors clustered by respondent.
estimated under the assumption that people who do provide an answer are equally certain regard-
less of whether a DK option is present (that is, E[C | would not say DK if allowed, DK allowed] =
E[C | would not say DK if allowed, DK not allowed]). Then the average DK respondent’s confidence
level can be expressed as
E[C | DK not allowed]−(E[C | DK allowed]× (1− Pr[said DK | DK allowed])
)Pr[said DK | DK allowed]
, (4)
all components of which are observable (see the appendix for a proof). To estimate statistical uncertainty
about this estimator, I used the block bootstrap, which accounts for the fact that each respondent
contributed more than one observation (Hainmueller et al. 2015).
This formula suggests that the average respondent who said DK would have been much closer
to a pure guess than knowledge, but would not have been completely uncertain. Overall, the average
respondent who said DK would have had a certainty scale score of 20.0 (s.e. = 4.1), or about 0.60 on
the original 0.5 to 1 probability scale. This includes 16.9 on general knowledge questions (s.e. = 4.4),
20.6 on controversy questions (s.e. = 5.8), and 27.6 on social and economic questions (s.e. = 4.9). These
results are consistent with research that shows that DK responses hide only a small amount of knowledge
(Luskin and Bullock 2011; Sturgis et al. 2008).
With these estimates in hand, it is possible to be more precise about why DK options only mod-
estly shrink the gap between correct and incorrect responses. Allowing this fairly uncertain group of
respondents to opt out of answering the question boosts certainty among those who answer correctly by
about 9 percent of the scale, from 65.5 to 74.3. Because respondents who answer incorrectly tend to be
16
less certain to begin with, allowing the most uncertain respondents to opt out boosts certainty among
this group by a larger amount: 13 percent of the scale, from 40.9 to 53.7. Permitting DK responses moves
both types of answers a little closer to absolute certainty, but plenty of uncertainty remains among those
who answer the question.
Certain and Wrong?
So far, the analysis has shown that certainty scales capture meaningful variation in certainty and
that incorrect answers do not reflect “incorrect knowledge” to the same degree that correct answers
reflect true knowledge, regardless of whether a DK option is provided. Though these particular results
are novel, scholars have long recognized that not all incorrect answers are stated with certainty, and have
consequently recommended the use of certainty scales to distinguish the misinformed from the ignorant
(e.g., Kuklinski et al. 2000; Pasek et al. 2015; Graham 2020a). Does this practice isolate respondents who
have been misinformed about the facts themselves, or merely identify a group whose mistaken inferences
were made with greater confidence?
To examine this question, I compare correct and incorrect answers using two of the same outcome
metrics — response stability and revealed belief — that were used above in the undifferentiated assess-
ment of whether certainty scales measure uncertainty. To the extent that one’s claim to be certain of
the incorrect answer reveals “incorrect knowledge,” respondents should stick with their initial response
upon a second measurement, at least to the same degree as respondents who answered correctly. But if
respondents who claim to be certain of incorrect answers display a greater tendency to “back off” their
initial response, claims to be certain and incorrect disproportionately reflect mistaken inferences.
Whereas the previous two sections pooled across questions to draw out general rules of thumb, this
section zooms in on specific questions and analyzes the heterogeneity between them. It draws primarily
on the revealed belief metric collected in the February and March 2020 surveys, which as noted above,
provides the best validity as a distinct measure of beliefs and offers the best protection against learning
and information search.
Beginning with the usual general knowledge benchmark, Figure 6 examines four questions. On
each one, a striking difference between correct and incorrect answers emerges: while certainty is a strong
predictor of revealed beliefs for respondents who answer correctly, it is a weak predictor of revealed
beliefs for those who answer incorrectly. Visually, this is evident from the green circles’ tendency to sit
above the red squares, especially at high levels of certainty. The pattern is most striking on the question
about John Roberts’ job, which asked respondents to say whether he is the Chief Justice of the Supreme
Court or the Secretary of Defense. When asked to make this inference a second time, respondents who
initially claim to be certain that he is the Chief Justice reveal a highly certain belief that this is true. By
contrast, respondents who say that he is Secretary of Defense tend to reveal that their beliefs are really
17
Figure 6: Revealed belief in initial answers by certainty level, general knowledge questions.
●
●● ●
●
●
●
●
●● ●
●
●
●
Correct: 0.70 (0.04)Incorrect: −0.04 (0.10)
● ●●
●
●●
●
● ●●
●
●●
●
Correct: 0.57 (0.04)Incorrect: 0.20 (0.08)
● ●●
●
●
●
●
● ●●
●
●
●
●
Correct: 0.69 (0.06)Incorrect: 0.23 (0.09)
●
●●
●
●
●●
●
●●
●
●
●●
Correct: 0.50 (0.08)Incorrect: 0.05 (0.13)
John Roberts isChief Justice
Jerome Powell isFed Chair
Democrats holdHouse majority
Reconciliationvs. filibuster
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty
Revealedbelief in
initialresponse
Initial responsewas...
●● Correct
Incorrect
split about 50/50 between the two response options, regardless of how certain they claimed to be on the
initial measure.
On the other three general knowledge questions, the same asymmetry emerges, but respondents
display a modestly stronger tendency to stick with their incorect answers. Among those who incorrectly
say that Jerome Powell is the Treasury Secretary (rather than the Chairman of the Federal Reserve),
there is a slight tendency among the more-certain to stick with their answer, suggesting that at least
some people drew on the same misleading heuristics when making the same inference a second time.
The same goes for respondents who incorrectly say that Republicans have the majority in the House of
Representatives (rather than Democrats) and respondents who incorrectly say that the filibuster allows
changes to budget bills via a simple majority vote (rather than reconciliation). Some respondents may
be consistently misled by their hazy knowledge that Republicans currently control much of the federal
government, or that the filibuster has something to do with majorities and the Senate, but those who
claim to be certain reveal themselves not to be misinformed about the facts.
Although no reader likely expects many people to be truly certain that the wrong party controls
the House of Representatives, respondents’ tendency to stick with this claim becomes a useful benchmark
for the other results. The probability that allegedly-certain respondents assign to this possibility on a
second measurement, 0.75, is close to the high water mark for respondents’ true level of attachment to
their claims to be certain of falsehoods.
Turning next to questions about economic statistics (Figure 7), respondents display a similar
tendency to back off their incorrect answers, but a weaker tendency to stick with their correct answers.
Claims to be certain seemed most meaningful on the question that asked respondents whether GDP
growth had been above or below 4 percent over the past year. This question was also the only one of the
five with clear misinformation in media: although neither annual or quarterly GDP growth reached 4
percent at any point during Trump’s presidency, Republican boosters of the 2017 Tax Cuts and Jobs Act
predicted that growth would exceed this threshold, and widely-covered preliminary estimates of 2018’s
18
Figure 7: Revealed belief in initial answers by certainty level, economic/social questions.
●
●●
●
●
●
●
●
●●
●
●
●
●
Correct: 0.51 (0.05)Incorrect: 0.11 (0.10)
● ●●
●
●●
●
● ●●
●
●●
●
Correct: 0.50 (0.05)Incorrect: 0.03 (0.06)
●● ●
●● ●
●
●● ●
●● ●
●
Correct: 0.56 (0.05)Incorrect: 0.31 (0.06)
Unemploymentdeclined
Inflation belowaverage
GDP growthbelow 4%
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty
Revealedbelief in
initialresponse
Initial responsewas...
●● Correct
Incorrect
second quarter growth pegged the figure at 4.1 percent (later revised to 3.5). At levels of stated certainty
between 0.5 and 0.99, respondents who answered correctly and incorrectly are about equally stable. Yet
among the most certain, a clear divergence emerges: respondents who answered correctly revealed a
belief of 0.87 that GDP growth was below 4 percent, while those who answered incorrectly revealed a
belief of 0.72 that it was above 4 percent. On this question, responses seem to be mostly inferences based
on general perceptions about the economy, with a smattering of correct knowledge.
The other two economic questions cover trends in unemployment and inflation, the same statistics
examined in Bartels’ (2002) influential study. On these questions, incorrect answers seem to consistently
reflect mistaken inferences. Among those who answered incorrectly claimed to be absolutely certain, the
average revealed belief never exceeds 0.65 — a full 0.10, or 20 percent of the scale, less certain than
respondents who claimed to be certain that Republicans control the House of Representatives. Correct
answers show a more variable pattern. When it comes to the unemployment rate and the rate of inflation,
respondents seem to place a fair bit of trust in their heuristics. Respondents who claim to be absolutely
certain back off to a greater extent than those who say they know that Roberts is the Chief Justice or
that Democrats control the House, but they are at least fairly sure, based on what they do know about
economic trends, that unemployment declined and that inflation has been below the postwar average
in recent years. The typical claim to be certain of the incorrect answer to a question about economic
statistics is clearly reflective of a miseducated guess, not incorrect knowledge.
Turning next to fiscal questions (Figure 8), respondents who answer incorrectly display a stronger
tendency to stick with their initial level of certainty, suggesting that misleading heuristics play a larger
role on these questions than they do on the economic questions. Few respondents incorrectly answer that
the budget deficit decreased between the 2018 and 2019 fiscal years, but those who do are only modestly
less attached to this answer than respondents who answer correctly. This result notwithstanding, if one
were to hunt for an exception to the rule that correct answers are more stable than incorret answers
conditional on certainty, repeating the deficit question in a good year would be a good place to start.
19
Figure 8: Belief in initial answers by certainty level, fiscal questions.
(a) Revealed belief (Feb. and Mar. 2020)
● ●●
●
● ●
●
● ●●
●
● ●
●
Correct: 0.49 (0.04)Incorrect: 0.31 (0.10)
●● ●
●
●
●
●
●● ●
●
●
●
●
Correct: 0.59 (0.05)Incorrect: 0.28 (0.05)
Budget deficitincreased
Foreign aid $< Medicare
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty
Revealedbelief in
initialresponse
(b) Response stability (Oct. 2019-Nov. 2019)
●
● ●●
●
●
●
●
● ●●
●
●
●
Correct: 0.97 (0.10)Incorrect: 0.24 (0.14)
.5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty, wave 1
Belief ininitial
response,wave 2
Initial responsewas...
●● Correct
Incorrect
Foreign aid $< Medicare
Graham (2020a) reports a negative relationship between confidence and accuracy among both Democrats
and Republicans for the trend in the budget deficit under President Obama.
The question about foreign aid spending relative to Medicare is a two-choice version of the ANES
question on the same subject. This question was also included in the October-November 2019 panel
survey. At levels of certainty between 0.5 and 0.99, respondents who incorrectly said that spending is
higher on foreign aid stick with their answer to about the same extent as respondents who correctly said
that Medicare spending is higher. These respondents appear to largely be relying on heuristics, some of
which are misleading. Among those who said they were absolutely certain, a substantial gap emerges.
Whereas some respondents appear genuinely certain in their knowledge that foreign aid spending is lower
than Medicare spending, respondents who claim to be certain that Medicare spending is higher do not
stick to this claim. Many respondents are clearly mistaken in their perceptions of foreign aid, but claims
to be absolutely certain that more is spent on foreign aid are largely illusory.
Among the categories of questions considered, misinformation research provides the strongest rea-
son to expect respondents to stick to their incorrect responses on questions about politicized controversies.
Across the surveys, six unique questions about politicized controversies featured a follow-up measure.
Here, five of these are examined. The question about Hillary Clinton’s private email server is excluded
because too few people got it wrong.
In the controversy concerning false rumors that Barack Obama was born outside the United States,
an oft-repeated false claim asserts that Obama has not released his birth certificate. On this question,
respondents who answer correctly provide trustworthy reports of certainty: those who claim to be totally
uncertain assigned about the same probability to their answer when asked a year later, while those who
reported being certain that Obama had released his birth certificate assigned a probability of about 0.9
to their initial response (Figure 9b). By contrast, certainty was not all that predictive of subsequently-
measured beliefs among those who answered incorrectly. Even those who claimed to be absolutely certain
20
Figure 9: Belief in initial answers by certainty level, controversy questions.
(a) Revealed belief (Mar. 2020)
●
●●
● ●
●
●
●
●●
● ●
●
●
Correct: 0.67 (0.06)Incorrect: 0.22 (0.09)
● ●
●
● ●
●
●
● ●
●
● ●
●
●
Correct: 0.76 (0.09)Incorrect: 0.34 (0.09)
●
●●
● ●●
●
●
●●
● ●●
●
Correct: 0.43 (0.06)Incorrect: 0.42 (0.08)
● ● ●● ●
●
●
● ● ●● ●
●
●
Correct: 0.27 (0.08)Incorrect: 0.22 (0.06)
Mueller didn't sayTrump colluded
with Russia
Trump recordedsaying 'grab them'
Trump claimedunlimited Article
II power
Obama saidDAPA would
ignore the law
.5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1 .5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty
Revealedbelief in
initialresponse
(b) Response stability (June 2019-2020)
●
● ●
●
●
● ●
●
● ●
●
●
● ●
Correct: 0.54 (0.13)Incorrect: 0.03 (0.22)
.5 .6 .7 .8 .9 1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Certainty, wave 1
Belief ininitial
response,wave 2
Initial responsewas...
●● Correct
Incorrect
Obama releasedbirth certificate
that Obama had released his birth cerficate backed off substantially, assigning an average probability of
0.58 to this possibility.
Among the other four controversies, the two most prominent relate to President Trump. Prior to
the release of special counsel Robert Mueller’s report on Russian interference in the 2016 presidential
election, left-leaning pundits confidently claimed that Mueller would surely find evidence of personal
collusion between Trump and Russia. Mueller’s report contained no such evidence. Among respondents
who correctly chose this answer, a high degree of correspondence emerged between stated certainty and
revealed beliefs; those who initially claimed to be absolutely certain revealed that they assign a probability
of 0.9 to this answer (Figure 9a, far left). Respondents who incorrectly answered that Mueller’s report
had contained such evidence displayed a weaker tendency to stick to their answer. Those who claimed
to be absolutely certain that Mueller had reported such evidence revealed a probability of 0.73. In the
June 2019-June 2020 panel survey, the corresponding figure was 0.52 (see the appendix).
The other prominent Trump controversy, concerning his tape recorded boast that he kisses women
and grabs them between the legs without their consent. After first acknowledging the recording’s au-
thenticity, Trump later claimed that it was faked. This question yielded a similar pattern as the other
two (Figure 9a, middle left). At low and middling levels of certainty, respondents who answered correctly
and incorrectly display some tendency to stick with their answer, indicating that respondents who claim
to be more certain are drawing on some sort of consistent basis for their inference. Among the absolutely
certain, a substantial gap emerges: those who answered correctly reveal a belief of 0.92 in their initial
answer, while those who answered incorrectly reveal a belief of about 0.77. As a rule, respondents who
answer incorrectly and claim to be certain have either not heard or not fully bought into Trump’s claim
that the recording was faked.
The remaining two controversies are less prominent matters relating to executive aggrandizement
21
under Trump and Obama. In a September 2013 interview, Obama argued that extending the protections
under Deferred Action for Childhood Arrivals (DACA) to protect unauthorized immigrants with U.S.
citizen children would amount to “ignoring the law.” Before taking precisely this action in November
2014, Obama falsely claimed not to have changed his position. Although some respondents claimed
modest or high levels of certainty about their answers to this question, the revealed belief measure
made clear that few respondents knew the facts of the controversy. The relationship between certainty
and revealed beliefs crests in a revealed belief of about 0.75 among respondents who had claimed to
be absolutely certain, regardless of whether their answer was correct or incorrect. As with the other
question types, when the gap between correct and incorrect answers disappears, it is not because people
strongly believe their incorrect answers. Instead, it is because almost nobody actually knows the correct
answer.
In contrast to Obama’s uneasy embrace of executive aggrandizement, Trump has made no secret
of his view that Article II of the Constitution grants him the power to do “whatever I want.” When
asked whether Trump has made such a claim, respondents’ claims to be certain do seem to indicate some
underlying heuristics to which they give genuine credence. Those who claim to be absolutely certain
of the incorrect answer later reveal that they assign a probability of 0.78 to this possibility — a higher
point estimate than any other question considered in this paper, but still indistinguishable, statistically
and substantively, from claims to be certain that the wrong party controls the House of Representatives.
What explains respondents’ greater tendency to back off their incorrect answers? Existing research
on political misperceptions focuses squarely on one form of measurement error: respondents’ tendency
to “cheerlead” by selecting responses that are favorable to their party (Bullock et al. 2015; Peterson and
Iyengar 2020; Prior et al. 2015; Schaffner and Luks 2018). Though cheerleading could sometimes influence
the estimates presented here, it cannot explain why the same pattern would appear on questions without
any clear partisan valence. Some other explanation is needed to account for the systematic difference
between correct and incorrect answers. The next section provides one.
Interpretation
This section uses a simple parametric model to clarify how it could be that certainty scales capture
meaningful variation in certainty, but fall short of identifying genuine certainty about incorrect answers.
Two elements enable the simulations to capture the results: (1) the prevalence of genuine certainty in
respondents’ latent, “true” beliefs and (2) unsystematic error in the measure of beliefs.
Suppose that each survey question has two response options and that each respondent has some
latent, long-run response tendency p∗i , representing the average belief the respondent would form if
asked the same survey question repeatedly. In any particular instance, respondent i forms the belief
pit ≡ p∗i + εit, where pit is the probability assigned to the correct answer and εit captures measurement
22
Figure 10: Simulation results.
(a) Distribution of latent probabilistic beliefs.
Best guessis incorrect
Best guessis correct
Knowledge &false facts
Knowledge &mistaken inferences Inferences only
0 0.5 1 0 0.5 1 0 0.5 1
Probabilistic belief
Density
(b) Response stability by in/correct answer, conditional on certainty.
Knowledge &false facts
Knowledge &mistaken inferences Inferences only
0.5 1 0.5 1 0.5 1
0.50
0.75
1.00
Certainty, initial measure
Beliefin initial
response,follow−upmeasure
Correct Incorrect
error. In the spirit of Zaller (1992), the simulations assume that i’s considerations are centered at the log-
odds of their true probabilistic belief, log(p∗i
1−p∗i). Each i draws from their considerations with error, then
uses the inverse logit function to format their considerations into a spontaneously-formed probabilistic
belief between 0 and 1.8 The simulations consider three population distributions of p∗i , each of which
corresponds to the presence or absence of certain values of p∗i in the population.
The first scenario, knowledge and false facts, assumes that the latent distribution of beliefs covers
the full spectrum from true knowledge (p∗i = 1) to belief in outright false information (p∗i = 0). The
left panel of Figure 10a displays the latent distribution of beliefs, and the bottom-left panel displays
the results of the response stability and revealed belief tests presented above. When absolute certainty
about incorrect answers really exists, both types of respondents stick with their initial response to an
equal degree (Figure 10b, left panel). Unsystematic measurement error in the scale nonetheless produces
some regression to the mean. In the results above, no question closely resembles the knowledge and false
facts scenario.
The second scenario, knowledge and mistaken inferences, is most typical of the results presented
above. In this scenario, the population distribution of p∗i spans from 0.2 to 1.0 (Figure 10a, middle
panel). This means that while some are certain of the correct answer, nobody has an outright false
8Formatting one’s response is an explicit step in Tourangeau and colleagues’ (2000) account of the survey response.
23
belief about the incorrect answer. Despite this, some respondents’ noisy draws of their considerations
lead to a fleeting sense of certainty at any point in time. In the second measurement, respondents who
initially claimed to be certain about their incorrect answer assign a lower probability to their response
than did respondents who claimed to be certain about their correct answer (Figure 10b, middle panel).
Nevertheless, because some respondents’ considerations are genuinely misleading, there is a positive
relationship between certainty and belief stability among those who answer incorrectly.
The third and final scenario, inferences only, is similar to the pattern that appeared on the more
obscure economic statistics and controversies. In this scenario, the population distribution of p∗i spans
from 0.2 to 0.8 (Figure 10a, right panel). This means that nobody’s underlying perceptions are dispositive
enough for them to be completely certain of either the correct or incorrect answer. In this scenario,
respondents again display an equal tendency to back off their initial response, regardless of whether they
answered correctly or incorrectly (Figure 10b, right panel).
In the results above, most of the questions fall somewhere along the spectrum between the latter
two scenarios. On many questions, respondents who answer correctly and claim to be certain stick with
their answer to the same degree as the simulated respondents in the scenarios with true knowledge. By
contrast, the average respondent who answers incorrectly and claims to be certain never sticks with their
answer to the same degree as equally-certain respondents who answer correctly. Such respondents put
more stock in their miseducated guesses than respondents who admit to complete uncertainty, but do
not possess incorrect knowledge.
Viewed in this light, the following generalizations can be drawn out with respect to the individual
question categories:
• General knowledge questions are generally consistent with the “knowledge only” scenario. However,respondents who answer incorrectly do sometimes have a consistent basis to mistakenly infer thatthe wrong answer is correct.
• Economic, social, and fiscal questions are generally consistent with the “inferences only” scenario,with a smattering of correct knowledge.
• Correct answers to controversy questions often represent correct knowledge, but incorrect answersgenerally represent mistaken inferences.
Implications
As concern about political misinformation surged over the past few decades, a new purpose was
demanded of old measurement technology. Unlike classic political knowledge questions, on which mis-
information is implausible and unlikely to have significant behavioral consequences, the degree to which
the public holds false beliefs about politicized controversies and economic performance really matters.
Unfortunately, too little research has heeded Kuklinski and colleagues’ warning to be mindful of the
difference between the misinformed and the uninformed, and what research has heeded this warning has
24
not gone far enough to check whether the recommended strategy — focusing on those who claim to
be certain — actually succeeds. In demonstrating systematic differences in the degree to which correct
and incorrect answers capture respondents’ perceptions, this paper provided important evidence that
survey-based studies of political misperceptions have systematically exaggerated the breadth and depth
at which the public believes in political misinformation.
The most important limitation of the evidence presented here is generalizability. Though a wide
variety of topics were considered, a fair amount of survey space was devoted to general knowledge
questions, which primarily serve as a benchmark for the other topics, and questions about economic and
social statistics that are the focus of influential work in this area (Kuklinski et al. 2000; Bartels 2002).
Moreover, the performance of the scales examined here may not be the ceiling; with different scale
features or more extensive training, perhaps a smaller subset of certain and incorrect respondents could
be isolated with greater precision. Though evidence presented here was not broad enough to establish
its results as a universal pattern, but it was broad enough to clarify that absent concrete evidence to
the contrary, consumers of survey data should assume that incorrect answers are miseducated guesses.
Researchers have good reasons to be suspicious of stronger interpretations of incorrect answers, and those
who wish to justify such interpretations can use this paper’s approach as a template for shouldering the
burden of proof.
This view of incorrect answers to factual questions has broad implications for research on misin-
formation’s origins and consequences. Studies that examine the correlates of endorsing false statements
in surveys frequently attribute apparent misperceptions, or susceptibility to false claims, to larger un-
derlying dispositions like strength of partisanship or conspiracy-mindedness (). Without disputing the
observed correlations between these various types of survey responses, the results presented here suggest
an alternative interpretation: people with these dispositions may be more likely to reason that false
statements they have never thought about are more likely than not to be true. Given the current state
of the evidence, researchers who describe incorrect answers as “endorsements” rather than beliefs (e.g.,
Oliver and Wood 2014) adopt the more defensible posture. Survey data provide a reasonably sound basis
to evaluate what predispositions might lead someone to agree with a particular claim if they encountered
it in the real world, but do not provide a sound basis to conclude that this has already happened.
The results also offer lessons for research on correcting misperceptions. While many have noted
a strong theoretical expectation that more-certain misperceptions should be harder to correct, the few
studies that are equipped to test whether certainty predicts resistance to correction have either found
no heterogeneity (Thorson 2015b) or have not reported a test for such heterogeneity (Kuklinski et al.
2000). This paper showed that an important barrier to testing this prediction is identifying respondents
who are actually certain of the falsehood to begin with.
Often, studies that correct alleged misperceptions are interested in the effect of factual beliefs on
25
attitudes. Findings in this literature are mixed: whereas some studies find that providing respondents
with correct numerical information changes their attitudes (Ahler and Broockman 2018; Gilens 2001;
Grigorieff et al. 2018), others find no such evidence (Hopkins et al. 2019; Kuklinski et al. 2000; Lawrence
and Sides 2014). Research in this area has not rigorously examined the possibility that the responses
interpreted as baseline “misperceptions” were virtually meaningless to the respondent. It may not be
a coincidence that failed treatments often focus on numerical statistics, which respondents have trouble
connecting to their perceptions absent a concerted effort to provide context (Ansolabehere et al. 2013).
The provision of correct information may not be better-interpreted as an information treatment than
as a correction. Baseline correlations between errant guesses and attitudes may merely be evidence
that survey responses, not misperceptions, are “a consequence, rather than a cause, of attitudes toward
those groups” (Hopkins et al. 2019, abstract). Absent evidence of the degree to which respondents’
incorrect guesses actually reflect underlying perceptions — which no study in this area provides — the
interpretation of factual information treatments, as well as the implications of their failure to move
attitudes, are unclear.
Perhaps the most important implication of this paper’s findings concerns the social impact of survey
research. While scholars vary in their commitment to judicious interpretation of survey evidence, public-
facing journalists and pollsters are nearly unanimous in their interpretation that “[p]eople have decided
beliefs, and surveys elicit those beliefs” (Bullock and Lenz 2019, 326). Outside of the occasional generic
reference to Converse (1964), Zaller (1992), or Tourangeau et al. (2000), survey researchers have produced
little concrete basis for pushback against these misleading interpretations of polling data. Consequently,
even as misinformation research has done a great deal to advance public concern about misinformation,
it has overlooked is own contribution to widespread public-facing misinformation about the degree to
which members of the public are misinformed. Even researchers who do not intend “misinformed” to be
a literal description cannot assume that readers are on the same page.
Until concerete evidence emerges that some question or survey techonology reliably captures false
beliefs about the facts themselves, researchers who wish to meet their obligation to provide accurate
information to their audience should strive to be judicious in their interpretation of incorrect answers to
factual questions. Political misinformation is a serious problem, and taking it seriously requires serious
attention to the alignment between measured quantities and the phenomenon of interest.
26
References
Ahler, Douglas J. and David E. Broockman. 2018. “The Delegate Paradox: Why Polarized PoliticiansCan Represent Citizens Best.” The Journal of Politics .
Ansolabehere, Stephen, Jonathan Rodden and James M. Snyder. 2008. “The strength of issues: Usingmultiple measures to gauge preference stability, ideological constraint, and issue voting.” AmericanPolitical Science Review 102(2):215–232.
Ansolabehere, Stephen, M. Meredith and Erik Snowberg. 2013. “Asking About Numbers: Why andHow.” Political Analysis 21(1):48–69.
Bartels, Larry M. 2002. “Beyond the Running Tally: Partisan Bias in Political Perceptions.” PoliticalBehavior 24(2):117–150.
Berinsky, Adam J. 2017. “Rumors and Health Care Reform: Experiments in Political Misinformation.”British Journal of Political Science 47(2):241–262.
Berinsky, Adam J. 2018. “Telling the Truth about Believing the Lies? Evidence for the Limited Preva-lence of Expressive Survey Responding.” The Journal of Politics 80(1):211–224.
Bishop, George F., Robert W. Oldendick and Alfred J. Tuchfarber. 1984. “What Must My Interest inPolitics Be If I Just Told You ’I Don’t Know’?” The Public Opinion Quarterly 48(2):510–519.
Bullock, John G, Alan S Gerber, Seth J Hill and Gregory A Huber. 2015. “Partisan Bias in FactualBeliefs about Politics.” Quarterly Journal of Political Science 10:1–60.
Bullock, John G and Gabriel Lenz. 2019. “Partisan Bias in Surveys.” Annual Review of Political Science22:325–342.
Clifford, Scott and Jennifer Jerit. 2016. “Cheating on Political Knowledge Questions in Online Surveys:An Assessment of the Problem and Solutions.” Public Opinion Quarterly 80(4):858–887.
Converse, Philip E. 1964. The nature of belief systems in mass publics. In Ideology and Discontent,edited by D. Apter. New York: Free Press.
Converse, Philip E. 1970. Attitudes and Non-Attitudes: Continuation of a Dialogue. In The QuantitativeAnalysis of Social Problems, edited by Edward R. Tufte. Reading, MA: Addison-Wesley pp. 168–189.
Delli Carpini, Michael X. and Scott Keeter. 1996. What Americans Know About Politics and Why itMatters. New Haven: Yale University Press.
Flynn, D. J., Brendan Nyhan and Jason Reifler. 2017. “The Nature and Origins of Misperceptions:Understanding False and Unsupported Beliefs About Politics.” Political Psychology 38(682758):127–150.
Flynn, D.J. 2016. “The Scope and Correlates of Political Misperceptions in the Mass Public.” Unpublishedmanuscript .
Gilens, Martin. 2001. “Political Ignorance and Collective Policy Preferences.” American Political ScienceReview 95(2):379–396.
Graham, Matthew H. 2020a. “Self-Awareness of Political Knowledge.” Political Behavior 42:305–326.
Graham, Matthew H. 2020b. “Two Sources of Bias in Measures of Partisan Differences in Factual Beliefs.”Unpublished Manuscript .
Grigorieff, Alexis, Christopher Roth and Diego Ubfal. 2018. “Does Information Change Attitudes To-wards Immigrants? Representative Evidence from Survey Experiments.” SSRN .
Hainmueller, Jens, Daniel J Hopkins and Teppei Yamamoto. 2015. “Causal Inference in Conjoint Analy-sis: Understanding Multi-Dimensional Choices via Stated Preference Experiments.” Poltical Analysis22(1):1–30.
Hill, Seth J. 2017. “Learning Together Slowly: Bayesian Learning about Political Facts.” The Journalof Politics 79(4):1403–1418.
27
Hochschild, Jennifer L. and Katherine Levine Einstein. 2015. Do Facts Matter? Norman: University ofOklahoma Press.
Hofstetter, C. Richard, David C. Barker, James T. Smith, Gina M. Zari and Thomas A. Ingrassia. 1999.“Information, Misinformation, and Political Talk Radio.” Political Research Quarterly 52(2):353–369.
Holt, Charles A. and Angela M. Smith. 2016. “Belief Elicitation with a Synchronized Lottery ChoiceMenu That Is Invariant to Risk Attitudes.” American Economic Journal: Microeconomics 8(1):110–39.
Hopkins, Daniel J., John Sides and Jack Citrin. 2019. “The muted consequences of correct informationabout immigration.” Journal of Politics 81(1):315–320.
Jerit, Jennifer, Tine Paulsen and Joshua Aaron Tucker. 2020. “Confident and Skeptical: What ScienceMisinformation Patterns Can Teach Us About the COVID-19 Pandemic.” SSRN Electronic Journalpp. 1–26.
Kuklinski, James H, Paul J Quirk, David W Schwieder and Robert F Rich. 1998. “‘Just the Facts,Ma’am’: Political Facts and Public Opinion Source.” The Annals of the American Academy of Politicaland Social Science 560(November):143–154.
Kuklinski, James H, Paul J Quirk, Jennifer Jerit, David Schwieder and Robert F Rich. 2000. “Misinfor-mation and the currency of democratic citizenship.” The Journal of Politics 62(3):790–816.
Lawrence, Eric D. and John Sides. 2014. “The consequences of political innumeracy.” Research & Politicspp. 1–8.
Lazer, David M. J., Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Fil-ippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, MichaelSchudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, Jonathan L.Zittrain, David M. J. Lazer and Jonathan L. Zittrain. 2018. “The science of fake news.” Science359(6380):1094–1096.
Lee, Seonghui and Akitaka Matsuo. 2018. “Decomposing political knowledge: What is confidence inknowledge and why it matters.” Electoral Studies 51(1):1–13.
Lewandowsky, Stephan, Ullrich K. H. Ecker, Colleen M. Seifert, Norbert Schwarz and John Cook. 2012.“Misinformation and Its Correction.” Psychological Science in the Public Interest 13(3):106–131.
Lewandowsky, Stephan, Werner G.K. Stritzke, Klaus Oberauer and Michael Morales. 2005. “Memoryfor Fact, Fiction, and Misinformation: The Iraq War 2003.” Psychological Science 16(3):190–195.
Luskin, Robert C., Guarav Sood and Joshua Blank. 2018. “Misinformation about Misinformation: OfHeadlines and Survey Design.” Unpublished Manuscript .
Luskin, Robert C and John G Bullock. 2011. “‘Don’t Know’ Means ‘Don’t Know’: DK Responses andthe Public’s Level of Political Knowledge.” Journal of Politics .
Marietta, Morgan and David C. Barker. 2019. One Nation, Two Realities: Dueling Facts in AmericanDemocracy. Oxford University Press.
Nadeau, Richard and Richard G Niemi. 1995. “Knowledge Questions in Surveys.” Public Opinion Quar-terly 59:323–346.
Oliver, J. Eric and Thomas J. Wood. 2014. “Conspiracy Theories and the Paranoid Style of MassOpinion.” American Journal of Political Science 58(4):952–966.
Pasek, Josh, Gaurav Sood and Jon A. Krosnick. 2015. “Misinformed About the Affordable CareAct? Leveraging Certainty to Assess the Prevalence of Misperceptions.” Journal of Communication65(4):660–673.
Peterson, Erik and Shanto Iyengar. 2020. “Partisan Gaps in Political Information and Information-Seeking Behavior: Motivated Reasoning or Cheerleading.” The American Journal of Political Science(Forthcoming).
28
Prior, Markus, Gaurav Sood and Kabir Khanna. 2015. “You Cannot be Serious. The Impact of AccuracyIncentives on Partisan Bias in Reports of Economic Perceptions.” Quarterly Journal of Political Science10(July):489–518.
Schaffner, Brian F. and Samantha Luks. 2018. “Misinformation or Expressive Responding?” PublicOpinion Quarterly 82(1):135–147.
Schuman, Howard and Stanley Presser. 1981. Questions and Answers in Attitude Surveys. San Diego:SAGE Publications.
Strack, Fritz and Leonard L Martin. 1987. Thingking, Judging, and Communicationg: A Process Accountof Context Effects in Attitudes Surveys. In Social information processing and survey methodology,edited by Hans J. Hippler, Norbert Schwarz and Seymour Sudman. New York: Springer-Verlagpp. 123–148.
Sturgis, Patrick, Nick Allum and Patten Smith. 2008. “An experiment on the measurement of politicalknowledge in surveys.” Public Opinion Quarterly 72(1):90–102.
Swire, Briony, Adam J. Berinsky, Stephan Lewandowsky and Ullrich K. H. Ecker. 2017. “Processingpolitical misinformation: comprehending the Trump phenomenon.” Royal Society Open Science 4:1–21.
Thorson, Emily. 2015a. “Belief Echoes: The Persistent Effects of Corrected Misinformation.” PoliticalCommunication 4609(November):1–21.
Thorson, Emily A. 2015b. “Identifying and Correcting Policy Misperceptions.” Unpublished Manuscript,George Washington University .
Tourangeau, Robert, Lance J. Rips and Kenneth Rasinski. 2000. The Psychology of Survey Response.Cambridge University Press.
Zaller, John. 1992. The Nature and Origins of Mass Opinion. Cambridge: Cambridge University Press.
Zaller, John and Stanley Feldman. 1992. “A Simple Theory of the Survey Response : Answering Ques-tions versus Revealing Preferences.” American Journal of Political Science 36(3):579–616.
29