evaluation and analysis of public discussion and deliberation in … · 2016-04-19 · i evaluation...

i

Evaluation and Analysis of Public Discussion and Deliberation in the AmericaSpeaks & Face the Facts USA Joint Civic Engagement Program for the 2012 Presidential Campaign

Report Submitted: October 31, 2013 John Gastil, David Brinker, and Robert Richards Department of Communication Arts and Sciences The Pennsylvania State University

Contact: [email protected]

Funding for this research was provided by Tufts University, with a grant from the Omidyar Foundation.

ii

CONTENTS

Executive Summary ......................................................................................................................... 1

Introduction .................................................................................................................................... 3

I: Differential Effects of Experimental Conditions .......................................................................... 4

Experimental Conditions ............................................................................................................. 4

Factual Retention ........................................................................................................................ 5

Civic Attitudes ............................................................................................................................. 6

AmericaSpeaks Goals .................................................................................................................. 7

Participants’ Program Assessments ............................................................................................ 9

People Would Use the Program .............................................................................................. 9

People Would Benefit from the Program ................................................................................ 9

Democracy Would Benefit from the Program ....................................................................... 10

Topic-Specific Evaluations ..................................................................................................... 11

II: Analyses Specific to the Hangouts ............................................................................................ 13

Civil Treatment .......................................................................................................................... 13

Communication Obstacles ........................................................................................................ 13

Facilitation ................................................................................................................................. 16

Multiple Regressions ................................................................................................................. 17

III: Effects of Repeat-Participation ................................................................................................ 18

Hangout after Another Program ............................................................................................... 18

Information (In)Sufficiency ....................................................................................................... 19

IV: Media Analysis ......................................................................................................................... 20

Current Findings ........................................................................................................................ 20

Ongoing Media Analysis Research ............................................................................................ 29

V: Analysis of Face-to-Face Deliberations ..................................................................................... 30

Facilitation Instructions and Behavior ...................................................................................... 30

Discussion Quality and Outcomes ............................................................................................. 33

Ongoing Research ..................................................................................................................... 35

VI. Summary and Recommendations ........................................................................................... 36

Main Findings ............................................................................................................................ 36

Recommendations .................................................................................................................... 37

Appendix A: Online Survey Experiment Methodology ................................................................. 38

iii

Appendix B: Guide for Face the Facts Discussions Held at the Pennsylvania State University in Spring, 2013 .................................................................................................................................. 40

Introduction............................................................................................................................... 40

Overview ................................................................................................................................ 40

Stage 1: Preparation .............................................................................................................. 40

Stage 2: Forum Facilitation .................................................................................................... 40

Stage 3: Post-Discussion Questionnaire ................................................................................ 40

What is Face the Facts USA? ................................................................................................. 40

Facilitation Style ........................................................................................................................ 40

General Tips for Facilitating your Discussion ............................................................................ 41

Two Distinct Styles .................................................................................................................... 41

The Meeting Agenda ................................................................................................................. 43

Appendix C: Debt and Deficit Pre-Discussion Fact Sheet ............................................................. 44

Appendix D: Questionnaire for Face-to-Face Discussions ............................................................ 48

1

EXECUTIVE SUMMARY

During the course of the 2012 presidential campaign season, AmericaSpeaks and Face the Facts USA partnered to create and execute online citizen engagement programs with the goal of promoting civil, fact-based political discourse on important issues facing our country. This report describes the evaluation of those programs, begun August, 2012, by a research team at the Pennsylvania State University headed by Dr. John Gastil.

The nature of the programs was such that voluntary, lay participation in the organized discussions was at what even the organizers considered extremely low levels. Our evaluation task was not to judge the efficacy of the campaign’s outreach, but rather to look at the measurable impact on those who did come in contact with the campaign. Our job was to assess the value of different deliberative elements, and toward that end, the bulk of our research relied on paid research participants.

Thus, we paid a diverse group of online workers (through Amazon Mechanical Turk) to take part in the campaign’s online programs, as well as in comparison programs designed by the researchers to test more precisely variations in online program design. The evaluation to date has consisted of measuring three kinds of outcomes: civic attitudes (e.g. efficacy), factual knowledge, and evaluation of the programs themselves.

Analyses of these data show that the Google Hangout video chat sessions had a significant impact on participants, particularly in terms of fostering pro-deliberative attitudes. In addition, it appears the programs had the desired effect on participants’ perceptions that the country needs citizens who are better informed, willing to work across different points of view, and who are more engaged.

Two comparison programs designed by the researchers, which consisted of isolated factual text and videos produced by Face the Facts USA, show those media perform well in conveying specific facts. The Spreecast webcast and Hangouts video chats appear to have affected moderate knowledge gains, outperforming the control group, but not (as expected) the condensed factual conveyance tools.

The closest we come to evaluating the outreach of the AmericaSpeaks/Face the Facts USA program (beyond our paid research participants) comes in the form of a sizable media analysis. Though we are still coding data we gathered online, current findings show the specific facts that were well received through traditional and social media channels, as evidenced by data showing the interaction with those facts, for example through readers’ use of available comment space, or the “re-tweeting” of facts disseminated through Twitter. A postscript in that section of the report details the work still ongoing.

Finally, we conducted an analysis of face-to-face discussions that we organized in lieu of AmericaSpeaks/Face the Facts USA forums, which never came to fruition. Based on the importance of experiential differences in the Hangouts, especially with regard to facilitation, we varied the facilitators’ styles in these forums and observed modest differences in outcomes as a result. The key finding was that the focus on factual content drove participants’ perceptions of the process’ value, which augurs well for fact-based discussion formats in the future.

3

INTRODUCTION

This report addresses outcomes of online citizen engagement programs executed by AmericaSpeaks and Face the Facts USA during the months leading up to the 2012 presidential election. The programs were designed to provide factual content to participants, to engage citizens in productive civil discourse, and to encourage citizens to feel empowerment through political self-expression. These programs push forward the application of technology to address the oft-bemoaned dearth of productive civic engagement and discourse. Using Google Hangouts web-chat and Spreecast™ webcast technology, AmericaSpeaks and Face the Facts USA attempted to bring opportunities for elevated discourse into the home.

To evaluate the impact of these efforts, the research team recruited paid participants to experience the programs. These participants had no known prior involvement with either organization or, to our knowledge, an atypical average interest in politics generally. They represent a fairly diverse sampling of average Americans. The advantage of using these participants is that they had no motivation to give favorable responses – they were paid regardless of their evaluation.

The task of this report is to provide a sense of the impact these programs. We first compare the data collected from multiple experimental groups. This gives a sense of sense of participants’ retention of factual knowledge, civic attitudes, and assessments of the broader social value of the programs. The second section turns to an analysis of the online discussion groups in particular. These analyses focus on the influence of conversation-specific variables such as perceived discussion civility and facilitator effectiveness. The third section estimates the effects of repeat participation. The fourth section discusses our analysis of media coverage and the social media presence of the campaign. Afterward, we present an analysis of the face-to-face forums we conducted to supplement the online forums, then we provide a summary of findings and recommendations.1

1 Permit us to make a brief stylistic note. Tables and figures are numbered in a sequence that repeats through different

parts of the report. Those for Sections I-III are numbered consecutively, but the numbering then returns to 1 for the media analysis (Section IV), and it again returns to 1 for the facilitation study (Section V). We hope this does not provide too great a distraction to the reader.

4

I: DIFFERENTIAL EFFECTS OF EXPERIMENTAL CONDITIONS

EXPERIMENTAL CONDITIONS The research design hinges on comparing the responses of participants in one of five ‘groups,’ or experimental conditions. The data analyzed in this report reflect the independent responses of participants who have experienced different programs, rather than how individual participants compare each of the programs.

Control Group participants (n=407) were asked to proceed directly to an online questionnaire without any exposure. Thus, the control module is not compared to the others for inapplicable outcome variables (e.g. value of the program). It is useful, however, as a baseline for other comparisons (e.g. efficacy).

Textual Group participants (n = 250) were asked to proceed directly to an online questionnaire. Factual text paragraphs taken from the Face the Facts USA website was embedded into the instrument, without any other peripheral content. The first pages of the survey consisted of exposure to this factual content, after which participants responded to the questionnaire questions.

Video Group participants (n=350) followed a similar procedure to the textual group. Instead of text content, they were initially exposed to two factual videos from the Face the Facts USA website. These one-minute videos were drawn from the same particular facts as the text. The text and video groups serve as comparison groups for the Spreecast and Hangouts by isolating the informational aspect of the program.

Participants in the Spreecast condition were instructed to register for a live webcast, and to arrive at a URL at a designated time. Two webcast times were made available. Using the Spreecast.com webcasting platform, Face the Facts USA conducted a program to discuss political issues relevant to the 2012 US presidential campaign. The program was largely pre-prepared and consisted of a discussion of factual background information and an interview with an expert. It was semi-interactive, however, insofar as participants were able to submit questions to the host in real-time, and use chat interface to communicate with other participants. Each webcast lasted 30 minutes, after which participants were asked to complete the online questionnaire. In total, 138 Spreecast Group participants were given directions for registering with Spreecast, viewing it live, and completing the post-viewing questionnaire. Only participants who viewed the Spreecast live are included in these analyses. After viewing the video, participants completed the online questionnaire.

Participants in the Hangout Group condition were instructed to register for a small-group discussion section, and to arrive at a URL at a designated time. The groups included four to eight participants . They were instructed to have access to a working webcam. Using Google Hangout technology, AmericaSpeaks facilitators lead a group discussion using a discussion guide on the topic of jobs and the economy. The session consisted of participant introductions, an introductory factual background presentation, a facilitated discussion between the participants about their values, perspectives, and opinions, and concluded with a discussion of actions that might be taken to address national economic challenges. Each session lasted about one hour, after which participants were asked to complete the online questionnaire. In sum, Hangout Group participants (n=112) were asked to complete an initial questionnaire with basic demographic information and to register with AmericaSpeaks for a Hangout session. After completing the session, respondents completed the follow-up questionnaire, which contained the comparison response items. More details on the data collection process for the Hangout group are available in the appendix.

5

The following sections present comparisons of responses between these five groups (four, when the control group is not applicable). Each section provides the wording of the questionnaire items, the response scale used, and statistically significant differences between means (based on One-Way ANOVA and Bonferroni post-hoc tests).

FACTUAL RETENTION Respondent factual retention was evaluated based on answers to factual questions, which included both point recall (i.e. identifying a particular number) and trend recall (i.e. recalling the relative amount or number ranges). The measurement reported here is the percent of the items correctly recalled – thus the potential values are dependent on the number of questions asked of a particular group. For example, the Spreecast group was asked three questions, resulting in four possible values: 0 (none correct), .33, .66, and 1 (all correct). The control group was asked four questions, and the text and video groups two questions (which were the facts discussed in the paragraph or video). The two Spreecast groups (from different weeks) were asked two questions relevant to the published facts from the prior week, which were reviewed by the hosts at the beginning of the webcast. The Hangout group was asked two questions relevant to jobs and the economy (the Hangout topic). Unfortunately, we have no way of reviewing the sessions to identify if, or how prominently, the facts were presented during the Hangouts. These data are illustrated in two ways to aid interpretation – as group averages in Figure 1 and within-group percentages in Figure 2.

Figure 1: Average Percent of Factual Questions Correctly Answered by Group

6

Figure 2: Number of Factual Questions Correctly Answered as a Within-Group Percentage

Both figures show that the control group, as anticipated, had a very low rate of correct responses; only two (0.5%) of 407 control participants correctly answered all three questions, while 176 (43%) incorrectly answered all three. In other words, they were essentially random; the probability of correctly guessing the answers is 0.8%, while the probability of guessing none of the answers correctly is 51%. By comparison, the video appears most effective; 199 (56%) of 355 answered both questions correctly (random chance would expect 4%), while only 31 (8.7%) answered neither correctly (random chance would expect 64%). Note that the baseline probabilities are different between the two groups, so the percentages are not directly comparable.

CIVIC ATTITUDES Four civic attitudes were collected, and the results are discussed below. Political information efficacy (PIE) is the reflexive perception that one possesses enough information to be efficacious in political participation. Four questions were employed to measure this concept:

If a friend asked me about the presidential election, I feel I would have enough information to help my friend figure out who to vote for. I think that I am better informed about politics and government than most people. I consider myself well-qualified to participate in politics. I feel that I have a pretty good understanding of the important political issues facing our country.

PIE did not significantly differ between the groups, with the exception of the mean difference between the control group (M=4.77, S.D.=1.28) and Spreecast (M=5.16, S.D.=1.27).

Second, external efficacy is a measure that reflects a participant’s perception about the responsiveness of government to his or her concerns.

(Reversed) I don’t think public officials care much about what people like me think.

7

Under our form of government, the people have the final say about how the country is run, no matter who is in office. There are many legal ways for citizens to successfully influence what government does. (Reversed) People like me don't have any say about what the government does.

External efficacy varied only slightly between the interventions. The average reported external efficacy for the control group (M=3.61, S.D.=1.09) was lower than all other groups except text (M=3.7, S.D.=1.17). The video (M=3.88, S.D.=1.2), Spreecast (M=3.94, S.D.=1.34), and Hangout (M=4.01, S.D.=1.24) groups were not statistically different from each other.

We also measured a variable we call “value factual discourse.” Participants were asked three questions to assess their agreement with statements measuring the degree to which factual discourse is seen as valuable:

When we hear claims that are false or misleading, we have a responsibility to speak up and CORRECT them. People who make political claims must ALWAYS back up their arguments with solid EVIDENCE. People should always present LOGICAL arguments in support of their views.

The participants’ scores indicating the degree to which they value factual discourse did not significantly differ between the groups.

Finally, we measured faith in deliberation. Participants were asked to assess their attitude toward deliberation through four questions:

Even people who strongly disagree can make sound decisions if they sit down and talk. Everyday people from different parties can have civil, respectful conversations about politics. The first step in solving our common problems is to discuss them together. (Reversed) Do you believe that individual leaders make better decisions than groups like committees, more often than not?

The participants in the Hangout group (M=6.05, S.D.=.84) indicated higher faith in deliberation than the control (M=5.69, S.D.=.87), textual (M=5.59, S.D.=.96), and video (M=5.69, S.D.=.88) groups. Those three groups did not differ significantly between each other.

AMERICASPEAKS GOALS At the request of AmericaSpeaks, the questionnaire posed six questions measuring specific civic attitudes, which the organization wished to address with the Hangout program. These are presented as two sets of agree/disagree questions. The first set of prompts was: “To help our political system function effectively, we need a big increase in the number of citizens who are:” followed by three 7-point scale items: “better informed,” “more engaged,” and “willing to work across different points of view.” More than 90% of participants from all groups indicated agreement on all three of the questions in the first set. Their magnitude of agreement (between somewhat agree, agree, and strongly agree) did vary, however, as indicated in Figure 3. The Hangout group was significantly higher than the control group for the “better informed” and “different points of view” questions, and was significantly higher than all but the Spreecast for the “more engaged” question.

8

Figure 3: Average Group Agreement with AmericaSpeaks Attitude Statements (Set 1)

The second set of attitudes were responses to the statement: “When it comes to the big issues facing our country, for example economic growth, creating jobs, healthcare, and energy policy:” followed by three 7-point scale items: “the government cannot always fix the problem,” “many factors are outside anyone's control,” and “the problems are ultimately pretty simple.” The different mean responses from the groups are shown in Figure 4.

Figure 4: Average Group Agreement with AmericaSpeaks Attitude Statements (Set 2)

The Hangout group differs significantly from at least one other group for two of the three questions in the second set. Participants in the Hangout gave significantly higher average responses to the “government cannot always fix the problem” question than all but the Spreecast group. There was no statistically significant difference between groups in response to the question about factors beyond anyone’s’ control. Finally, the average Hangout group response was significantly lower than the control and video, but not the text or Spreecast groups, when asked if the issues are ultimately pretty simple, indicating Hangout participants disagreed that the issues were ultimately pretty simple.

5.50

6.00

6.50

7.00

Better Informed More Engaged Work Across Diff.Points of View

Control Text Video Spree Hangout

2.00

3.00

4.00

5.00

6.00

Gov. Can't Fix Factors BeyondControl

Issues Simple

Control Text Video Spree Hangout

9

PARTICIPANTS’ PROGRAM ASSESSMENTS The freeform comments seem to suggest that participants believed that the Spreecasts and Hangouts were unique and valuable forums. A Spreecast viewer told us:

I admit I was initially worried that I would be bored stiff or feel out of my element joining a study on politics, but I ended up feeling neither as the Spreecast format was actually informative, stimulating, and interactive. More people should be informed and challenged this way for we would all be better for it.

A participant in an AmericaSpeaks Big Issues Hangout also found the experience unexpectedly valuable and enjoyable:

I was really surprised by this experience. I didn't not (sic) expect to get much (or anything) from it. But I actually enjoyed it a lot, and I think it was useful. And I think this sort of experience, with a skilled moderator, could actually be incredibly beneficial for our democracy. There are so many forces acting to the detriment of American democracy, and I can't really think of ANY forces helping preserve the quality of our democracy. …. I think there's a lot of potential here, to counteract the pernicious effects of cable news and special interest campaign financing.

The following sections attempt to capture these sentiments from the data by examining indicators that participants believe the programs were valuable.

PEOPLE WOULD USE THE PROGRAM Participants were asked to assess the likelihood that average citizens would use the program if they knew about it. This was posed as two questions, with seven response options ranging from strongly disagree to strongly agree:

Citizens would seek out this sort of online discussion if they knew it existed. Most likely, the average citizen would be UNLIKELY to spend time participating in an online discussion like this. (REVERSE)

Although the participants from the Hangout agreed most strongly with statements indicating people would likely spend time participating in the program, the difference was not statistically significant. This was in comparison to questions about the perceived likelihood the average citizen would take the time to read the textual statement, watch the one-minute videos, or watch the Spreecast. The Hangout mean agreement score (on a scale of 1-7) was 4.5, while the mean for each of the other categories were approximately 4.2. The non-difference is unsurprising; it suggests respondents were apathetic or unsure about the actual likelihood of the typical citizen participating in any citizen engagement program. The result is encouraging, however, insofar as it demonstrates that participants reflected on the evaluation questions and gave different responses to their different evaluations.

PEOPLE WOULD BENEFIT FROM THE PROGRAM Participants were also asked to agree or disagree with statements suggesting that if people were to participate in the program, they would find it beneficial. Two questions were employed for this assessment:

Generally, average citizens would find participating in this sort of online discussion valuable.

10

Most citizens would NOT get much out of participating in an online discussion like this. (REVERSE)

As Figure 5 shows, participants in the Hangout condition agreed more strongly with questions about the potential benefit of participation in the program. That evaluation exceeded the next highest evaluation – the video group – by .61 on the seven-point scale.

Figure 5: Average Group Agreement that People Would Benefit from the Program

DEMOCRACY WOULD BENEFIT FROM THE PROGRAM Participants were also asked to assess the degree to which program might aid democracy. Two questions were asked:

If more people participated in this sort of online discussion, our democracy would be stronger. Even if many people participated in online discussions like these, it would NOT make much of a difference. (REVERSE)

Figure 6 shows the average response score to this statement.

11

Figure 6: Average Group Agreement that Democracy Would Benefit from the Program

Participants gave statistically significant differences in ratings between the Hangout, Spreecast, and two comparison conditions. The Hangout was rated significantly higher on the benefit to democracy measure (M=5.7, S.D.=1.16) than all other groups, and the Spreecast (M=5.0, S.D.=1.48) was significantly higher than the text (M=4.5, S.D.=1.44) and video (M=4.47, S.D.=1.40) groups.

TOPIC-SPECIFIC EVALUATIONS In addition to questions about the broader value of the program, participants were asked if they thought the topic was important, and if the program was helpful for learning about it.

The importance of the topic was not particularly variable between groups. Figure 7 shows the average response to the question “Do you think that the topic addressed was an important one?” Statistically, the Hangout (M=4.44, S.D.=1.00) importance average is higher than that of the textual (M=3.85, S.D.=.79) or Spreecast (M=4.05, S.D.=.91) groups, the video (M=4.27, S.D.=.92) is higher than the textual group, and the textual group rating is lower than the video and Hangout groups. In terms of practical significance, however, the importance of the topic does not appear to vary substantially between groups.

12

Figure 7: Average Group Assessment of Topic Importance

Participants were also asked, “Overall, did the [program] help you better understand an issue or a political topic?” As Figure 8 shows, participants in the Hangout program found it considerably more helpful than the other programs, while the video and the webcast were generally rated about the same, more helpful than the text-only and but less than the Hangout.

Figure 8: Average Group Assessment of Program Helpfulness

Only one of 112 (0.9 percent) Hangout participant said the Hangout was not helpful at all, compared to 44 of 253 (17.4 percent) text participants, and 13 of 138 (9.4 percent) Spreecast participants.

13

II: ANALYSES SPECIFIC TO THE HANGOUTS

Participants in the Hangouts were asked about three aspects of their experience: discussion civility, communication obstacles, and facilitation. The following sections show the distribution of these scores for the 112 Hangout participants.

CIVIL TREATMENT Participants were asked to evaluate their personal experience with the Hangout. Five questions measured the civil treatment of respondent, measured on a 1-7 agree/disagree scale:

I had PLENTY of chances to speak during our group discussion. I felt REJECTED by the other group members because of the views I expressed. (Reversed) I felt that the other group members LISTENED carefully to what I had to say. The other group members RESPECTED the views I expressed during the discussion. Some of the other group members were RUDE towards me during the discussion. (Reversed) I think the other group members UNDERSTOOD what I had to say.

A high score thus means a conversation was perceived to be more civil (toward the participant him/herself). The scale average was 6.2, with a standard deviation of 0.83. Figure 9 shows that Hangout participants were generally inclined to report that the conversation was civil, with about a quarter giving maximum scores on all six indicators.

Figure 9: Histogram of Composite Civility Score

COMMUNICATION OBSTACLES Communication obstruction was indicated by four questions, measured on a 1-7 agree/disagree scale. Two of these pertained to communication style and skill, and two had to do with personality:

The other group members had a discussion style that was very different from my own style.

14

The other group members were more skilled at communicating that I was. There were some “personal” conflicts during our group’s discussion. There were some clear personality conflicts between some of the members of our group.

The Hangout participants had an evenly distributed perception of their communication differences. Figure 10 shows the average “discussion style” differences score is near the scale average (M=4.06, S.D.=1.86). Similarly, Figure 11 shows participants had an even distribution of perceived communication skill strength (M=4.14, S.D.=1.94).

Figure 10: Histogram of Participants’ Perceived Communication Obstacles

15

Figure 11: Histogram of Participant’s Perceived Communication Differences

By contrast, participants reported very few “personal” conflicts (M=1.95, S.D.=1.51), ), as shown in Figure 12, or conflicting personalities (M=2.54, S.D.=1.81), as shown in Figure 13.

Figure 12: Histogram of Participant’s Perceived Personal Conflicts

16

Figure 13: Histogram of Participant’s Perceived Personality Conflicts

FACILITATION Hangout participants were asked to evaluate their facilitator by answering “Never,” “Rarely,” “Sometimes,” “Often,” or “All of the Time” to four questions:

The moderator intervened to interject his or her own opinions. The moderator effectively kept the conversation moving forward. The moderator seemed to disrupt the conversation unnecessarily. The moderator was helpful in guiding the discussion.

Participants gave facilitators very high marks on all four questions. When the scores from the first and third question are reversed such that high scores are ‘good,’ and the four questions are combined, the average “Facilitation Score” is 4.49 (S.D.=.533), or nearly the maximum possible value. Figure 14 shows the distribution of this combined score.

Figure 14: Histogram of Participant’s Composite Evaluation of the Facilitator

17

MULTIPLE REGRESSIONS AmericaSpeaks was interested in measuring participants’ perceptions of the program’s value, which was done with the three measures described in section 1. We examined select variables discussed previously as predictors of these evaluations. Figure 15 shows the results of ordinary least squares hierarchical multiple regression analyses, a method used to disentagle the predictive value of a wide range of variables.

Figure 15: Regressions Predicting Evaluations of the Hangout

Dependent Variable:

(n=92) (n=92) (n=92)

b (SE) t b (SE) t b (SE) t

Controls

Age (Year of Birth) -0.028 0.01 -1.94+ -.012 .01 -1.263 0.005 0.01 0.571

Male Dummy -0.491 0.36 -1.38 -.421 .23 -1.82+ -0.11 0.24 -0.464

Income (Brackets) -0.014 0.06 -0.25 .007 .04 .194 0.073 0.04 1.89+

Education (Highest) -0.131 0.10 -1.28 -.125 .07 -1.88+ -0.07 0.07 -1.047

NonWhite Dummy -0.057 0.37 -0.16 -.047 .24 -.201 0.007 0.24 0.027

Ideology (Cons. >) -0.170 0.10 -1.70 -.203 .06 -3.14** -0.1 0.07 -1.531

Initial R 2 .103 .248 .182

Attitudes

Polit. Info. Eff. 0.057 0.11 0.53 .072 .07 1.026 0.058 0.07 0.813

External Efficacy 0.243 0.12 2.048* .135 .08 1.76+ 0.191 0.08 2.43*

R 2 Change .064 .084 .103

Discourse Qualities

Facilitation 0.247 0.39 0.63 .451 .25 1.78+ 0.722 0.26 2.78**

Different Convo Style -0.04 0.10 -0.36 .064 .06 1.020 0.069 0.06 1.068

Discussant Conflicts 0.261 0.16 1.60 .325 .11 3.08** -0.02 0.11 -0.214

Personal Treatment -0.04 0.30 -0.13 .543 .19 2.81** 0.28 0.20 1.422

R 2 Change .036 0.121 .156

Final R 2.203 .452 .442

Adjusted R 2.081 .369 .357

Regression coefficients are unstandardized. Standard errors in parentheses.

2-tail tests: +p<0.1, *p<.05, **p<.01

People Would

Participate

People Would

Benefit

Democracy Would

Benefit

The first block shows control variables, of which ideology is a notable predictor of the perception that citizens would personally benefit from participating in the discussions. Because higher values of ideology are more conservative, the negative sign means that we would expect more conservative respondents to perceive that people would benefit from participating to be lower. The second block of attitudes shows that external efficacy, or the perception that the government is effective and responsive, is consistently positively related to the three perception dependent variables. The final block is composed of variables pertaining to the conversation itself. Facilitation is positively related to program value perceptions, despite low variance in that variable. Percieved differences in conversation styles appears to make no difference. Interestingly, conflict between group members seems to increase perceptions that the process would be valuable to individuals if more people participated in programs like the Hangouts. Finally, when participants report being treated well (e.g. their views were respected, other group members were not rude to them), they percieved significantly more individual benefit of the program.

18

III: EFFECTS OF REPEAT-PARTICIPATION

HANGOUT AFTER ANOTHER PROGRAM The analyses shown in Figure 16 compare average scores of first-time participates in a Hangout, who had not participated in any other group, and those responses from Hangout participants who had previously participated in one other group. Participants’ program assessments of the Hangout are not compared because the causal link is ambiguous (that is, we cannot say if people who give high value assessments were more likely to be repeat participates, or if those who participate repeatedly come to give higher evaluations). Regardless, when tested no significant difference between one-time and repeat participants was observed.

Figure 16: Comparison of First-Time & Repeat Hangout Participants

Mean Scores

First-Timer (n=81)

Prior Participant (n=21)

Statistical Significance

Knowledge

Factual Retention 0.35 0.33 NO

Information (In)Sufficiency -11.85 8.67 YES (t = -2.13, p = .036)

Civic Attitudes

Political Information Efficacy 5.01 5.08 NO

External Efficacy 3.91 4.12 NO

Value Factual Discourse 6.01 6.02 NO

Faith in Deliberation 6.03 6.03 NO

AmericaSpeaks Goals

Better Informed 6.56 6.29 NO

More Engaged 6.44 6.29 NO

Work Across Diff. Points of View 6.41 6.24 NO

Gov. Can't Always Fix the Problem 5.57 4.86 YES (t = 2.10, p = .038)

Many Factors Beyond Anyone's Control 4.25 3.95 NO

The Problems are Ultimately Pretty Simple

2.91 2.48 NO

19

Two significant differences emerge from these comparisons: Information (in)sufficiency and agreement with the statement “The government cannot always fix the problem.”

INFORMATION (IN)SUFFICIENCY A measure of information sufficiency was included in the data collection. The score is calculated as the difference between the responses (scaled 0-100) to two questions:

1) “Please rate your knowledge about political issues like the ones you saw during [the group]. Use the scale from 0 to 100, where 0 means knowing nothing and 100 means knowing everything you could possibly know about political issues. Using this scale, how much do you think you currently know about political issues?”

2) “Think of that same scale again. This time, please estimate about how much knowledge you would need in order to feel confident about your own views regarding political issues in your own life. Of course, you might feel you need the same, more, or possibility even less, information about political issues.”

The average difference between the responses to these questions statistically higher in the repeat group, indicating that on-average the repeat participants indicated a larger ratio of information possessed to information needed. The two mechanisms that could cause this are an increase in knowledge or decrease in perceived need for knowledge. Figure 17 compares the average scores for first-time and repeat participants on both items separately.

Figure 17: Comparison of Information Sufficiency Responses: First-Time & Repeat Participants

Mean Scores

First-Timer (n=81)

Prior Participant (n=21)

Statistical Differences

Current Knowledge

55.38 59.19 (t = -.422, p = .67)

Knowledge Desired

67.23 50.52 (t = 1.74, p = .085)

Although neither item alone significantly differs between first-time and repeat participants,2 the direction of change in both score pairs is correct to explain the statistical difference between the overall sufficiency scores. Although the increase in current knowledge reported is fairly small (3.8 on a 100 point scale), the decrease in knowledge desired is sizable (16.7 on a 100 point scale).

2 Note that the change in “knowledge desired” is statistically significant if a 1-tail assumption is used (p=.0425).

20

IV: MEDIA ANALYSIS

As part of our inquiry into the media coverage of the Face the Facts USA civic engagement events and public responses to that coverage, we analyzed users’ responses to each “Fact” posted on the Face the Facts USA Website during 2012. These responses took the form of comments left directly on that Website, as well as Facebook “likes,” Twitter “tweets,” Google Plus “+1” responses, LinkedIn “shares,” and Pinterest “pins,” all as recorded on the Face the Facts USA Website. Descriptive statistics regarding these responses appear in Table 1.

CURRENT FINDINGS Table 1 shows that, in general, Face the Facts USA Website posts generated substantial public response. Each post generated an average of 84 interactions, including an average of 15 comments directly on the Website, 41 Facebook “likes,” and 23 Twitter “tweets.” Further, the Website comments to most posts featured dialogue among the commenters, as well as interjections by Face the Facts USA personnel, who acted as facilitators. Of the social media services, Facebook and Twitter generated by far the greatest share of user responses, accounting for 48% and 27% respectively of all public responses to posts.

Table 1: Web and Social Media Responses to Face the Facts USA Fact Posts, July 31-December 31, 2012

# of Comments

# FB Likes

# Tweets

# Google+ +1s

# LinkedIn shares

# Pinterest pins

Total Public Responses

Total 1650 4504 2502 187 307 195 9345 % of Total Public Responses

0.177 0.482 0.268 0.020 0.033 0.021 1.000

Mean 14.9 40.6 22.5 1.7 2.8 1.8 84.2 Median 11 19 15 1 2 1 62 Mode 0 15 0 0 0 0 48 Maximum 109 337 142 21 15 35 396 Minimum 0 1 0 0 0 0 8 SD 16.2 58.5 24.7 2.9 3.5 4.0 73.6 N = 111 posts. “Comments” are user comments posted directly to the Face the Facts USA Website, http://www.facethefactsusa.org/. Other social media figures are those recorded on the Face the Facts USA Website.

To learn more about the nature of users’ interactions with posts on the Face the Facts USA Website, the ten posts that received the most public responses in all formats, the ten posts that received the most comments on the Website, and the ten posts that received the most Facebook “likes” and Twitter “tweets” were identified, and are listed in Tables 2 through 5. (Because posts generally received very few responses through Google Plus, LinkedIn, or Pinterest, those responses are not considered in this report.)

Table 2 shows the ten posts that received the most public responses of all kinds. Of particular note is the quantity of responses, ranging from approximately 200 to nearly 400 responses per post. This reflects significant public interaction with the Face the Facts USA online material. In addition, the

http://www.facethefactsusa.org/

21

posts relate to a variety of topics, viz., eight of the 18 topics addressed by Face the Facts USA Website posts during 2012.3 This suggests that the public was willing to engage with Web-based materials concerning a range of issues. Further, nearly all of the topics covered by the posts described in Table 2 were issues debated during the 2012 campaign, most prominently the topic of health care, which is the subject of three of the posts and which received a combined total of 679 public responses.

Also notable is that half of the posts were published during the early months of the 2012 election campaign—July and August—while the other half of the posts appeared after election day. The high levels of engagement during the summer may reflect heightened public desire to gather information and exchange views about political issues immediately following the national party conventions, followed by reduced need to engage as election day approached. The high levels of activity in the two months following the election are striking, and may be evidence of a reinforcement effect: citizens’ participation in politics during the fall may have reinforced citizens’ sense of political empowerment and sense of responsibility for their communities (Burkhalter, Gastil, & Kelshaw, 2002). This in turn may have increased some citizens’ willingness to interact with political information on the Face the Facts USA Website in the weeks following the election.

A notable finding from Tables 3, 4, and 5 is the variation in the popularity of posts and topics between different platforms. Although certain topics were very popular on the Web, in Facebook, and on Twitter—such as health care, taxes, and Medicare / Social Security—certain subjects received more attention on particular platforms, and less on others. For example, posts concerning jobs and the economy and infrastructure were among those that received the most comments on the Face the Facts Website but were not mentioned among the posts that received the most attention on Facebook and Twitter. The reason may be that infrastructure and economics are comparatively complex and technical topics that may be difficult to represent with images or brief statistics, and so those topics are more likely to yield responses in the form of discourse in extended comments than via gestural responses such as “likes” or “tweets.”

By contrast, posts about food and national security—topics easily depicted in images and having strong emotional appeal—were among the most popular on Facebook and Twitter, but not in comments on the Face the Facts USA Website. In addition, virality,4 which occurs on regularly on social media but less often regarding Website comments, may account for the greater popularity of some posts on social networks than in Website comments. For example, Table 5 shows that a post about immigration entitled “The melting pot story isn’t ours alone,” appears to have “gone viral” exclusively on Twitter: the post received 142 tweets, making it the most popular Face the Facts USA post on Twitter, but it was not among the most popular posts on Facebook or in terms of Website comments.

3 Those topics were: Automobile standards, debt and deficit, education, energy and environment, family, food, foreign aid, health care, immigration, infrastructure, jobs and the economy, life expectancy, Medicare and Social Security, national security, prisons, taxes, technology, and trade. 4 Anstead, N., & O’Loughlin, B. (2011). The emerging viewertariat and BBC Question Time: Television debate and real-time

commenting online. International Journal of Press/Politics, 16, 440-462. doi: 10.1177/1940161211415519

22

Table 2: Face the Facts USA Website Posts that Received the Most Public Responses, in the Form of Website Comments and Social Media Interactions, July 31-December 31, 2012

Post Title Subject # of Public Responses

% of All Public Responses

Date

Americans go hungry in a land of massive food waste

Food 396 0.042 11/18/2012

Shocking PTSD, suicide rates for vets

National Security 373 0.040 11/10/2012

Money can't buy genius Education 317 0.034 8/1/2012

The U.S. -- holding its own… debt

Debt & Deficits 286 0.031 12/15/2012

More US mothers dying despite expensive care

Health Care 258 0.028 8/24/2012

Pull over here and fill up Energy & Environment

233 0.025 8/8/2012

Use of Painkiller Drugs Triples in 20 Years

Health Care 216 0.023 12/30/2012

A premium rise in premium prices

Health Care 205 0.022 8/22/2012

Ask for the corporate rate. Or not.

Taxes 203 0.022 7/31/2012

Social Security keeps seniors out of poverty

Medicare / Social Security

192 0.021 11/17/2012

N = 9345 total public responses; N = 111 posts. “Comments” are user comments posted directly to the Face the Facts USA Website, http://www.facethefactsusa.org/. Other public responses are Facebook “likes,” Twitter “tweets,” Google Plus “+1s,” LinkedIn “shares,” and Pinterest “pins,” in all cases as recorded on the Face the Facts USA Website. See the note to Table 1 for details.


23

Table 3: Posts that Received the Most Comments on the Face the Facts USA Website, July 31-December 31, 2012

Post Title Subject # of Comments

% of All Comments

Date

A premium rise in premium prices

Healthcare 109 0.066 8/22/2012

Medicare Docs Face 27% Fee Cuts

Social Security / Medicare

72 0.044 10/30/2012

One of every 50 federal dollars buys food stamps

Debt & Deficits 66 0.040 8/28/2012

Mortgage tax break skews to the more affluent

Taxes 61 0.037 9/7/2012


Healthcare 51 0.031 8/24/2012

A government that keeps on giving

Debt & Deficits 39 0.024 8/18/2012

One nation under debt Debt & Deficits 37 0.022 12/10/2012

Hello trolley! Light rail is a public transit star

Infrastructure 37 0.022 9/9/2012

Can you put in some overtime? Social Security needs you

Social Security / Medicare

34 0.021 9/20/2012

Factories bounce back, hiring hangs back

Jobs & Economy 34 0.021 8/17/2012

N = 1650 total comments; N = 111 posts. “Comments” are user comments posted directly to the Face the Facts USA Website, http://www.facethefactsusa.org/.


24

Table 4: Face the Facts USA Website Posts that Received the Most Facebook “Likes,” July 31-December 31, 2012

Post Title Subject # of FB Likes

% of All FB Likes

Date

Shocking PTSD, suicide rates for vets


Money can't buy genius Education 283 0.063 8/1/2012


Food 260 0.058 11/18/2012

Pull over here and fill up Energy & Environment

200 0.044 8/8/2012


Debt & Deficits 180 0.040 12/15/2012


Health Care 164 0.036 8/24/2012

Social Security keeps seniors out of poverty

Medicare / Social Security

149 0.033 11/17/2012

In dollars for defense, we’re still #1 with a bullet [INFOGRAPHIC]


A tax impact from Grandpa’s day

Taxes 134 0.030 9/12/2012


Health Care 127 0.028 12/30/2012

N = 4504 total Facebook “likes”; N = 111 posts. Facebook “likes” are as recorded on the Face the Facts USA Website, http://www.facethefactsusa.org/.


25

Table 5: Face the Facts USA Website Posts that Received the Most Twitter “Tweets,” July 31-December 31, 2012

Post Title Subject # of Tweets % of All Tweets

Date

The melting pot story isn’t ours alone

Immigration 142 0.057 12/28/2012

Tax cuts on the cliff Taxes 100 0.040 11/26/2012

Catch of the day comes from abroad

Food 94 0.038 11/16/2012


Food 93 0.037 11/18/2012


Debt & Deficits 89 0.036 12/15/2012

Feds flooded with new vet disability claims


The sweet life and what it costs us

Food 69 0.028 12/22/2012

One nation under debt Debt & Deficits 68 0.027 12/10/2012


Health Care 67 0.027 12/30/2012

Ask for the corporate rate. Or not.

Taxes 55 0.022 7/31/2012

N = 2502 total Twitter “tweets”; N = 111 posts. Twitter “tweets” are as recorded on the Face the Facts USA Website, http://www.facethefactsusa.org/.

Posts were also analyzed to determine which subjects garnered the most responses overall, and on which particular platforms. Results of this analysis appear in Tables 6 through 9.

Table 6 shows that four of the five topics of Face the Facts USA Website posts receiving the most public responses per post were leading issues debated during the 2012 election campaigns: debt and deficits, national security, health care, and taxation. The fifth topic, food, likely received substantial attention because of the emotional appeal and amenability to simplification and visual depiction discussed above, as well as the publication of posts about that topic in the context of holidays during November and December 2012. When the total number of public responses per topic, rather than responses per post, is considered, the fifth most popular topic of posts was Medicare and Social Security, another prominent campaign issue.

Regarding Website comments, consistent with the discussion above regarding popular individual posts, Table 7 shows that the topics of infrastructure and jobs and the economy are among those receiving the greatest number of popular responses—the latter ranks fourth when popularity is measured in terms of total number of comments received—along with the major campaign issues of fiscal policy, entitlements, health care, and taxes. Again, economics and public works, likely because of their complexity and the difficulty of representing them simply in the images and short phrases


26

that social networks favor, may be more readily addressed in extended prose comments on a Website than in social media.

Tables 8 and 9 show topical rankings largely consistent with the findings above regarding the popularity of individual posts on social media. Although posts on prominent campaign topics received substantial attention on Facebook and Twitter, on both social networks posts about food received the most public responses, likely for the reasons discussed above regarding individual posts on that subject, but food was rarely discussed in Website comments. National security was also very popular on both of the social networks, but not in Website comments, likely because of the emotional appeal and ease of representation noted earlier. In addition, the high ranking of posts about immigration on Twitter seems to stem from a single post, “The melting post story,” having gone viral.

Table 6: Topics of Face the Facts USA Website Posts that Received the Most Public Responses per Post, in the Form of Website Comments and Social Media Interactions, July 31-December 31, 2012

Subject # of Public Resonses

Public Responses / Post

% of All Public Responses

# of Posts

Food 788 157.6 0.084 5 Debt & Deficits 1340 121.8 0.143 11 National Security 1069 106.9 0.114 10

Healthcare 1259 104.9 0.135 12 Taxes 1038 94.4 0.111 11 Immigration 253 84.3 0.027 3

Education 730 81.1 0.078 9 Medicare & Social Security 818 74.4 0.088 11 Energy & Environment 503 62.9 0.054 8 Infrastructure 392 49.0 0.042 8 Jobs & Economy 728 48.5 0.078 15 N = 9345 total public responses; N = 111 posts. “Comments” are user comments posted directly to the Face the Facts USA Website, http://www.facethefactsusa.org/. Other public responses are Facebook “likes,” Twitter “tweets,” Google Plus “+1s,” LinkedIn “shares,” and Pinterest “pins,” in all cases as recorded on the Face the Facts USA Website. See the note to Table 1 for details.


27

Table 7: Topics of Face the Facts USA Website Posts that Received the Most Website Comments per Post, July 31-December 31, 2012

Subject # of Comments Comments / Post

% of All Comments

# of Posts

Debt & Deficits 254 23.1 0.154 11 Medicare & Social Security 244 22.2 0.148 11 Healthcare 258 21.5 0.156 12 Infrastructure 110 13.8 0.067 8 Taxes 150 13.6 0.091 11 Education 120 13.3 0.073 9

Jobs & Economy 178 11.9 0.108 15 Energy & Environment 88 11.0 0.053 8 National Security 106 10.6 0.064 10 Immigration 19 6.3 0.012 3 Food 20 4.0 0.012 5 N = 1650 total comments; N = 111 posts. “Comments” are user comments posted directly to the Face the Facts USA Website, http://www.facethefactsusa.org/.

Table 8: Topics of Face the Facts USA Website Posts that Received the Most Facebook “Likes” per Post, July 31-December 31, 2012

Subject # of FB Likes FB Likes / Post

% of All FB Likes

# of Posts

Food 387 77.4 0.086 5 National Security 650 65.0 0.144 10 Debt & Deficits 574 52.2 0.127 11 Education 465 51.7 0.103 9 Healthcare 605 50.4 0.134 12 Energy & Environment 336 42.0 0.075 8 Taxes 447 40.6 0.099 11 Medicare & Social Security 398 36.2 0.088 11

Immigration 64 21.3 0.014 3 Jobs & Economy 301 20.1 0.067 15 Infrastructure 160 20.0 0.036 8 N = 4504 total Facebook “likes”; N = 111 posts. Facebook “likes” are as recorded on the Face the Facts USA Website, http://www.facethefactsusa.org/.



28

Table 9: Topics of Face the Facts USA Website Posts that Received the Most Twitter “Tweets” per Post, July 31-December 31, 2012

Subject # of Tweets Tweets / Post % of All Tweets

# of Posts

Food 304 60.8 0.122 5 Immigration 163 54.3 0.065 3 Debt & Deficits 398 36.2 0.159 11 Taxes 341 31.0 0.136 11 National Security 273 27.3 0.109 10 Healthcare 287 23.9 0.115 12 Jobs & Economy 194 12.9 0.078 15

Infrastructure 102 12.8 0.041 8 Medicare & Social Security 138 12.5 0.055 11

Education 91 10.1 0.036 9 Energy & Environment 47 5.9 0.019 8 N = 2502 total Twitter “tweets”; N = 111 posts. Twitter “tweets” are as recorded on the Face the Facts USA Website, http://www.facethefactsusa.org/.

Overall, then, public response to posts about issues on the Face the Facts USA Website was substantial—averaging more than 80 interactions per post—and featured some noteworthy variation apparently related to the characteristics of different issues and platforms. Facebook was the most popular medium—generating nearly half of public responses—followed by Twitter, with more than a quarter of responses. Comments on the Face the Facts USA Website elicited 18% of total responses. Website comments were often extensive and interactive and were usually facilitated by Face the Facts USA personnel.

The major topics of the 2012 election campaign featured prominently among the most popular posts. Yet certain topics received much more public interaction on some platforms than on others. In particular, the public responded to posts about the relatively complex topics of infrastructure and jobs and the economy more frequently through Website comments—that welcome extended prose that may have been relatively well suited to those topics—than on social media. Posts about food and national security—which had strong emotional appeal and were easily depicted with images—were highly popular on Facebook and Twitter, but received few Website comments. Further, the topic of immigration garnered much public attention on Twitter because a post about that subject went viral, although posts on that issue received much less public response on Facebook and in Website comments. This variation in responses indicates the wisdom of distributing political content about different topics on multiple platforms that enable a variety of modes of expression. Such a strategy appears to increase the likelihood that willing citizens will encounter information about an issue in a context that allows them to respond with technology that is appropriate both to the issue and to their preferences and skill level.

Moreover, the pattern of user responses to the Face the Facts USA posts furnishes some evidence of the citizen-engagement reinforcement effect.5 High levels of public response to posts about issues such as health care and the national debt in the weeks following the November 2012 election

5 Burkhalter, S., Gastil, J., & Kelshaw, T. (2002). A conceptual definition and theoretical model of public deliberation in small face-to-

face groups. Communication Theory, 12, 398-422.


29

suggest that participation in the 2012 campaign may have increased citizens’ sense of political empowerment and concern for the public interest and may have motivated them to continue engaging with political issues after having cast their votes.

ONGOING MEDIA ANALYSIS RESEARCH Since our last update in April 2013, we’ve completed the following tasks regarding the media analysis of the fall 2012 Face the Facts USA/AmericaSpeaks (FtF/AS) online educational programs. First, we completed gathering news articles and transcripts from LexisNexis and Newsbank and public comments on the Web versions of those articles. We found a total of 603 articles and transcripts, published by 62 different media outlets. Next, we pre-tested our coding scheme and revised it to add new variables—bringing the total number of variables to 80—and to provide additional instructions, particularly for addressing social media comments. In addition, we tested the student coders’ work for inter-rater reliability. After training, the student coders coded all 603 of the news articles and transcripts and 287 public comments on the Web versions of those resources, as well as 1,650 public comments on the 111 posts on the Face the Facts USA Website. In the next stage of coding, the student coders coded 1,897 comments on the Face the Facts USA Facebook page, as well as 2,945 comments about those comments and “shares” and “likes” of the Face the Facts USA Facebook page updates or comments. Currently, the student coders are coding 137 Twitter “tweets” by members of the public about Face the Facts USA “facts” or events from summer and fall 2012. The next steps are to code public comments on Face the Facts USA materials posted on YouTube and the Face the Facts USA tumblr, and then to perform qualitative and quantitative analysis of all of these coded data, which we hope to complete that by spring 2014.

In effect, we’ve reached far beyond the initial conception of the media analysis because we think there is more to be learned. We’ll continue the research on Penn State funds, but we expect that the paid coding will end this semester.

30

V: ANALYSIS OF FACE-TO-FACE DELIBERATIONS

One component of the AmericaSpeaks program that never came to fruition was hosting a series of face-to-face discussions to be held across the country on the same subject as the online forums. Our charge included assessing such forums, so we hosted a series of them on campus using a study-participant pool available through the Department of Communication Arts & Sciences. Because doctoral dissertation research takes precedence in the use of that pool, we were forced to use participant payments to complete our data collection, which we did in the fall of 2013.

Into these face-to-face discussions we inserted a key variable, which was highlighted in our analysis of the Hangout discussions held online. Recall from Section II of this report that the nature of the facilitation was an important factor in participants’ assessments of the Hangouts. More favorable views of the facilitator were associated with a perception that the Hangouts would benefit the general public and the wider democratic system. The assessments of facilitators were based on a four item scale that measured neutrality, keeping the discussion on pace, and guiding (rather than disrupting) the flow of talk. Those are all good characteristics of forum facilitation, but beyond those four indicators, there are considerable variations in the approach one can take to facilitation. This study, therefore, provided a systematic contrast among theoretically distinct styles to see if a particular approach could maximize the positive impact discussion has on civic attitudes.

FACILITATION INSTRUCTIONS AND BEHAVIOR Appendices B and C provide details on the different instructions we gave facilitators, but herein we provide a summary to give a better sense of how facilitators organized these face-to-face discussions. Some of the facilitators were the project leaders (John Gastil, David Brinker, and Robert Richards), but others were trained graduate and undergraduate students. We explained the basic features to those facilitators in this way:

The discussion will address the debt and deficit, one of many issues facing the US.

It will last 40-45 minutes, no less and no more. There is plenty of material in this guide to keep it going, and you may need to move the group forward to make sure it finishes on time.

Your discussion will have as few as three participants and fewer than ten. (They were actually between 3 and 7, with four-fifths between 4 and 6 people in size.)

You will facilitate the discussion to either emphasize and encourage analytic rigor or democratic social relations among the participants.

Your job is to facilitate the discussion effectively, which means moving through the agenda efficiently and keeping the discussion focused on the student participants, not on yourself. An effective facilitator does just that—facilitates an exchange among the discussion participants that stays on the task at hand. You will begin and end the discussion and interject at key points during the discussion, but be careful to avoid dominating or directing the discussion.

Though we would vary the style of the facilitators, we wanted to make sure that each followed the basic tenets of effective issue discussion facilitation. Thus, each was given three basic guidelines to follow in every case:

Serve as a Neutral Guide. Your role is to help support the discussion. The best way to do that is to maintain a neutral position and to hold back on voicing your own opinions. Instead, focus on keeping the Discussion going and making sure everyone has a voice.

31

You Are Not the Expert. Don’t feel that you must be an expert on the issues. Before your Discussion, read the materials you’ve been provided. If questions come up in your Discussion, participants themselves may provide answers, but if none speak up, you can serve as a resource if you know the answer from the Fact Sheet. Do not offer any personal expertise beyond that, lest our different facilitators vary in this regard.

Be Mindful of Time. You will want to pay attention to the time of each task to ensure that you are able to complete each step of the meeting. Do not expect the participants to keep track of time. If the discussion is going well, they will lose track of time as they become engaged in the discussion.

We then explained the style differences we would assign in this way:

Though all facilitators will follow the guidelines described above, this study contrasts two variations on facilitation—one focused on rigorous analysis of the facts and the other focused more on ensuring the democratic social relationships among the participants. Both are important elements of democratic deliberation, but in this study, you will emphasize one of these over the other. The following page contrasts these two styles.

Appendix B provides details on how we operationalized those two styles, but the proof would be in the experience of the participants and the facilitators’ actual behaviors. As for the latter, we have two undergraduate students coding the transcripts from these sessions to assess the degree to which the facilitators modeled the different deliberative styles. Each is coding separate transcripts, but they both coded some of the same transcribed data to permit us to calculate the degree of consistency in their coding.

The results thus far are promising. In terms of the main codes (for analytic versus social behavior), they agreed on 224 of 254 observations, for an 88% agreement rate, which is 63% better than could be expected by chance. The formal statistic of association, Krippendorff’s alpha, was in excess of .80, which suggests sufficient inter-rater reliability. Agreement was also strong for those sub-codes for which there were at least six codings (minimum alpha = .82). Once the coding is done, we will have systematic observational evidence on the question of facilitators’ manifest behavior.

In the meantime, we can look at participants’ perceptions of the different facilitation styles by reviewing the results of the questionnaires participants filled out. The study participants came from the research participant pool for Communication Arts and Sciences, which draws across the full campus. In the spring of 2013, we used that pool to hold 27 sessions with 124 participants. Table 1 summarizes their ratings of facilitator behavior in regard to two generally desirable and two undesirable features of facilitation, and the averages across the three facilitation types were not significantly different. In other words, consistent with our intention, the face-to-face discussion facilitators were neither disruptive nor opinionated, and they kept the conversation moving forward and guided it through the agenda effectively. (Full details of questionnaire items are provided in Appendix D.)

32

Table 1. Ratings of General Facilitator Behavior (1 = Strongly Disagree to 5 = Strongly Agree)

Analytic Style

Blended Style

Social Style

The facilitator seemed to disrupt the conversation unnecessarily.

1.0 1.1 1.1

The facilitator intervened to interject his or her own opinions. 1.7 1.5 1.4

The facilitator effectively kept the conversation moving forward.

4.7 4.8 4.9

The facilitator was helpful in guiding the discussion. 4.7 4.8 4.8 No scores were significantly different across the rows.

When asked about behaviors that we had intended facilitators to vary across the analytic, social, and blended discussion conditions, the results were mixed. For only two of the six behaviors measured did participants see differences, though both fit our intended pattern. First, the analytic facilitators were the most likely to be seen as “looking for tensions and tradeoffs” between different approaches to debt/deficit. Second, the social facilitators were perceived as the ones most concerned with “sharing speaking time equitably.” Table 2 shows these differences, along with the relatively equal scores for the other behaviors participants observed.6

Table 2. Ratings of Facilitators’ Intended Style (1 = Strongly Disagree to 5 = Strongly Agree)

How would you characterize your facilitator’s goals for this session?

Analytic Style

Blended Style

Social Style

Keeping the conversation to the point 4.4 4.5 4.4

Keeping the conversation open to disagreement 4.5 4.6 4.5

Looking for the tensions and tradeoffs between different approaches

4.0 3.7 3.4

Increasing the group’s understanding of the facts 4.3 4.3 4.4

Sharing speaking time equitably between discussion participants

4.5 4.4 4.7

Showing respect and appreciation for each discussion participant

4.9 4.9 4.9

Differences in “tensions and tradeoffs” were significant (F = 3.12, p = .048), and those for “speaking time” were trending toward significance (F = 1.40, p = .099).

A variation on that question produced comparable results. This time, we asked participants, “Regardless of the facilitator’s efforts, how well does each of the following actually describe the conversation you had?” When put this way, participants tended to perceive the analytic facilitator as keeping the group more open to disagreement, with the blended style being most effective at increasing the group’s understanding of the facts. As before, equal speaking time was the hallmark of the social facilitation style.

6 Note that all the analyses in this section of the report will benefit from multi-level analysis, which we will conduct to distinguish individual-level variations from group-level variations. Such analysis can simultaneously assess how those things that vary across individuals, such as gender or initial expectations for a discussion, influence outcomes in ways that are distinct from group-level variables, such as group size or facilitator style type.

33

Table 3. Ratings of Actual Discussion Quality (1 = Strongly Disagree to 5 = Strongly Agree)

Regardless of the facilitator’s efforts, how well does each of the following actually describe the conversation you had?

Analytic Style

Blended Style

Social Style

Keeping the conversation to the point 4.5 4.5 4.5

Keeping the conversation open to disagreement 4.7 4.5 4.3

Looking for the tensions and tradeoffs between different approaches

4.1 3.9 3.8

Increasing the group’s understanding of the facts 4.1 4.5 4.2

Sharing speaking time equitably between discussion participants 4.0 4.2 4.6

Showing respect and appreciation for each discussion participant

4.9 4.8 4.9

Differences in “open to disagreement” were near significance (F = 2.20, p = .115), as were those for “understanding the facts increased” (F = 2.34, p = .101). Differences for “speaking time” were significant (F = 4.67, p = .011).

Once we complete the transcript codings for these and the fall 2013 data (see Ongoing Research, below), we will have a more complete portrait of the style differences we tried to create in this study. In the meantime, we will focus on the more general participant perception of discussion quality—the variables listed on the left-hand side of Table 3.

DISCUSSION QUALITY AND OUTCOMES To get a sense for how we will assess the relationship between the features of the face-to-face discussions and the kind of quality metrics used in Sections I-III of this study, we show how we use regression analysis to disentangle the relevance of different discussion features. We added to the mix of variables those obstacles commonly associated with democratic small group discussion, particularly style and skill differences and personality conflicts.7

In regression analysis, a set of predictors are simultaneously used to predict an outcome, and this shows the unique impact of each predictor. In doing so, one must guard against the predictors being too strongly correlated with one another, and after factor analyses, we chose to keep each predictor separate, as each was only modestly associated with one another. That said, we have reduced the number of predictors shown in Tables 4-6 for ease of presentation, omitting only those that failed to reach significance in each regression equation.8

Table 4 begins by showing that the perceived level of disagreement was driven by personality conflicts and conversations that failed to stay “to the point.” It is unfortunate, really, that overall disagreement has negative connotations for participants, but it’s clear that it’s associated with personality overriding substance in the dicsussion. What participants wanted was to simply understand the issue better, and this experience was thwarted by personality conflict. What aided it was the identification of tradeoffs/tensions and the perception that the whole group was learning, partly owing to the articulateness of other group participants. The topic was perceived as important

7 See Gastil, J. (1993). Democracy in small groups: Participation, decision making, and communication. Philadelphia, PA:

New Society Publishers. 8 Importantly, one such omitted variable was the assurance of equal time, which probably failed to predict differences owing to its limited variation: 82% of all participants said that they “agreed” or “strongly agreed” that time was shared equitably. Also, variation in group size was not important, again owing to only modest differences (with four-fifths of groups between four and six persons in size).

34

when discussion stayed on point, though differences in communication style could undermine perceived importance.

Table 4. Predicting Disagreement Level, Issue Understanding, and Topic Importance

Feature of discussion Level of Disagreement

Help Understand Issue9

Topic Was Important

Conversation was to the point -.142** .123 .322**

Conversation open to disagreement .109 -.160 -.018

Identified tensions/tradeoffs .119 .261** .018

Understanding of the facts increased -.152 .343** .104

Respect and appreciation -.080 .051 -.084

Others’ discussion style different from mine

.082 -.126 -.244**

Others more skilled at communicating that me

.022 .279** -.076

Clear personality conflicts among members

.294** -.203* .063

Asterisks are used to indicate statistically significant results. * indicates p < .05, ** indicates p < .01.

Table 5 shows that discussion style differences also undermined the sense that other citizens would find a face-to-face debt/deficit discussion valuable. As with personal knowledge gains, the keys to perceiving positive value for other citizens was a conversation that identified tensions/tradeoffs and generally increased the group’s understanding of the facts. The latter was the key predictor for whether participants thought other citizens would also “get much out of” such discussions. When asked if others would seek out such discsussions, if they knew of them, participants said “yes” if they’d experienced a discussion in which others were more conversationally skilled; that leaves the impression that such participants guessed public discussions worked best for people more gifted than themselves.

Table 5. Predicting How Citizens Would View Face-To-Face Discussions

How citizens would generally view discussions like this

Feature of discussion Find it valuable Not get much out of participating

Seek it out if they knew it existed

Conversation was to the point -.042 .019 .137

Conversation open to disagreement -.065 -.086 -.184

Identified tensions/tradeoffs .250* .004 .022

Understanding of the facts increased

.259** -.242* .088

Respect and appreciation .160 -.032 .086

Others’ discussion style different -.244** .074 -.140

Others more skilled at communicating that me

.067 -.043 .261**

Clear personality conflicts among members

.148 .021 .040

Asterisks are used to indicate statistically significant results. * indicates p < .05, ** indicates p < .01.

9 This survey item’s response scale was reversed, so that high scores indicated the peception that the discussion was

helfpul.

35

The last set of results, in Table 6, shows participants’ sense of what would happen if such forums were offered far-and-wide (as, in fact, they were in 2012, to no effect). If the conversation didn’t stay “to the point”, participants were inclined to say others would participate. If conversations identified tradeoffs/tensions, participants thought they’d strengthen democracy. The latter perception was also associated with lower style conflicts and greater personality conflicts, a result that suggests that participants think discussions like these could be useful for transcending such interpersonal tension in political discussion. Finally, style differences left participants fearing participation would have no consequence for other participants, although if the group learned facts from the discussion, participants were more hopeful that it would make a difference.

Table 6. Predicting Perceived Utility of Convening Face-To-Face Discussions

Net outcome of such discussions

Feature of discussion Avg citizens wouldn’t

participate

If people did them, our democracy

would be stronger

If people participated,

would not matter

Conversation was to the point -.275** .040 -.041

Conversation open to disagreement .057 -.048 .049

Identified tensions/tradeoffs .020 .261* -.160

Understanding of the facts increased -.029 .146 -.339**

Respect and appreciation -.082 .086 -.023

Others’ discussion style different .055 -.192* .190*

Others more skilled at communicating -.125 -.011 .101

Clear personality conflicts .091 .218* -.090 Asterisks are used to indicate statistically significant results. * indicates p < .05, ** indicates p < .01.

Again, these are just some of the findings we will be extracting from these data, and results may shift when we incorporate the new data we have just finished collecting, which more than doubles our sample.

ONGOING RESEARCH The results presented above are limited, in part, owing to sample size. Thus, we augmented our initial sample with a fall recruitment in October, 2013, using a $40-per-participant stipend. This was lucrative enough to ensure adequate participation, and we are now adding to our dataset 38 new sessions with 208 participants. In total, we will have a sample of 65 face-to-face discussions involving 332 participants. This will be sufficient for more fine-grained analysis, even at the group level, as it’s a sample comparable to some that have provided considerable insight into group discussion.10 Moreover, the aforementioned coding of transcripts will continue through the fall, and that will provide another point of reference for analysis. Thus, we expect to produce more than one full-length article analyzing the impact of different facilitation styles in 2014.

10 See, for example, the 57 group dataset that generated these studies. Gastil, J., Black, L., & Moscovitz, K. (2008). Ideology, attitude change, and deliberation in small face-to-face groups. Political Communication, 25, 23-36. Sager, K. L., & Gastil, J. (1999). Reaching consensus on consensus: A study of the relationships between individual decision-making styles and use of the consensus decision-rule. Communication Quarterly, 47, 67-79.

36

VI. SUMMARY AND RECOMMENDATIONS

MAIN FINDINGS The findings described in this report suggest the Hangout program was successful in affecting some attitudinal changes in its participants, compared to both a control baseline and changes from other comparison programs. In particular, the sessions seem to have imbued participants with a sense of ‘faith in deliberation,’ and

Unsurprisingly, knowledge gain was best delivered by the Face the Facts videos. Although factual learning was not the focus of the Hangouts, it appears they produced learning effects beyond the baseline knowledge of the control group.

The data also furnish some evidence that AmericaSpeaks accomplished its stated attitudinal goals with the Hangouts. Participants were more likely to agree with statements about reforming the political citizen in terms of informedness, engagement, and cross-partisan thinking. Importantly, the Hangout also saw higher evaluations from its participants in terms of how helpful it was for understanding the issue, their belief that the similar online deliberative events would be beneficial to both the people participating in them, and to the broader health of the U.S. democracy. However, participants were not inclined to say that the average citizen would be more inclined to participate in a Hangout-type program than they would in any of the other programs.

Within the Hangouts, it appears that while the participants’ evaluation of the personal benefit of participation is related to their experience of civility during the session, their broader perception of the program’s value to democracy is staked on the facilitator’s performance. As for repeat participation, it appears there is little impact on knowledge – both in terms of actual performance on fact retention questions and in self-reports – but a significant effect on the amount of information participants’ perceive they need. This may mean that repeat participation helps participants ‘get a handle’ on the topic, which is a valuable sort of learning objective.

The media analysis shows that different factual topics addressed by Face the Facts USA found interactive traction on different media; the engagement platform appears to have depended on the complexity and emotional pull of the topic. This dynamic presence has the advantage of momentum after the campaign season; that is, the content appears to have interactive potential even after the campaign season for which it was produced had ended.

The face-to-face deliberation demonstrated that across discernible variations in facilitation style—from a more analytic to a more social approach—it’s possible to maintain high general standards for leading discussions. Future research will tease apart the effects of those different facilitation styles, but it appears that the analytic approach is better at keeping conversations open to disagreement, that the social approach equalizes participation (to limited ultimate effect, later analyses showed), and the balanced style may be the best at raising participants’ issue knowledge.

This augurs well for the balanced style, because when we associated group experiences with outcome variables, the perception of the group increasing its factual knowledge had multiple significant associations—some of the largest ones seen in the data. Participants associated group learning with getting a firmer grasp on the debt/deficit issue, having a valuable discussion, and really making a difference, if only people would participate in such forums. Identification of tensions/tradeoffs, and keeping the conversation on point, were also key predictors, and this suggests a general pattern that the more analytic features of discussion were, in general, more important outcome predictors than the social dimensions, such as respect.

37

The most important social variables were one that has been associated with obstructing democratic deliberation in past research.11 When participants perceived personality conflicts in their group, they found that this stimulated disagreement—but not productive disagreement. Instead, it hindered understanding the issue. Likewise, differences in communication styles led participants to discount the importance of the topic and find the discussions less valuable, both for themselves and for society or democracy as a whole. The one surprising finding in this regard was that the existence of personality conflicts led participants to perceive the discussions as more vital for democracy, perhaps as a corrective to the way personal conflicts can override substance in political debate.

RECOMMENDATIONS Based on our findings, the research team offers three recommendations for future online deliberative engagements. First, the production of short, well produced media pieces like the Face the Facts USA videos appear effective for the delivery of specific factual content. The integration of these pieces, as suggested in AmericaSpeaks discussion guides, could be expected to contribute to a better informed discussion.

Second, it appears that the participants perceive that real understanding of a topic occurs best through the discursive engagement, particularly if it is well facilitated. It appears that facilitators have the ability to directly impact participants’ evaluation of the program as a valuable process. Thus, special attention should be paid to facilitation style and procedures. More investigation into the specifics of a moderator’s influence is advisable.

Finally, the media analysis shows the value of designing multi-channel media executions and web presence. It appears that producing a diversity of engagement strategies allows audiences to interact appropriately to the sophistication and emotional dynamic of the content.

As for the face-to-face deliberation, the clearest findings might be these. First, the emphasis on factual information, focused discussion, and tradeoffs that are the hallmark of National Issues Forums-style discussions, as well as these Face the Facts forums, appears to be well-placed. Such emphasis on the analytic dimension of democratic deliberation resonates with the student participants in these discussions, and the more analytic style of facilitation might help draw out these qualities in discussion. To the extent that one facilitates with a more democratic emphasis, the key might be to attend not to speaking-turn inequality so much as style and personality differences, which can get the group off-track and lead participants to doubt the efficacy of discussion. Thus, a we (in a very preliminary way) advocate a balanced facilitation style that emphasizes factual content learning and manages the potential conflicts in style and temperament among participants.

11 Gastil, J. (1993). Identifying obstacles to small group democracy. Small Group Research, 24, 5-27.

38

APPENDIX A: ONLINE SURVEY EXPERIMENT METHODOLOGY

The online survey experiment was conducted between October 11 and November 5, 2012. Participants were drawn from Amazon Mechanical Turk (AMT). AMT is a website which allows “requesters” to issue web-based tasks to “workers” for completion at a fixed fee. To recruit participants, task descriptions and instructions were posted to the website, along with the fee amount. Table i provides information on these task postings.

Table 1: AMT Task Posting Information

Task Issued Closed Fee ($) # Enrolled

# Approved Completions

Spreecast 1 Oct. 9 Oct. 9 & 10 0.5 + 4 61 61

Hangout Week 1 (10/15-10/18)

Oct. 12 Oct. 14 .01 + 5.49

3 3

Hangout Week 1 (Revised Fee 1)

Oct. 12 & 14 Oct. 14, 15, 16 2 + 6 123 109

Hangout Week 1 (Revised Fee 2)

Oct. 16 Oct. 23 8 25 20

Hangout Week 2 Oct. 18, 22, 23

Oct. 19, 23, 24, 25

8 160 123

Spreecast 2 (10/25)

Oct. 23 Oct. 24 6 200 112

Control Survey Oct. 30 Oct. 30 1 375 375

Text Survey Nov. 5 Nov. 5 1 250 250

Video Survey Nov. 5 Nov. 5 3 350 350

Totals - - - 1172 1028

Because the Hangout participation task is beyond the normal scope of tasks assigned on AMT, both in terms of the duration and activity, the payment structure required some modification. The AMT system allows fees for completion of a task, and allows bonuses for individual workers. Because the time from enrollment to participation in the Hangout was a few days (much longer than the normal AMT task listing), it was desirable to list it a small-fee short-duration task for enrolling in the study, with a sizable bonus for participation and completion of the questionnaire. The initial attempt (line “Hangout Week 1 (10/15-10/18)”) was thus to offer one cent for enrollment, with a promise of a $5.49 bonus. Thus workers could “complete” the task simply to signal their intent to participate, and would be rewarded a bonus upon confirmed participation. This approach only attracted three workers, so the structure was

39

revised to two dollars up-front for enrollment and six dollars upon completion. In an attempt to increase enrollment further, a final modification was offering $8, but the task duration was extended to several days, such that payment was only delivered after confirmed participation in the Hangout. This thus means that the Hangout “Approved Completions” in Table [x] means either completed enrollment or completed participation based on the payment structure of that particular week.

Similarly, the Spreecast structure was modified between the first and second webcast. The first Spreecast task consisted of a 50 cent enrollment fee plus a $4.00 completion bonus, whereas the second Spreecast task was a single $6 fee for both attendance at the webcast and submission of the questionnaire. To confirm attendance, screen names of participants were recorded during the webcast, which were then matched to a field in the questionnaire asking for the screen name. Participants in the first Spreecast were asked to complete a pre-survey to collect demographic information, while participants in the second webcast completed only one single survey after viewing.

After being posted to the AMT task board, 375 control questionnaires were collected in four days, taking an average of 13 minutes to complete. The 250 text exposure questionnaires were collected in two days, taking 25 minutes on average to complete. The video exposure questionnaires were collected in two days, taking 19 minutes on average to complete.

In total, recruitment across all modules cost $4,491.50, of which $4083.13 were payments to participants and $408.37 were paid in fees to AMT.

Demographic information for the five groups is presented in Table ii. The demographics are generally evenly balanced across the module groups, except for a significantly higher representation of females in the Video, Spreecast, and Hangout modules, and a significantly higher representation of minorities (African American and Hispanic) in the Spreecast and Hangout modules. The percentage of participants who identified themselves as Republicans or politically conservative was low (under 25%) for all five conditions.

Table 2: Demographic Information by Group Control Text Video Spreecast Hangout Total

Median Age 31 29 32 31 31 % Male 56.8 55.5 47.1 47.1 46.3

% Democrat 46.7 45.4 49.4 47.1 56.9 % GOP 18.5 19.5 20.1 15.2 10.8

% Politically Liberal 54.5 57.4 55.4 58.1 59.8 % Politically Conservative 19.6 21.2 25.1 23.5 23.5

Modal Educational Level | - Some college, or an associate degree - | % White 84.7 80.9 79.3 77.5 73.4

Median Income Bracket | ------------- $37,500 - $49,999 ------------- |

40

APPENDIX B: GUIDE FOR FACE THE FACTS DISCUSSIONS HELD AT THE PENNSYLVANIA STATE UNIVERSITY IN SPRING, 2013

INTRODUCTION This Guide is intended to provide instructions for hosting an engaging Face the Facts USA conversation in one of two distinct styles.

OVERVIEW Here are some basic facts about your discussion:

It will address the debt and deficit, one of many issues facing the US.

It will last 40-45 minutes, no less and no more. There is plenty of material in this guide to keep it going, and you may need to move the group forward to make sure it finishes on time.

Your discussion will have as few as three participants and fewer than ten. The exact number can’t be known until people show up for a session and get assigned a room for their discussion.

You will facilitate the discussion to either emphasize and encourage analytic rigor or democratic social relations among the participants.

STAGE 1: PREPARATION To prepare for the discussion, you should study this guide thoroughly and practice a discussion in each of the two moderator styles described in this guide. In addition, you must read the Fact Sheet you were provided for the discussion topic. This is what participants will read, and it’s essential that you remember its details well.

STAGE 2: FORUM FACILITATION This includes a welcome, introductions, a series of discussion topics, then a thank you to the group before handing out the questionnaires. This guide details this process.

STAGE 3: POST-DISCUSSION QUESTIONNAIRE After the discussion, you should administer and collect questionnaires from every participant. While the participants are completing their questionnaires, you will fill out a moderator survey, in which you characterize the group you just convened.

WHAT IS FACE THE FACTS USA? Participants might ask you this question. Here’s a quick answer: Face the Facts USA is hosted at The George Washington University School of Media and Public Affairs. AmericaSpeaks is part of the Face the Facts USA team and responsible for producing this Discussion Guide and a variety of online engagement tools. It promotes serious, real-world discussion about the problems we face—slow job growth, the debt and deficit, income inequality, etc.

FACILITATION STYLE Your job is to facilitate the discussion effectively, which means moving through the agenda efficiently and keeping the discussion focused on the student participants, not on yourself. An effective facilitator does just that—facilitates an exchange among the discussion participants

41

that stays on the task at hand. You will begin and end the discussion and interject at key points during the discussion, but be careful to avoid dominating or directing the discussion.

Below are more general tips on facilitation, then a description of the two different facilitation styles we will use in this study.

GENERAL TIPS FOR FACILITATING YOUR DISCUSSION Serve as a Neutral Guide. Your role is to help support the discussion. The best way to do that is to maintain a neutral position and to hold back on voicing your own opinions. Instead, focus on keeping the Discussion going and making sure everyone has a voice.

You Are Not the Expert. Don’t feel that you must be an expert on the issues. Before your Discussion, read the materials you’ve been provided. If questions come up in your Discussion, participants themselves may provide answers, but if none speak up, you can serve as a resource if you know the answer from the Fact Sheet. Do not offer any personal expertise beyond that, lest our different facilitators vary in this regard.

Be Mindful of Time. You will want to pay attention to the time of each task to ensure that you are able to complete each step of the meeting. Do not expect the participants to keep track of time. If the discussion is going well, they will lose track of time as they become engaged in the discussion.

TWO DISTINCT STYLES Though all facilitators will follow the guidelines described above, this study contrasts two variations on facilitation—one focused on rigorous analysis of the facts and the other focused more on ensuring the democratic social relationships among the participants. Both are important elements of democratic deliberation, but in this study, you will emphasize one of these over the other. The following page contrasts these two styles.

Facilitating to Ensure Analytic Rigor Facilitating a Democratic Social Process

Key objective Knowledge gains, insights, and more sophisticated opinions on the issue

A sense of shared respect, the experience of listening deeply, and the sense of being heard by one’s peers

Primary facilitation behaviors

Encourage accurate understanding of the facts, in-depth analysis of issue, and exploration of different points of view and the tradeoffs among them

Encourage a respectful and equal exchange of views, careful listening, and consideration of each person’s point of view

Discussion ground rules to emphasize

• Keep your comments to the point • Remain open to disagreement and

probing questions that help us learn • Look for the tensions and tradeoffs

between different approaches • Always be seeking to sharpen your point

of view and your understanding of the facts

• Try to be brief so that we can share time equitably

• Listen carefully to each other and avoid interruptions

• Consider what each other has to say and feel free to ask clarifying questions

• Show respect and appreciation for each other even when you disagree

Participant Specification of facts, analysis of Turn taking, active listening, vocal appreciation of

42

behaviors to encourage

information, elaboration of different views, open disagreement (and/or opinion change if participants persuade each other)

each other’s views, introspection, and showing respect

Pacing and timing

Keep the discussion moving quickly. Encourage in-depth exchanges when they dig deeper into an issue. Always keep the discussion on-topic and pull it back when it digresses into personal exchanges that don’t advance the discussion.

Be mindful of the stages of the agenda, but let discussion linger when participants are bonding and listening carefully to each other. If the discussion veers too far off topic, gently steer it back.

Equality of participation

Do not concern yourself with the balance of speaking turns across discussion participants. It is more important that the most engaged and knowledgeable participants speak than turns be shared equally. If someone is completely silent for more than fifteen minutes, simply ask them a question or invite them to speak to keep them engaged.

Draw out all participants who are quiet to ensure equal opportunity to speak during the discussion. Check in every so often with those not speaking to keep them engaged.

When to encourage participants

Draw out and encourage those participants who are capable of moving forward the analysis of the issue. Reinforcing this behavior with words like “that’s helpful,” “go on,” and “can you say more?” are only necessary when the most effective participants require prompting.

Show appreciation when participants actively listen to one another, show mutual respect, or draw out their quieter fellow participants. Simply saying “thank you” should suffice, though often nodding is sufficient.

When to interrupt participants

Interrupt or quiet those participants who are seeking premature consensus or moving conversation away from disagreement or in-depth analysis. Remind the group to remember the ground rules you laid out.

Interrupt participants who are shutting down a participant, showing signs of rudeness, or failing to listen to one another. Gently prompt the group to remember the ground rules you laid out.

Questions to encourage

Factual questions, pointed questions each other’s views, probing questions about value tradeoffs

Probing questions about each other’s deeper values and questions seeking clarification/understanding

What to ask participants

• Is that view consist with the facts? • What is the key point here? • Does anyone want to challenge that idea?

[If nobody disagrees]: What would someone who disagrees say?

• Could you give an example to illustrate that point?

• What are the tradeoffs with that position?

• Does anyone share that view? • Who would like to add something? • What do you think of [name]’s view? • Are there any ideas on which most of us agree? • [When participants disagree]: Do you

understand [name’s] point?

Reflections to offer the group

If the group moves into a premature agreement, you might say: “so, it sounds like several of you are saying...” and follow with “but what about [what someone said or a contrary view in the Fact Sheet or Discussion Guide]

As you begin to hear common themes or areas of agreement, you may reflect back to the group, “so, it sounds like several of you are saying...Is that right.” Ideally, the participants do this on their own, without your prompting.

How to spark more discussion

Raise a point of view or introduce information that might stimulate further discussion or disagreement, such as something the group has not considered.

Ask the participants if anyone has shared an experience mentioned earlier or has had any new thoughts after listening to one of the participants.

43

Use of participant names

Simply gesture or look to someone when speaking to them directly. Use participants’ names only when necessary. Use their first name when doing so.

Whenever possible, add a participant’s first name when addressing him or her, as in “Thank you, Karen.” Only use first names.

THE MEETING AGENDA We provide both a summary and detailed agenda. The summary agenda can serve as a quick one-page reference, and the detailed agenda gives you more specific instruction on how to handle each step of the agenda. You may choose to refer to the detailed agenda when you need more guidance during the discussion. Be sure to have studied the full agenda carefully well before the discussion.

Agenda Item Activity Summary for Discussion Hosts

Prior to Meeting Start Make sure each participant is in the correct room. Help participants feel welcome as they arrive.

Ensure there are enough fact sheets, questionnaires, and pens for the expected number of participants.

Make sure you have a copy of the facilitator survey Be sure to ask them to complete IRB Consent Form. Check the forms

for consent and file. Anyone not consenting may leave and email Dave ([email protected]) for the alternate assignment.

Distribute the pre-discussion fact sheet along with the questionnaire. Give everyone 4-5 minutes to read over the information

1) Welcome, overview, and introductions DURATION: 5 mins.

Welcome participants and explain the meeting’s purpose.

Summarize what will happen during the discussion.

Introduce the guidelines for group discussion.

Ask each person to say his/her name and hometown [gotta keep it short]

2) Learning about the issue DURATION: 10-15 mins.

Read a brief statement introducing the issue.

Ask everyone in the group if they have had a chance to read the background information

3) Issue discussion DURATION: 20-25 mins.

Pose a variety of questions that engage participants in discussing the issue.

After some general discussion, direct participants to the section of the Issue Fact Sheet that lists different approaches to the issue and ask some questions to probe their thoughts about those ideas.

4) Concluding Comments and Commitments DURATION: 5 mins.

Ask what else people want to know about this issue

Invite descriptions of any actions participants plan to take to be a more active and effective citizen.

After the Discussion Distribute and collect participant questionnaires

Collect participant ID numbers to confirm participation

Complete the facilitator survey; on the back of the survey, attach the participants’ nametags to this form

mailto:[email protected]

44

APPENDIX C: DEBT AND DEFICIT PRE-DISCUSSION FACT SHEET

The debt hanging over our nation’s head is $16 trillion and the federal government spends far more than we collect in taxes. The problem is bad now, but expected to get even worse as the baby boomer generation ages. Where do we spend our money? What options does the federal government have? How does the debt impact us?

To start, let’s get some definitions clear:

The deficit is the amount that spending exceeds revenues in a given year. The government must borrow this additional amount to pay its bills.

The debt is the total amount of deficits (offset by the total amount of surpluses) that the government has accumulated over time.

According to the Congressional Budget Office, the federal government’s budget deficit for fiscal year 2011 was 1.3 trillion, the third largest shortfall in the past 40 years. In fiscal year 2011, the government spent $3.6 trillion, but received only $2.3 trillion in revenues.

FACT: The debt rises every day, at a rate of $3 billion daily in the first half of 2012. The government spent $114,253 per second in 2011 while only $73,043 in revenue came in per second.

So, when thinking about the deficit, we’re thinking about spending, revenues, and borrowing. Let’s take a closer look at all three.

Where does the federal government spend its money?

FACT: Interest on the debt alone is the 4th largest expenditure of tax dollars. $3 billion a day goes to service the debt.

45

What are the sources of revenue that the government collects?

FACT: Federal tax revenue is at its lowest level in 60 yrs, 15% of GDP, well below historical average of 18%.

What is GDP? Gross Domestic Product (GDP) is the sum of a nation’s economic output. Everything people, companies and the government spend on goods, services, and investment in the country.

46

Where do we borrow money from? How much do we borrow?

The chart below from the Washington Post provides answers.

47

What are the implications of the deficit and the national debt?

If the federal government takes on too much debt, its creditors might doubt its ability to pay the money back. They may charge more interest or even refuse to lend more money. This could negatively impact the stock market and the economy at large. As more money is spent on the debt at the federal level, the less money there is available to support state and local services. Moreover, many states and local jurisdictions are prohibited from running deficits. When their revenues fall and federal funding isn’t available to fill the gap, they have to reduce services and programs.

The federal government might seem far away, but its funding choices can be felt on a personal level. If you or your family members use Medicare, Social Security, public transportation, or public schools, or even drive on interstate highways, you’re using governmental services supported by federal dollars. The federal government’s choices about the debt may impact the level of services that you receive in your community and may also result in higher taxes on individuals.

Regarding the budget deficit and the national debt, there is some agreement:

Many agree that the projected level of spending into the future is unsustainable because of the aging of our population and rising healthcare costs pressuring entitlement program growth.

Most also agree that the longer we wait to address the long-term debt trajectory, the harder our choices will be in the future.

There is also broad agreement that, while an economic revival would help reduce the debt, it alone will not solve the problem.

Many people also believe that viable long term solutions to the national debt will require some combination of tax reform, entitlement reform and cuts in other government spending.

There is also disagreement. Many believe that that the national debt is a crisis that requires immediate action, from spending cuts to entitlement reforms. Others who are concerned about the debt believe that, in times of economic downturn, a sharp reduction in government spending will only cause more short term economic problems. They support more limited budget cuts coupled with tax increases—especially on wealthy Americans.

What do you think about the federal debt and deficit? What other information would you like to know? What do you think we should do? This is what we will discuss today.

48

APPENDIX D: QUESTIONNAIRE FOR FACE-TO-FACE DISCUSSIONS

This study asks for your opinion about the discussion you’ve participated in. All responses to this survey are confidential. Before completing this questionnaire, you should have completed the IRB informed consent form – please ask your discussion facilitator if you have not yet done so.

Nametag Number:

_____________________________

1. About how long did you spend reading the pre-discussion fact sheet?.

_____ minutes

Please indicate your agreement or disagreement with each statement by circling one number. There are no correct or incorrect responses to these opinion questions.

2. We have gone too far in pushing equal rights in this country.

Strongly Disagree : 1 : 2 : 3 : 4 : 5 : Strongly Agree

3. Society as a whole has become too soft and feminine.


4. It seems like blacks, women, homosexuals and other groups don’t want equal rights, they

want special rights just for them.


5. It’s not the government’s business to try to protect people from themselves.


6. Too many people today expect society to do things for them that they should be doing for

themselves.


7. The government should stop telling people how to live their lives.


8. The government should do more to advance society’s goals, even if that means limiting the

freedom and choices of individuals.

49


9. Sometimes government needs to make laws that keep people from hurting themselves.


10. Government should put limits on the choices individuals can make so they don’t get in the

way of what’s good for society.


11. Our society would be better off if the distribution of wealth was more equal.


12. Discrimination against minorities is still a very serious problem in our society.


13. We need to dramatically reduce inequalities between the rich and the poor, whites and

people of color, and men and women.


14. I usually discuss politics with people of the same ethnic, social, and economic background as

myself.


15. Even people who strongly disagree can make sound decisions if they sit down and talk.


16. Everyday people from different parties can have civil, respectful conversations about

politics.


17. The first step in solving our common problems is to discuss them together.


18. Individual leaders make better decisions than groups like committees, more often than not.


19. I don’t think public officials care much about what people like me think.

50


20. Under our form of government, the people have the final say about how the country is run,

no matter who is in office.


21. There are many legal ways for citizens to successfully influence what government does.


22. People like me don't have any say about what the government does.


23. When we hear claims that are false or misleading, we have a responsibility to speak up and

CORRECT them.


24. People who make political claims must ALWAYS back up their arguments with solid

EVIDENCE.


25. People should always present LOGICAL arguments in support of their views.


The following questions ask about your experience during the discussion. Please indicate your agreement or disagreement with each statement.

26. I had PLENTY of chances to speak during our group discussion.


27. I felt REJECTED by the other group members because of the views I expressed.


28. I felt that the other group members LISTENED carefully to what I had to say.


29. The other group members RESPECTED the views I expressed during the discussion.

51


30. Some of the other group members were RUDE towards me during the discussion.


31. I think the other group members UNDERSTOOD what I had to say.


32. I carefully CONSIDERED what other group members said during our discussion.


33. I tried NOT TO EXCLUDE anyone from the group, even if their views were different.


34. I tried very hard to treat the other group members with RESPECT.


35. I UNDERSTOOD almost everything that other group members said during our discussion.


36. I was POLITE during the discussion, even if others disagreed with me.


37. The other group members had a discussion style that was very different from my own style.


38. The other group members were more skilled at communicating that I was.


39. There were some “personal” conflicts during our group’s discussion.


40. There were some clear personality conflicts between some of the members of our group.


41. Generally, average citizens would find participating in this sort of discussion valuable.

52


42. Most citizens would NOT get much out of participating in a discussion like this.


43. Citizens would seek out this sort of discussion if they knew it existed.


44. Most likely, the average citizen would be UNLIKELY to spend time participating in a

discussion like this.


45. If more people participated in this sort of discussion, our democracy would be stronger.


46. Even if many people participated in discussions like these, it would NOT make much of a

difference.


47. How much disagreement was there during your discussion? (circle one)

1. Almost no disagreement at all

2. A little bit of disagreement

3. A lot of disagreement

4. Almost constant disagreement

48. Overall, did the discussion help you better understand an issue or a political topic? (circle

one)

1. Yes it was very helpful

2. Yes, it was somewhat helpful

3. Yes, it was a little helpful

4. No, it was not very helpful

49. Do you think that the topic addressed was an important one?

53

Very Unimportant : 1 : 2 : 3 : 4 : 5 : Very Important

The following questions ask about the person who facilitated your conversation. Please indicate your agreement or disagreement with each statement.

50. The facilitator seemed to disrupt the conversation unnecessarily.


51. The facilitator effectively kept the conversation moving forward.


52. The facilitator was helpful in guiding the discussion.


53. The facilitator intervened to interject his or her own opinions.


54. Different facilitators have different styles. How would you characterize your facilitator’s

goals for this session?

A) Keeping the conversation to the point:

Not Emphasized : 1 : 2 : 3 : 4 : 5 : Strongly Emphasized

B) Keeping the conversation open to disagreement:


C) Looking for the tensions and tradeoffs between different approaches:


D) Increasing the group’s understanding of the facts:


E) Sharing speaking time equitably between discussion participants:


F) Showing respect and appreciation for each discussion participant:


54

55. Regardless of the facilitator’s efforts, how well does each of the following actually describe

the conversation you had?

G) The conversation was to the point:

Does NOT Describe : 1 : 2 : 3 : 4 : 5 : Describes Well

H) The conversation was open to disagreement:


I) During the conversation, we identified the tensions and tradeoffs between different

approaches:


J) The group’s understanding of the facts increased during the discussion:


K) Speaking time was shared equitably between discussion participants:


L) Respect and appreciation was shown to each discussion participant:


56. Did you find that beliefs you held were challenged by arguments or comments in the discussion? If so, please tell us about one such instance in detail. Include the belief and why it was challenged, and whether or not you changed your mind as a result.

57. Name a few facts, arguments, or comments in the discussion that you found interesting or

surprising, and tell us why you were interested or surprised.

55

58. Which comes closest to representing your political philosophy?

Far Left (Liberal) : 1 : 2 : 3 : 4 : 5 : Far Right (Conservative) Moderate

59. With what political party do you identify?

Democratic Republican

No party affiliation Other party affiliation

If “other,” please indicate here ______________________________________

60. Are you a United States citizen?

Yes No (If no, skip question 61)

I do not wish to share this information.

61. Are you a registered voter?

Yes No


62. What is the highest level of education you have completed? (check one)

Some high school High school graduate

Some college College graduate Graduate degree


1

63. In what year were you born? 19______


64. What is your sex?

Male Female


65. What is your ethnic origin or race?

Black/African American Hispanic Asian or Pacific Islander American Indian White Other I do not wish to share this information

evaluation and analysis of public discussion and deliberation in … · 2016-04-19 · i evaluation...

Documents