supporting comment moderators in identifying high quality
TRANSCRIPT
Supporting Comment Moderators in Identifying High
Quality Online News CommentsDeokgun Park, Simranjit Sachar, Nicholas Diakopoulos, Niklas Elmqvist
University of MarylandCollege Park, MD, USA
{intuinno, ssachar, nad, elm}@umd.edu
HumanComputerInteractionLaboratory
Comments
In an ideal world,
In reality,
To save the world
6,367 submitted
6,367 submitted
4,447 accepted
6,367 submitted
4,447 accepted
1,920 filtered
6,367 submitted
4,447 accepted
1,920 filtered
In 3 hours
9,000 comments per day
Good comments
Heroic
Drawbacks?
Scalable way?
The damage is already done.
Wait
INST 728Q – Visual Analytics 27
INST 728Q – Visual Analytics 28
INST 728Q – Visual Analytics 29
The world of comments
BestWorst
BestWorst
RedundantNo new informationConsensus
BestWorst
RedundantNo new informationConsensus
IrrelevantAdd noise
BestWorst
RedundantNo new informationConsensus
IrrelevantAdd noise
InappropriateHarmful
Negative Loop
A Bad comment
A Bad comment
Journal of Communication ISSN 0021-9916
ORIGINAL ARTICLE
Online and Uncivil? Patternsand Determinants of Incivility in NewspaperWebsite CommentsKevin Coe1, Kate Kenski2,3, & Stephen A. Rains2
1 Department of Communication, University of Utah, Salt Lake City, UT 84112, USA2 Department of Communication, University of Arizona, Tucson, AZ 85721, USA3 Department of Government & Public Policy, University of Arizona, Tucson, AZ 85721, USA
Incivility in public discussions has received increasing attention from academic and popularcommentators in recent years. In an effort to better understand the nature and determi-nants of such incivility, this study examined a 3-week census of articles and commentsposted to a local newspaper’s website—totaling more than 300 articles and 6,400 com-ments. The results of the content analysis show that incivility occurs frequently and is asso-ciated with key contextual factors, such as the topic of the article and the sources quotedwithin the article. We also find that, contrary to popular perceptions, frequent commentersare more civil than are infrequent commenters, and uncivil commenters are no less likelythan civil commenters to use evidence in support of their claims.
doi:10.1111/jcom.12104
Civility is a crucial principle of public life, one that speaks to “the fundamental toneand practice of democracy” (Herbst, 2010, p. 3). Indeed, a commitment to civildiscourse—the free and respectful exchange of ideas—has been viewed as a demo-cratic ideal from the ancient Athenian forums to the mediated political debates ofmodern times (Papacharissi, 2004; Sapiro, 1999). This is not to say the ideal is alwaysrealized. Public discourse has always had its share of incivility, and the current era isno different in this respect. What is different now, however, is that the 21st century’svast, interactive media environment has created broader opportunities for publicdebate, and that moments of incivility now spread more rapidly and widely than everbefore (Sobieraj & Berry, 2011). In this milieu, incivility has become a central concernof citizens and scholars. For example, a 2010 survey found that more than 8 in 10Americans viewed “the lack of civil or respectful discourse in our political system” asa “somewhat serious” or “very serious” problem (Public Religion Research Institute,2010). Across the United States, various organizations have been created in recent
Corresponding author: Kevin Coe; e-mail: [email protected]
658 Journal of Communication 64 (2014) 658–679 © 2014 International Communication Association
Journal of Communication ISSN 0021-9916
ORIGINAL ARTICLE
Online and Uncivil? Patternsand Determinants of Incivility in NewspaperWebsite CommentsKevin Coe1, Kate Kenski2,3, & Stephen A. Rains2
1 Department of Communication, University of Utah, Salt Lake City, UT 84112, USA2 Department of Communication, University of Arizona, Tucson, AZ 85721, USA3 Department of Government & Public Policy, University of Arizona, Tucson, AZ 85721, USA
Incivility in public discussions has received increasing attention from academic and popularcommentators in recent years. In an effort to better understand the nature and determi-nants of such incivility, this study examined a 3-week census of articles and commentsposted to a local newspaper’s website—totaling more than 300 articles and 6,400 com-ments. The results of the content analysis show that incivility occurs frequently and is asso-ciated with key contextual factors, such as the topic of the article and the sources quotedwithin the article. We also find that, contrary to popular perceptions, frequent commentersare more civil than are infrequent commenters, and uncivil commenters are no less likelythan civil commenters to use evidence in support of their claims.
doi:10.1111/jcom.12104
Civility is a crucial principle of public life, one that speaks to “the fundamental toneand practice of democracy” (Herbst, 2010, p. 3). Indeed, a commitment to civildiscourse—the free and respectful exchange of ideas—has been viewed as a demo-cratic ideal from the ancient Athenian forums to the mediated political debates ofmodern times (Papacharissi, 2004; Sapiro, 1999). This is not to say the ideal is alwaysrealized. Public discourse has always had its share of incivility, and the current era isno different in this respect. What is different now, however, is that the 21st century’svast, interactive media environment has created broader opportunities for publicdebate, and that moments of incivility now spread more rapidly and widely than everbefore (Sobieraj & Berry, 2011). In this milieu, incivility has become a central concernof citizens and scholars. For example, a 2010 survey found that more than 8 in 10Americans viewed “the lack of civil or respectful discourse in our political system” asa “somewhat serious” or “very serious” problem (Public Religion Research Institute,2010). Across the United States, various organizations have been created in recent
Corresponding author: Kevin Coe; e-mail: [email protected]
658 Journal of Communication 64 (2014) 658–679 © 2014 International Communication Association
Journal of Computer-Mediated Communication
The ‘‘Nasty Effect:’’ Online Incivility and RiskPerceptions of Emerging Technologies∗
Ashley A. Anderson
Center for Climate Change Communication, George Mason UniversityUniversity of Wisconsin-Madison Nanoscale Science and Engineering Center in Templated Synthesis andAssembly at the Nanoscale
Dominique Brossard
Department of Life Sciences Communication, University of Wisconsin-MadisonCenter for Nanotechnology in Society at Arizona State UniversityUniversity of Wisconsin-Madison Nanoscale Science and Engineering Center in Templated Synthesis andAssembly at the Nanoscale
Dietram A. Scheufele
Department of Life Sciences Communication, University of Wisconsin-MadisonCenter for Nanotechnology in Society at Arizona State UniversityUniversity of Wisconsin-Madison Nanoscale Science and Engineering Center in Templated Synthesis andAssembly at the Nanoscale
Michael A. Xenos
University of Wisconsin-Madison Nanoscale Science and Engineering Center in Templated Synthesis andAssembly at the NanoscaleDepartment of Communication Arts, University of Wisconsin-Madison
Peter LadwigDepartment of Life Sciences Communication, University of Wisconsin-MadisonUniversity of Wisconsin-Madison Nanoscale Science and Engineering Center in Templated Synthesis andAssembly at the Nanoscale
Uncivil discourse is a growing concern in American rhetoric, and this trend has expanded beyondtraditional media to online sources, such as audience comments. Using an experiment given to asample representative of the U.S. population, we examine the effects online incivility on perceptions
Paper forthcoming in Journal of Computer-Mediated CommunicationAll correspondence regarding this manuscript should be addressed to the first author in the Center for Climate ChangeCommunication, George Mason University, Mail Stop 6A8, 4400 University Drive, Fairfax, VA, 22030 ph: 703-993-8368;e-mail: [email protected] material is based upon work supported by a grant from the National Science Foundation to the UW-MadisonNanoscale Science and Engineering Center in Templated Synthesis and Assembly at the Nanoscale (Grant No. SES-DMR-0832760). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authorsand do not necessarily reflect the views of the National Science Foundation.∗Accepted by previous editor Maria Bakardjieva
Journal of Computer-Mediated Communication 19 (2014) 373–387 © 2013 International Communication Association 373
BestWorst
RedundantNo new information
IrrelevantAdd noise
InappropriateHarmful
Negative Loop
High QualityInspiring
Add information
BestWorst
RedundantNo new information
IrrelevantAdd noise
InappropriateHarmful
Negative Loop
High QualityInspiring
Add informationPositive Loop
Normative Influences on Thoughtful Online Participation Abhay Sukumaran1, Stephanie Vezich2, Melanie McHugh1, Clifford Nass1
1Department of Communication 450 Serra Mall
Stanford University, Stanford, CA 94305 {abhays, mmchugh, nass}@stanford.edu
2Department of Psychology University of California Los Angeles
1285 Franz Hall, Los Angeles, CA 90095 [email protected]
ABSTRACT We describe two experiments on whether individual thoughtful effort during online commenting is shaped by situational norms derived from the behavior of social others and the design of the environment, respectively. By measuring the length of participants’ comments on a news website, the time taken to write them, and the number of issue-relevant thoughts they contain, we demonstrate that participants conform to high vs. low norms of thoughtfulness manifested through either the apparent behavior of other users or through visual, textual and interactional design features conceptually associated with thoughtfulness. Theoretical and applied insights for designing online participatory environments are discussed.
Author Keywords Online comments, user-generated content, thoughtfulness, social norms, environmental norms.
ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
ACM General Terms Experimentation.
INTRODUCTION A widespread form of public discourse is conducted through participation in online comments on shared media such as news stories and blog posts [25]. While such participation holds much potential for enabling a more participatory media culture [14] and fostering offline civic engagement [41], online comment spaces are often plagued by banality and thoughtlessness rather than edifying discussion. Improving participatory online behavior is thus a central concern for designers.
While such regulation has traditionally been achieved through moderation systems [18], recent research has also focused on subjective perceptions that people form about their online social situations. Online contributions can be motivated, for example, by the individual’s perception of their level of participation relative to others [4] or their degree of identification with a social group [33]. In the physical world, interpersonal situations are rife with behavioral influences traceable to social norms, i.e., shared standards of expected behavior inferred from what others are doing [6]. Research also suggests that norms can be derived not only from other people but also from features of the immediate environment that are mentally associated with specific categories of social behavior [1, 15]. We suggest that online comment spaces present fundamentally social yet rather ambiguous situations that are likely to be subject to such normative influence.
In two experimental studies, we explore whether thoughtful participation in online comment spaces can be induced via social norms. The first experiment shows that people tend to conform to standards of thoughtfulness in commenting behavior set by others. The second experiment demonstrates that a similar effect can be achieved by introducing design elements conceptually associated with thoughtfulness in an online comment space. We interpret these results in light of social psychological theories of how environmental and social cues influence perception and behavior in interpersonal situations, and discuss the implications of these findings for designers of participatory technologies.
RELATED WORK We briefly review prior research to establish support for two related ideas that correspond to these two empirical studies. First, individuals in social settings often turn to normative indicators in the behavior of others to guide their own decisions and actions, particularly when the setting is ambiguous. Second, even when others are not present, social environments can possess features mentally associated with such standards of normative conduct. Whenever available, we also cite evidence that such effects can exert themselves in online settings as well as physical ones.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00.
CHI 2011 • Session: Incentives & User Generated Content May 7–12, 2011 • Vancouver, BC, Canada
3401
Normative Influences on Thoughtful Online Participation Abhay Sukumaran1, Stephanie Vezich2, Melanie McHugh1, Clifford Nass1
1Department of Communication 450 Serra Mall
Stanford University, Stanford, CA 94305 {abhays, mmchugh, nass}@stanford.edu
2Department of Psychology University of California Los Angeles
1285 Franz Hall, Los Angeles, CA 90095 [email protected]
ABSTRACT We describe two experiments on whether individual thoughtful effort during online commenting is shaped by situational norms derived from the behavior of social others and the design of the environment, respectively. By measuring the length of participants’ comments on a news website, the time taken to write them, and the number of issue-relevant thoughts they contain, we demonstrate that participants conform to high vs. low norms of thoughtfulness manifested through either the apparent behavior of other users or through visual, textual and interactional design features conceptually associated with thoughtfulness. Theoretical and applied insights for designing online participatory environments are discussed.
Author Keywords Online comments, user-generated content, thoughtfulness, social norms, environmental norms.
ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
ACM General Terms Experimentation.
INTRODUCTION A widespread form of public discourse is conducted through participation in online comments on shared media such as news stories and blog posts [25]. While such participation holds much potential for enabling a more participatory media culture [14] and fostering offline civic engagement [41], online comment spaces are often plagued by banality and thoughtlessness rather than edifying discussion. Improving participatory online behavior is thus a central concern for designers.
While such regulation has traditionally been achieved through moderation systems [18], recent research has also focused on subjective perceptions that people form about their online social situations. Online contributions can be motivated, for example, by the individual’s perception of their level of participation relative to others [4] or their degree of identification with a social group [33]. In the physical world, interpersonal situations are rife with behavioral influences traceable to social norms, i.e., shared standards of expected behavior inferred from what others are doing [6]. Research also suggests that norms can be derived not only from other people but also from features of the immediate environment that are mentally associated with specific categories of social behavior [1, 15]. We suggest that online comment spaces present fundamentally social yet rather ambiguous situations that are likely to be subject to such normative influence.
In two experimental studies, we explore whether thoughtful participation in online comment spaces can be induced via social norms. The first experiment shows that people tend to conform to standards of thoughtfulness in commenting behavior set by others. The second experiment demonstrates that a similar effect can be achieved by introducing design elements conceptually associated with thoughtfulness in an online comment space. We interpret these results in light of social psychological theories of how environmental and social cues influence perception and behavior in interpersonal situations, and discuss the implications of these findings for designers of participatory technologies.
RELATED WORK We briefly review prior research to establish support for two related ideas that correspond to these two empirical studies. First, individuals in social settings often turn to normative indicators in the behavior of others to guide their own decisions and actions, particularly when the setting is ambiguous. Second, even when others are not present, social environments can possess features mentally associated with such standards of normative conduct. Whenever available, we also cite evidence that such effects can exert themselves in online settings as well as physical ones.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00.
CHI 2011 • Session: Incentives & User Generated Content May 7–12, 2011 • Vancouver, BC, Canada
3401
Journal of Computer-Mediated Communication
Changing Deliberative Norms on NewsOrganizations’ Facebook Sites
Natalie Jomini StroudDepartment of Communication Studies, University of Texas, 2504A Whitis Ave. (A1105), Austin, TX 78712-0115
Joshua M. ScaccoBrian Lamb School of Communication, Purdue University, 100 N. University St., Beering Hall 2154, WestLafayette, IN 47907-2098
Ashley MuddimanDepartment of Communication and Journalism, University of Wyoming, Ross Hall, 1000 E. University Ave., Dept.3904, Laramie, WY 82071
Alexander L. CurryDepartment of Communication Studies, University of Texas, 2504A Whitis Ave. (A1105), Austin, TX 78712-0115
Comments posted to news sites do not always live up to the ideals of deliberative theorists. Drawingfrom theories about deliberation and group norms, this study investigates whether news organi-zations can affect comment section norms by engaging directly with commenters. We conducted afield study with a local television station in a top-50 Designated Market Area. For 70 political postsmade on different days, we randomized whether an unidentified staff member from the station, arecognizable political reporter, or no one engaged with commenters. We assessed if these changesaffected whether the comments (n = 2,403) were civil, were relevant, contained genuine questions,and provided evidence. The findings indicate that a news organization can affect the deliberativebehavior of commenters.
Keywords: Online Forums, Deliberation, Journalism, Civility, News.
doi:10.1111/jcc4.12104
Comments posted in response to news stories, whether on a Facebook page or in a comment sectionbeneath an article, do not always live up to the ideals of deliberative theorists. As opposed to instances ofthoughtful, reasoned discourse, comments are often irrelevant, uncivil, and misleading (Coe, Kenski, &
The authors thank the Democracy Fund for funding this project and our news station partner for making this researchpossible as part of the Engaging News Project and Cynthia Peacock for her coding assistance.Editorial Record: First manuscript received on March 08, 2014. Revisions received on July 01, 2014. Accepted by JenniferStromer-Galley on August 01, 2014. Final manuscript received on August 08, 2014. First published online on October 15,2014.Correction made after online publication October 15, 2014: Natalie Jomini Stroud’s name has been updated.
188 Journal of Computer-Mediated Communication 20 (2015) 188–203 © 2014 International Communication Association
Normative Influences on Thoughtful Online Participation Abhay Sukumaran1, Stephanie Vezich2, Melanie McHugh1, Clifford Nass1
1Department of Communication 450 Serra Mall
Stanford University, Stanford, CA 94305 {abhays, mmchugh, nass}@stanford.edu
2Department of Psychology University of California Los Angeles
1285 Franz Hall, Los Angeles, CA 90095 [email protected]
ABSTRACT We describe two experiments on whether individual thoughtful effort during online commenting is shaped by situational norms derived from the behavior of social others and the design of the environment, respectively. By measuring the length of participants’ comments on a news website, the time taken to write them, and the number of issue-relevant thoughts they contain, we demonstrate that participants conform to high vs. low norms of thoughtfulness manifested through either the apparent behavior of other users or through visual, textual and interactional design features conceptually associated with thoughtfulness. Theoretical and applied insights for designing online participatory environments are discussed.
Author Keywords Online comments, user-generated content, thoughtfulness, social norms, environmental norms.
ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
ACM General Terms Experimentation.
INTRODUCTION A widespread form of public discourse is conducted through participation in online comments on shared media such as news stories and blog posts [25]. While such participation holds much potential for enabling a more participatory media culture [14] and fostering offline civic engagement [41], online comment spaces are often plagued by banality and thoughtlessness rather than edifying discussion. Improving participatory online behavior is thus a central concern for designers.
While such regulation has traditionally been achieved through moderation systems [18], recent research has also focused on subjective perceptions that people form about their online social situations. Online contributions can be motivated, for example, by the individual’s perception of their level of participation relative to others [4] or their degree of identification with a social group [33]. In the physical world, interpersonal situations are rife with behavioral influences traceable to social norms, i.e., shared standards of expected behavior inferred from what others are doing [6]. Research also suggests that norms can be derived not only from other people but also from features of the immediate environment that are mentally associated with specific categories of social behavior [1, 15]. We suggest that online comment spaces present fundamentally social yet rather ambiguous situations that are likely to be subject to such normative influence.
In two experimental studies, we explore whether thoughtful participation in online comment spaces can be induced via social norms. The first experiment shows that people tend to conform to standards of thoughtfulness in commenting behavior set by others. The second experiment demonstrates that a similar effect can be achieved by introducing design elements conceptually associated with thoughtfulness in an online comment space. We interpret these results in light of social psychological theories of how environmental and social cues influence perception and behavior in interpersonal situations, and discuss the implications of these findings for designers of participatory technologies.
RELATED WORK We briefly review prior research to establish support for two related ideas that correspond to these two empirical studies. First, individuals in social settings often turn to normative indicators in the behavior of others to guide their own decisions and actions, particularly when the setting is ambiguous. Second, even when others are not present, social environments can possess features mentally associated with such standards of normative conduct. Whenever available, we also cite evidence that such effects can exert themselves in online settings as well as physical ones.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00.
CHI 2011 • Session: Incentives & User Generated Content May 7–12, 2011 • Vancouver, BC, Canada
3401
Journal of Computer-Mediated Communication
Changing Deliberative Norms on NewsOrganizations’ Facebook Sites
Natalie Jomini StroudDepartment of Communication Studies, University of Texas, 2504A Whitis Ave. (A1105), Austin, TX 78712-0115
Joshua M. ScaccoBrian Lamb School of Communication, Purdue University, 100 N. University St., Beering Hall 2154, WestLafayette, IN 47907-2098
Ashley MuddimanDepartment of Communication and Journalism, University of Wyoming, Ross Hall, 1000 E. University Ave., Dept.3904, Laramie, WY 82071
Alexander L. CurryDepartment of Communication Studies, University of Texas, 2504A Whitis Ave. (A1105), Austin, TX 78712-0115
Comments posted to news sites do not always live up to the ideals of deliberative theorists. Drawingfrom theories about deliberation and group norms, this study investigates whether news organi-zations can affect comment section norms by engaging directly with commenters. We conducted afield study with a local television station in a top-50 Designated Market Area. For 70 political postsmade on different days, we randomized whether an unidentified staff member from the station, arecognizable political reporter, or no one engaged with commenters. We assessed if these changesaffected whether the comments (n = 2,403) were civil, were relevant, contained genuine questions,and provided evidence. The findings indicate that a news organization can affect the deliberativebehavior of commenters.
Keywords: Online Forums, Deliberation, Journalism, Civility, News.
doi:10.1111/jcc4.12104
Comments posted in response to news stories, whether on a Facebook page or in a comment sectionbeneath an article, do not always live up to the ideals of deliberative theorists. As opposed to instances ofthoughtful, reasoned discourse, comments are often irrelevant, uncivil, and misleading (Coe, Kenski, &
The authors thank the Democracy Fund for funding this project and our news station partner for making this researchpossible as part of the Engaging News Project and Cynthia Peacock for her coding assistance.Editorial Record: First manuscript received on March 08, 2014. Revisions received on July 01, 2014. Accepted by JenniferStromer-Galley on August 01, 2014. Final manuscript received on August 08, 2014. First published online on October 15,2014.Correction made after online publication October 15, 2014: Natalie Jomini Stroud’s name has been updated.
188 Journal of Computer-Mediated Communication 20 (2015) 188–203 © 2014 International Communication Association
Moderation, Response Rate, and MessageInteractivity: Features of OnlineCommunities and Their Effects on Intentto Participate
Kevin Wise
Missouri School of Journalism
University of Missouri-Columbia
Brian Hamman
Missouri School of Journalism
University of Missouri-Columbia
Kjerstin Thorson
School of Journalism & Mass Communication
University of Wisconsin-Madison
AbstractWe conducted two experiments to explore how moderation, response rate, and messageinteractivity affected people’s intent to participate in a web-based online community. Inour first experiment, 62 participants observed either a moderated or an unmoderatedonline community and answered questions about their intent to participate in the commu-nity. The participants who viewed the moderated community reported significantly higherintent to participate than participants who viewed the unmoderated community. In oursecond experiment, 59 participants observed a different online community in which wemanipulated both the rate (in time) of posted comments and the interactivity of each com-ment. We derived our manipulation of interactivity from Rafaeli’s (1988) definition ofinteractivity as message contingency. Participants reported significantly greater intent toparticipate in an online community featuring interactive messages, but only when responserate was slow. These results indicate that both structural features of interfaces and contentfeatures of interactions affect people’s intent to participate in online communities.
doi:10.1111/j.1083-6101.2006.00313.x
Introduction
Once the realm of the technically savvy, the Internet has become a practical com-munication channel for people of all abilities. While many people use the Internet tocommunicate with individuals through tools such as email and instant messaging,others use it to communicate with larger groups by posting their ideas to blogs,
Journal of Computer-Mediated Communication
24 Journal of Computer-Mediated Communication 12 (2006) 24–41 ª 2006 International Communication Association
HOW CAN WE SELECT GOOD COMMENTS?
HOW CAN WE SELECT GOOD COMMENTS?
(IN A MORE SCALABLE WAY)
Wait!What do you mean by good comments?
Methodology
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Design Study Methodology: Reflections from the Trenches and the StacksMichael Sedlmair, Miriah Meyer, and Tamara MunznerIEEE Trans. Visualization and Computer Graphics (Proc. InfoVis), 18(12): 2431-2440, 2012.
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design Evaluate
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design Evaluate
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design Evaluate
Target User Interviews
Bassey Etim, community manager at NYT
“We use real people because humans can absorb the variables of conversation and weigh them in more intricate ways.”
Give moderator power to sort as he/she wants
“We use real people because humans can absorb the variables of conversation and weigh them in more intricate ways.”
“The second comes from a fear that the thoughts you’re absorbing will seep into your own opinions.”
Give moderator power to sort as he/she wants
“We use real people because humans can absorb the variables of conversation and weigh them in more intricate ways.”
“The second comes from a fear that the thoughts you’re absorbing will seep into your own opinions.”
Give moderator power to sort as he/she wants
Need for balanced view
“We use real people because humans can absorb the variables of conversation and weigh them in more intricate ways.”
“Want to go up to the reporter and say here are the 10 from commenters that are from more thoughtful people in comments. “
“Want to go up to the reporter and say here are the 10 from commenters that are from more thoughtful people in comments. “
Multiple use case
“The comments section often reads like a PhD seminar. I am quite certain that your readers can and would offer up insightful but wry and amusing comments.”
“Want to go up to the reporter and say here are the 10 from commenters that are from more thoughtful people in comments. “
Commenter-based tasks
“The comments section often reads like a PhD seminar. I am quite certain that your readers can and would offer up insightful but wry and amusing comments.”
“Want to go up to the reporter and say here are the 10 from commenters that are from more thoughtful people in comments. “
Commenter-based tasks
Unexpected comments
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design Evaluate
VISUALANALYTICS
VS
Outliers
Trends
Get answers to the questions you didn’t ask yet.
Computational Analysis
InteractiveVisualization
Computational Analysis
InteractiveVisualization
Computational Analysis
Analytical Reasoning
Statistical Modelling
Visual Analytics
HOW?
Yay!Let’s read them all.
Previously in newsroom…
1. Score comments with NLP criteria
Proposed process
2. Interactive selection with Custom Ranked List and Overview visualization
Proposed process
3. Machine learns from User feedback
Proposed process
1
Scoring criterias
The Times’ Community API2815 articles
331782comments
12542Pick
Criteria Only
The Times’ Community API2815 articles
331782comments
12542Pick
Train TestPick comments Nonpick comments P N
Criteria Only
Results
Precision: 0.65Recall: 0.68F2 score: 0.68
1712 (True Positive) 926 (False Positive)
797 (False Negative) 1583 (True Negative)
Pick No Pick
Ground Truth
Prediction
Pick
No Pick
Results
Precision: 0.65Recall: 0.68F2 score: 0.68
1712 (True Positive) 926 (False Positive)
797 (False Negative) 1583 (True Negative)
Pick No Pick
Ground Truth
Prediction
Pick
No Pick
User history and Readability are best predictor for quality.
2
Custom Ranked List
Overview Visualization
http://moderator.comment-iq.com/#/demo
CommentIQ
3
PRECONDITIONpersonal validation
COREinward-facing validation
ANALYSISoutward-facing validation
learn implementwinnow cast discover design deploy reflect write
Learn Design Evaluate
ID OrganizationField-experience
(in years)Workflow
P1
Washington Post
10
Post-moderationP2 1
P3 4
P4New York Times
4Pre-moderation
P5 7
P6 Wall Street Journal 4 Post-moderation
P7 Baltimore Sun 7 Post-moderation
ID OrganizationField-experience
(in years)Workflow
P1
Washington Post
10
Post-moderationP2 1
P3 4
P4New York Times
4Pre-moderation
P5 7
P6 Wall Street Journal 4 Post-moderation
P7 Baltimore Sun 7 Post-moderation
ID OrganizationField-experience
(in years)Workflow
P1
Washington Post
10
Post-moderationP2 1
P3 4
P4New York Times
4Pre-moderation
P5 7
P6 Wall Street Journal 4 Post-moderation
P7 Baltimore Sun 7 Post-moderation
1 hour structured interview with test drive
Findings
“…shifting moderating to a reporting
research job.”
Paraphrased from one moderator
•Flexibility is great
•Comments are made by people!
•Lots of other use-cases
•Flexibility is great
•Comments are made by people!
•Lots of other use-cases
•Flexibility is great
•Comments are made by people!
•Lots of other use-cases
Discussion
Score by Algorithm
Score by Algorithm
Balanced humanjudgement
Beyond “Most Liked”
For general public
• Beyond
For general public
• Beyond
For general public
• Beyond
Conclusion
WHAT ARE GOOD COMMENTS?
HOW CAN WE SELECT GOOD COMMENTS?
Visual analytics approach helps moderators find high-quality comments.
What is a high-quality comment depends on the journalistic context.
Visual analytics approach can help moderators find high-quality comments.
What is a high-quality comment depends on the journalistic context.
Time to upgrade our comments section
TEAM WORK
Niklas Elmqvist@NElmqvistNicholas Diakopoulos
@ndiakopoulos
Simranjit [email protected]
Deokgun Park@intuinno
Niklas Elmqvist@NElmqvistNicholas Diakopoulos
@ndiakopoulos
Simranjit [email protected]
Questions?Try atComment-iq.com
Deokgun Park@intuinno