summarization and visualization of digital conversations
DESCRIPTION
Paper presented at SPIM worksop at LREC2010, Malta.TRANSCRIPT
Summarization and Visualization of Digital
Conversations
Vincenzo Pallotta�Joint work with�
Rodolfo Delmonte, University of Venice, Italy�Marita Ailomaa, EPFL, Switzerland�
Digital Conversations
• The Web �– Social Media�– Forums�– Blogs �
• Meetings�• VoIP�• Call centers�• Help Desk �
2 SPIM 2010 - Malta
Captured Meetings
3 SPIM 2010 - Malta
Virtual Collaboration
4 SPIM 2010 - Malta
5 SPIM 2010 - Malta
6 SPIM 2010 - Malta
7 SPIM 2010 - Malta
1st Hypothesis…
8 SPIM 2010 - Malta
V. Pallotta, Content-based retrieval of distributed multimedia conversational data. In E. Vargiu, A. Soro, G. Armano, G. Paddeu (eds.) Information Retrieval and Mining in Distributed Environments, Springer Verlag, series: Studies in Computational Intelligence (ISSN: 1860-949X) to Appear, 2010.
Challenges for (spoken) conversation processing
• dealing with multiple speakers�• dealing with foreign language and associated
accents�• incorporating non-speech audio dialogue acts �
– (e.g., clapping, laughter, silence?)�• conversational segmentation and summarization �• discourse analysis, such as: �
– analyzing speaking rates�– turn taking (frequency, durations)�– concurrence/disagreement �
• which often provides insights into speaker emotional state, �– attitudes toward topics and other speakers�– roles/relationships.�
SPIM 2010 - Malta
M. Maybury: Keynote at the SIGIR 2007 Workshop Searching Spontaneous Conversational Speech
9
Capturing and Processing Conversations
SPIM 2010 - Malta
• Informal Meetings • Focus Groups • Classes • Interviews • Debates • Podcasts • Comments • Forums
• Executive Summaries • Topic highlights • Issue tracking • Project management • Mediation • Semantic Search
10
2nd Hypothesis…
11 SPIM 2010 - Malta
What type of content is user looking for from conversations?
• Users look for argumentative information �– Decision Making �– Conflict Resolution �
• Information Retrieval is not sufficient �– Need for more context �– Answers not found in
words spoken �
05
10152025303540
Factual Thematic Process Outcome
IM2 setMS set
12 SPIM 2010 - Malta
0102030
4050607080
IR sufficient IR irrelevant IR insufficient
IM2 set:argumentativeMS set:argumentative
Pallotta, Seretan, Ailomaa ACL 2007
3rd Hypothesis…
13 SPIM 2010 - Malta
…in what form?
14 SPIM 2010 - Malta
…more demographic details
15 SPIM 2010 - Malta
…and still more
16 SPIM 2010 - Malta
4th Hypothesis…
17 SPIM 2010 - Malta
Two reviews from ACL…
• "The idea of using argument structure annotation to aid dialogue summarization is very promising. For an abstractive summary of dialogues this seems almost like an inevitable step and I am always glad to see people take on the hard task of abstractive summarization.“�
• "I think the general approach of detecting the argumentative structure is the correct one to take and the authors are laying groundwork for a solid abstractive system."�
18 SPIM 2010 - Malta
Our Approach…
• Topic Segmentation �• Recognition of argumentative episodes: �
– Based on the GETARUNS system�• Automatic recognition of argumentative
structure: �– Novel discourse parsing algorithm�
• Retrieval through: �– Question Answering �– Abstractive summaries �– Visualization of arguments�
SPIM 2010 - Malta 19
Meeting Description Schema DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a
REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a
PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+
ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b
PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a
ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}
PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}
PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+
Why was David’s proposal on microphones rejected?
20 SPIM 2010 - Malta
Abstractive Summary
21 SPIM 2010 - Malta
• David proposal was: “go ahead and get nine identical head mounted crown mikes” • David’s proposal was rejected. • John provided an alternative: “get one of these with crown with a different headset”. John’s proposal was accepted by the majority of participants.
DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a
REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a
PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+
ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b
PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a
ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}
PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}
PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+
• David proposal was: “go ahead and get nine identical head mounted crown mikes” • David’s proposal was rejected. • John provided an alternative: “get one of these with crown with a different headset”. John’s proposal was accepted by the majority of participants.
Argumentative Labeling with GETARUNS
• Primitive Discourse Relations labels: �– statement, narration, adverse, result,
cause, motivation, explanation, question, hypothesis, elaboration, permission, inception, circumstance, obligation, evaluation, agreement, contrast, evidence, hypoth, setting, prohibition. �
• Mapped into Argumentative labels: �– ACCEPT, REJECT/DISAGREE, PROPOSE/
SUGGEST, EXPLAIN/JUSTIFY, REQUEST EXPLANATION/JUSTIFICATION.�
22 SPIM 2010 - Malta
Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).
Evaluation
23 SPIM 2010 - Malta
Correct Incorrect Total Found Precision
Accept 662 16 678 98%
Reject 64 18 82 78%
Propose 321 74 395 81%
Request 180 1 181 99%
Explain 580 312 892 65%
Disfluency 19 0 19 100%
Total 1826 421 2247 81%
Precision: 81.26% Recall: 97.53%
ICSI corpus of meetings (Janin et al., 2003)
Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).
Applications for Visualization and Summarization of Digital Conversations
24 SPIM 2010 - Malta
Conversational Graphs
25 SPIM 2010 - Malta
[7:00] # Yes, uh, I've a question, uh, what's mean exactly advance chip on print? What's the meaning of that? [7:10] 7 5 [7:02] Yeah [7:2] [7:10] I think it's um uh a multiple uh chip design uh and it's maybe printed on to the (curcuit) board. [7:20] 8 7 [7:21] Mm-hmm. [7:21] [7:21] Uh I could find out more about that uh before the next fi- next meeting. [7:26] 8.1 8 [7:24] Yeah, is it means it's on the - x#x is it on the micro-processor based or uh - [7:30] 9 8 [7:32] I don't know, but I'll find out more on our next meeting. [7:35] 10 11 11:09
[7:34] [O]okay, uh, that would be great, so if you find out from the technology backgroud, okay, so that would be good[.] [7:39] 12 10 [7:39] Sounds good. [7:40] [7:41] Why was the plastic eliminated as a possible material? [7:44] 13 3 [7:43] Because um it gets brittle - [7:46] 14 13 3 [7:47] cracks - [7:48] 14 13 3 [7:48] uh-huh [7:49] [7:51] um [7:51] 14 13 3 [7:53] We want - we expect these um these remote controls to be around for several hundred years. [7:59] 14 13 3
[8:00] So $ we could $ (??) - good expression [8:6] [8:02] (I would gi-) [8:2] [8:02] Wow $ Good expression, (well) after us $ [8:12] [8:05] Which - [8:6]
[8:12] Um, speak for yourself, I (??) $ - [8:16] [8:13] Alth- I think - [8:15] [8:14] $ [8:16]
[8:16] I think with the wood though you'd run into the same types of problems (??) I mean it chips, it- if you drop it, ehm, it's - I'm not su- $ [8:27] 15 16
15:14 (15:3?) 16:15
Mapping to Bales IPA categories
26 SPIM 2010 - Malta
Improving Opinion Mining
27 SPIM 2010 - Malta
Attitude scores re-ranking
28 SPIM 2010 - Malta
NESTLÉ� twittrratr� Interanalytics� Δ�
Positive� 13% � 34% � 21% ��
Neutral� 85% � 40% � -45% �
Negative� 3% � 16% � 13% �Not Clear� 0% � 10% � 10% �Total� 100% � 100%�
Reliability Scores� 33% � 80% �
Powered by:
Abstractive Summaries of Digital Conversations
29 SPIM 2010 - Malta
Conversation Memos (1)
GENERAL INFORMATION ON PARTICIPANTS�• The participants to the meeting are 7.�• Participants less actively involved are Ami and Don who
only intervened respectively for 38 and 68 turns.�
LEVEL OF INTERACTIVITY IN THE DISCUSSION �• The speaker that has held the majority of turns is
Adam with a total of 722 turns, followed by Fey with a total of 561.�
• The speaker that has undergone the majority of overlaps is Adam followed by Jane.�
• The speaker that has done the majority of overlaps is Jane followed by Fey.�
• Jane is the participant that has been most competitive.�
30 SPIM 2010 - Malta
Conversation Memos (2)
DISCUSSION TOPICS�• The discussion was centered on the following topics: �" "schemas, action, things and domain.�
• The main topics have been introduced by the most important speaker of the meeting. �
• The participant who introduced the main topics in the meeting is: Adam.�
• The most frequent entities in the whole dialogue partly coincide with the best topics, and are the following: �action, schema, things, 'source-path-goal', person, spg, roles,
bakery, intention, specific, case, categories, information, idea.�
31 SPIM 2010 - Malta
Conversation Memos (3)
ARGUMENTATIVE CONTENT�The following participants: �
"Andreas, Dave, Don, Jane, Morgan �expressed their dissent 52 times. However
Dave, Andreas and Morgan expressed dissent in a consistently smaller percentage.�
The following participants: �"Adam, Andreas, Dave, Don, Jane, Morgan �
asked questions 55 times.�The remaining 1210 turns expressed positive
content by proposing, explaining or raising issues. However Adam, Dave and Andreas suggested and raised new issues in a consistently smaller percentage.�
The following participants: Adam, Andreas, Dave, Don, Jane, Morgan expressed acceptance 213 times.�
�
EPISODE ISSUE No. 7�In this episode we have the following
argumentative exchanges between the following speakers: Don, Morgan.�
Morgan provides the following explanation: �[oh, that-s_, good, .] �then he , overlapped by Don, continues: �[because, we, have, a_lot, of, breath, noises, .] �
Don accepts the previous explanation: �[yep, .] �
then he provides the following explanation: �[test, .] �
Morgan continues: �[in_fact, if, you, listen, to, just, the, channels, of, people,
not, talking, it-s_, like, ..., .] �
then he , overlapped by Don, disagrees with the previous explanation �
[it-s_, very, disgust, ..., .] �
Don, overlapped by Morgan, asks the following question: �
[did, you, see, hannibal, recently, or, something, ?] �
Morgan provides the following positive answer: �[sorry, .] �
then he provides the following explanation: �[exactly, .] �[it-s_, very, disconcerting, .] �[okay, .] �
…�� 32 SPIM 2010 - Malta
Conclusion
• Conversational Search and Condensation is extremely challenging �– Classical approaches simply don’t work �– Sense-making is needed�
• One possible “sense”: �– Argumentative structure�
• Possible outputs: �– Question Answering �– Abstractive Summaries�– Conversation Graphs�
• Future Work: �– Improving performance of the classifier�– Build the linking structure of arguments�– Approach generation �
33 SPIM 2010 - Malta
Summarization and Visualization of Digital
Conversations
Vincenzo Pallotta�Joint work with�
Rodolfo Delmonte & Marita Ailomaa�