summarization and visualization of digital conversations

Summarization and Visualization of Digital

Conversations

Vincenzo Pallotta�Joint work with�

Rodolfo Delmonte, University of Venice, Italy�Marita Ailomaa, EPFL, Switzerland�

Digital Conversations

•  The Web �– Social Media�– Forums�– Blogs �

•  Meetings�•  VoIP�•  Call centers�•  Help Desk �

2 SPIM 2010 - Malta

Captured Meetings

3 SPIM 2010 - Malta

Virtual Collaboration

4 SPIM 2010 - Malta

5 SPIM 2010 - Malta

6 SPIM 2010 - Malta

7 SPIM 2010 - Malta

1st Hypothesis…

8 SPIM 2010 - Malta

V. Pallotta, Content-based retrieval of distributed multimedia conversational data. In E. Vargiu, A. Soro, G. Armano, G. Paddeu (eds.) Information Retrieval and Mining in Distributed Environments, Springer Verlag, series: Studies in Computational Intelligence (ISSN: 1860-949X) to Appear, 2010.

Challenges for (spoken) conversation processing

•  dealing with multiple speakers�•  dealing with foreign language and associated

accents�•  incorporating non-speech audio dialogue acts �

–  (e.g., clapping, laughter, silence?)�•  conversational segmentation and summarization �•  discourse analysis, such as: �

–  analyzing speaking rates�–  turn taking (frequency, durations)�–  concurrence/disagreement �

•  which often provides insights into speaker emotional state, �–  attitudes toward topics and other speakers�–  roles/relationships.�

SPIM 2010 - Malta

M. Maybury: Keynote at the SIGIR 2007 Workshop Searching Spontaneous Conversational Speech

9

Capturing and Processing Conversations

SPIM 2010 - Malta

•  Informal Meetings •  Focus Groups •  Classes •  Interviews •  Debates •  Podcasts •  Comments •  Forums

•  Executive Summaries •  Topic highlights •  Issue tracking •  Project management •  Mediation •  Semantic Search

10

2nd Hypothesis…

11 SPIM 2010 - Malta

What type of content is user looking for from conversations?

•  Users look for argumentative information �–  Decision Making �–  Conflict Resolution �

•  Information Retrieval is not sufficient �–  Need for more context �–  Answers not found in

words spoken �

05

10152025303540

Factual Thematic Process Outcome

IM2 setMS set


0102030

4050607080

IR sufficient IR irrelevant IR insufficient

IM2 set:argumentativeMS set:argumentative

Pallotta, Seretan, Ailomaa ACL 2007

3rd Hypothesis…


…in what form?


…more demographic details


…and still more


4th Hypothesis…


Two reviews from ACL…

•  "The idea of using argument structure annotation to aid dialogue summarization is very promising. For an abstractive summary of dialogues this seems almost like an inevitable step and I am always glad to see people take on the hard task of abstractive summarization.“�

•  "I think the general approach of detecting the argumentative structure is the correct one to take and the authors are laying groundwork for a solid abstractive system."�


Our Approach…

•  Topic Segmentation �•  Recognition of argumentative episodes: �

– Based on the GETARUNS system�•  Automatic recognition of argumentative

structure: �– Novel discourse parsing algorithm�

•  Retrieval through: �– Question Answering �– Abstractive summaries �– Visualization of arguments�

SPIM 2010 - Malta 19

Meeting Description Schema DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a

REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a

PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+

ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b

PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a

ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}

PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}

PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+

Why was David’s proposal on microphones rejected?


Abstractive Summary


• David proposal was: “go ahead and get nine identical head mounted crown mikes” • David’s proposal was rejected. • John provided an alternative: “get one of these with crown with a different headset”. John’s proposal was accepted by the majority of participants.

DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a

REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a

PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+

ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b

PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a

ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}

PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}

PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+

• David proposal was: “go ahead and get nine identical head mounted crown mikes” • David’s proposal was rejected. • John provided an alternative: “get one of these with crown with a different headset”. John’s proposal was accepted by the majority of participants.

Argumentative Labeling with GETARUNS

•  Primitive Discourse Relations labels: �– statement, narration, adverse, result,

cause, motivation, explanation, question, hypothesis, elaboration, permission, inception, circumstance, obligation, evaluation, agreement, contrast, evidence, hypoth, setting, prohibition. �

•  Mapped into Argumentative labels: �– ACCEPT, REJECT/DISAGREE, PROPOSE/

SUGGEST, EXPLAIN/JUSTIFY, REQUEST EXPLANATION/JUSTIFICATION.�


Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).

Evaluation


Correct Incorrect Total Found Precision

Accept 662 16 678 98%

Reject 64 18 82 78%

Propose 321 74 395 81%

Request 180 1 181 99%

Explain 580 312 892 65%

Disfluency 19 0 19 100%

Total 1826 421 2247 81%

Precision: 81.26% Recall: 97.53%

ICSI corpus of meetings (Janin et al., 2003)

Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).

Applications for Visualization and Summarization of Digital Conversations


Conversational Graphs


[7:00] # Yes, uh, I've a question, uh, what's mean exactly advance chip on print? What's the meaning of that? [7:10] 7 5 [7:02] Yeah [7:2] [7:10] I think it's um uh a multiple uh chip design uh and it's maybe printed on to the (curcuit) board. [7:20] 8 7 [7:21] Mm-hmm. [7:21] [7:21] Uh I could find out more about that uh before the next fi- next meeting. [7:26] 8.1 8 [7:24] Yeah, is it means it's on the - x#x is it on the micro-processor based or uh - [7:30] 9 8 [7:32] I don't know, but I'll find out more on our next meeting. [7:35] 10 11 11:09

[7:34] [O]okay, uh, that would be great, so if you find out from the technology backgroud, okay, so that would be good[.] [7:39] 12 10 [7:39] Sounds good. [7:40] [7:41] Why was the plastic eliminated as a possible material? [7:44] 13 3 [7:43] Because um it gets brittle - [7:46] 14 13 3 [7:47] cracks - [7:48] 14 13 3 [7:48] uh-huh [7:49] [7:51] um [7:51] 14 13 3 [7:53] We want - we expect these um these remote controls to be around for several hundred years. [7:59] 14 13 3

[8:00] So $ we could $ (??) - good expression [8:6] [8:02] (I would gi-) [8:2] [8:02] Wow $ Good expression, (well) after us $ [8:12] [8:05] Which - [8:6]

[8:12] Um, speak for yourself, I (??) $ - [8:16] [8:13] Alth- I think - [8:15] [8:14] $ [8:16]

[8:16] I think with the wood though you'd run into the same types of problems (??) I mean it chips, it- if you drop it, ehm, it's - I'm not su- $ [8:27] 15 16

15:14 (15:3?) 16:15

Mapping to Bales IPA categories


Improving Opinion Mining


Attitude scores re-ranking


NESTLÉ� twittrratr� Interanalytics� Δ�

Positive� 13% � 34% � 21% ��

Neutral� 85% � 40% � -45% �

Negative� 3% � 16% � 13% �Not Clear� 0% � 10% � 10% �Total� 100% � 100%�

Reliability Scores� 33% � 80% �

Powered by:

Abstractive Summaries of Digital Conversations


Conversation Memos (1)

GENERAL INFORMATION ON PARTICIPANTS�•  The participants to the meeting are 7.�•  Participants less actively involved are Ami and Don who

only intervened respectively for 38 and 68 turns.�

LEVEL OF INTERACTIVITY IN THE DISCUSSION �•  The speaker that has held the majority of turns is

Adam with a total of 722 turns, followed by Fey with a total of 561.�

•  The speaker that has undergone the majority of overlaps is Adam followed by Jane.�

•  The speaker that has done the majority of overlaps is Jane followed by Fey.�

•  Jane is the participant that has been most competitive.�



DISCUSSION TOPICS�•  The discussion was centered on the following topics: �" "schemas, action, things and domain.�

•  The main topics have been introduced by the most important speaker of the meeting. �

•  The participant who introduced the main topics in the meeting is: Adam.�

•  The most frequent entities in the whole dialogue partly coincide with the best topics, and are the following: �action, schema, things, 'source-path-goal', person, spg, roles,

bakery, intention, specific, case, categories, information, idea.�



ARGUMENTATIVE CONTENT�The following participants: �

"Andreas, Dave, Don, Jane, Morgan �expressed their dissent 52 times. However

Dave, Andreas and Morgan expressed dissent in a consistently smaller percentage.�

The following participants: �"Adam, Andreas, Dave, Don, Jane, Morgan �

asked questions 55 times.�The remaining 1210 turns expressed positive

content by proposing, explaining or raising issues. However Adam, Dave and Andreas suggested and raised new issues in a consistently smaller percentage.�

The following participants: Adam, Andreas, Dave, Don, Jane, Morgan expressed acceptance 213 times.�

�

EPISODE ISSUE No. 7�In this episode we have the following

argumentative exchanges between the following speakers: Don, Morgan.�

Morgan provides the following explanation: �[oh, that-s_, good, .] �then he , overlapped by Don, continues: �[because, we, have, a_lot, of, breath, noises, .] �

Don accepts the previous explanation: �[yep, .] �

then he provides the following explanation: �[test, .] �

Morgan continues: �[in_fact, if, you, listen, to, just, the, channels, of, people,

not, talking, it-s_, like, ..., .] �

then he , overlapped by Don, disagrees with the previous explanation �

[it-s_, very, disgust, ..., .] �

Don, overlapped by Morgan, asks the following question: �

[did, you, see, hannibal, recently, or, something, ?] �

Morgan provides the following positive answer: �[sorry, .] �

then he provides the following explanation: �[exactly, .] �[it-s_, very, disconcerting, .] �[okay, .] �

…�� 32 SPIM 2010 - Malta

Conclusion

•  Conversational Search and Condensation is extremely challenging �–  Classical approaches simply don’t work �–  Sense-making is needed�

•  One possible “sense”: �–  Argumentative structure�

•  Possible outputs: �–  Question Answering �–  Abstractive Summaries�–  Conversation Graphs�

•  Future Work: �–  Improving performance of the classifier�–  Build the linking structure of arguments�–  Approach generation �


Summarization and Visualization of Digital

Conversations

Vincenzo Pallotta�Joint work with�

Rodolfo Delmonte & Marita Ailomaa�

summarization and visualization of digital conversations

Business