conversational informatics - 京都大学 · driving experiences data base interactive presentation...
TRANSCRIPT
![Page 1: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/1.jpg)
Conversational Informatics (E), October 3, 2018
1. Introduction—Conversational Informatics—
Toyoaki NishidaKyoto University
Copyright © 2018, Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, At ,Inc. All Rights Reserved.
![Page 2: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/2.jpg)
Why do we study conversation?
AI People
Data
Service
Common ground
Conversation as continuous update of the common ground
![Page 3: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/3.jpg)
Conversations in different settings
Laboratories (controlled) Wild (uncontrolled)
Augmented (unrestricted)
![Page 4: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/4.jpg)
Main Focus: Conversation
Shot by Ms. Sutasinee Thovuttikul with permission at a market in Chiang Mai, Thailand
![Page 5: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/5.jpg)
Eye gaze
Hand gesture PostureFacial expression
AskingNegotiating
Proposing
Conviviality
Social networksTrust
Conversation is a complex business
[A marketplace in Chiang Mai; courtesy of Sutasinee Thovuttikul]
![Page 6: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/6.jpg)
Challenge: A robot that can participate in conversation
Long-term goal
Not just conversational but also empathic
![Page 7: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/7.jpg)
History of conversational systems development
t1990 2000 201019801970
Natural language dialogue systems
Speech dialogue systems
Multi-modal dialogue systems
Embodied Conversational Agents / Intelligent Virtual Human
Story Understanding systems
Conversational Systems
Transactional systems
Interactional systems
Affective Computing
Cognitive systems
Natural language question answering systems
The Knowledge Navigator
![Page 8: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/8.jpg)
Our research group
SPOC (2005)
SituatedKeyword-based
EgoChat (2000)
CoMeMo Community (1998) POC TV(2001)
Physical robots
Vicky (2008)
Sustainable Knowledge Globe (2006)
Has addressed numerous aspects
![Page 9: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/9.jpg)
[Nishida 1998][Nishida 1998]
CoMeMo Community
![Page 10: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/10.jpg)
[Kubota 2000]
Using alterego agents
![Page 11: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/11.jpg)
[Kubota 2000]
EgoChat
![Page 12: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/12.jpg)
Knowledge cardKnowledge card
Knowledge cardKnowledge card
Knowledge card server
Knowledge card
presentation by ECA
Knowledge card editor
Knowledge channel control policy
landscapes
[Nishida DNIS 2003]
Knowledge Card Circulation
![Page 13: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/13.jpg)
EgoChat -- demo
[Kubota+ 2003]
![Page 14: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/14.jpg)
[Nishida 2002]
POC: Public Opinion Channel [Nishida 1999]
POC TV (KDDI FTTH Trial)
![Page 15: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/15.jpg)
Capturing conversation quanta using auxiliary devices
P
A
B
button press
utterance 1 utterance 2reaction to the present
reaction to the past
proaction to the future
the timing of button press
and conversation quantum
Analysis
[Kubota and Saito 2005]
![Page 16: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/16.jpg)
In-vehicle conversation circulation system
[Okamura+ 2007]
conversation assistance and capture
by recognizing nonverbal behaviors (e.g., pointing)
driving simulatorautomobile with video recorder capture and reuse conversation in automobile
- personal experience and social interaction -
detect pointing, display names and previous conversations
![Page 17: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/17.jpg)
Capture
Driving experiences data base
Interactive presentation
Driving experience recording system
Scenes+GPS info
[Nishida 2008]
User behavior
Intelligent Driving Simulator
The purpose: to help people share and evolve the driving experiences
data base by allowing them to conversationally annotate driving scenes.
Vickey: In-vehicle conversation circulation system
![Page 18: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/18.jpg)
The Sustainable Knowledge Globe (SKG)
[Kubota+ 2004]
![Page 19: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/19.jpg)
[Kubota+ 2004]
The Sustainable Knowledge Globe (SKG)
![Page 20: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/20.jpg)
[Nakano 2005]
SPOC
![Page 21: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/21.jpg)
[Terada+ 2002]
Autonomous mobile chair
![Page 22: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/22.jpg)
Conversational artifacts shows how synthetic
characters or intelligent robots use eye gaze, gestures
and other non-verbal communicators to interact.
Conversational contents looks at developing
techniques for acquiring, editing, distributing and utilising
the contents that are produced and consumed in
conversation.
Conversation environment design explains
techniques for creating intelligent virtual environments
and for representing individuals within a virtual
environment
Conversation measurement, analysis and modelling
demonstrate how conversational
behaviour can be measured and
analyzed.
Buy your copy here
or online at
www.Wiley.com
An Engineering Approach
http://as.wiley.com/WileyCDA/WileyTitle/productCd-0470026995,descCd-tableOfContents.html
![Page 23: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/23.jpg)
Our research group addressed
SPOC (2005)
SituatedKeyword-based
EgoChat (2000)
CoMeMo Community (1998) POC TV(2001)
Gesture-based
Vicky (2008)
Sustainable Knowledge Globe (2006)
Has addressed numerous aspects
Not realistic!
But limited
![Page 24: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/24.jpg)
Approach 1: Making environment playful
Smart conversation space that encompasses
participants and referents of conversation.
-> Engaged conversational interactions
-> More insights about the common ground
Augmentation
by MR (VR—AR)
Daily living space
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
![Page 25: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/25.jpg)
[Nitschke 2013]https://www.youtube.com/watch?v=V-9SKpcMrzk
Conversations in an artificially created environment
ICIE: Immersive Collaborative Interaction Environment
![Page 26: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/26.jpg)
Appearance Architecture Projection
Inside view
https://www.youtube.com/watch?v=wxkZ9armrI8
Interactive Dome
![Page 27: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/27.jpg)
DEAL: Platform for ICIE
[Ohmoto 2011]
![Page 28: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/28.jpg)
Learn from Playful interactions
Shared interaction environment cohabited by people and agents
[Suyama, Ohmoto 2015]
Player #1 in ICIE #1
Red hat
Player #2 in ICIE #2
Green hat
Player #3 (agent)
Blue hat
![Page 29: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/29.jpg)
Analyze the behaviors of
participants by integrating audio-
visual and
physiological .measurement
Approach 2: Understanding by measurement and analysis
![Page 30: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/30.jpg)
Approach 2: Understanding by measurement and analysis
Measurement in an open spaceMeasuring conversation in a fully equipped space
Conversation in a specific spatial arrangement Measurement in an augmented space
![Page 31: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/31.jpg)
Corneal Imaging CameraScene cameraEye camera
• Lightweight and versatile system
• Appl.: Google Glass like HMD, unconstrained setups
Corneal Image Feature Matching
Problem:
Local feature correspondence + RANSAC does
not work due to large noise in eye images
Approach:
1. Formulate problem as registration of 3D spherical light maps of eye and scene image
2. Single point algorithm for robust alignment
Non-intrusive Eye Gaze Tracking (EGT) by corneal imaging
![Page 32: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/32.jpg)
Intentions are dynamic and tacit
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Facilitative agent that can track dynamic and tacit intentions
Degrees of concentration
Level of proficiency
Tacit social signals and mental process beneath surface
![Page 33: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/33.jpg)
Approach 3: Learning by imitation
Endow robots with an ability
of autonomously imitating
human behaviors.
![Page 34: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/34.jpg)
Approach 3: Learning by imitation—Generic framework
Measurement Corpus Generalization Dialogue
patterns
[Nishida et al 2014]
Endow robots with an ability
of autonomously imitating
human behaviors.
![Page 35: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/35.jpg)
Interactions from observation—General framework
Causes
Causes
Causes
1a
2a
3a
t
t
t
[Nishida et al 2014]
![Page 36: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/36.jpg)
Problem formulation:Find approximately repeated subsequences in a longer time series.
(1) Motif Discovery—Finding Patterns of Interaction
[Nishida et al 2014]
![Page 37: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/37.jpg)
Future
Change angle
GH
Past Futuret
( ) ( ) ( );...; 1H t seq t n seq t= − −
( ) ( ) ( )1 ;...;G t seq t seq t n= + +
( ) ( ) ( )
( ) 1
1
1
ˆ
f
f
T
i i i
l
i i
i
l
i
i
s t t t
cs
x
c
t
=
=
= −
=
TtVtStUtH )()()()( =
Find optimal lPggT uutGtG =)()(
Find optimal lF
11and,)( +−= jjjF
g
ii liut
fT
ll
T
lli li
tUU
tUUt = ,
)(
)()(
)()()()()(ˆ)(~ tttttxtx PFPF −−=
Learning by imitation
Robust Singular Spectrum Transform
[Mohammad 2009]
![Page 38: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/38.jpg)
[Mohammad 2009]
Learning by demonstration
[Mohammad 2016]
![Page 39: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/39.jpg)
Application
Platform Evaluation
Content production
Model building
Analysis
Theory
Measurement
Conversational interactions
Conversational Informatics
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Building conversational
systems
Understanding conversation
![Page 40: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/40.jpg)
Limitations
Our conversational agents do not produce
a strong sense of presence.
We lack the common ground!
![Page 41: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/41.jpg)
Common ground
Physical configuration
Imaginary scene shared by participants
Communal background including the cultural and biological aspects
Common ground
![Page 42: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/42.jpg)
Bargaining scenario (successful)
C: Oh, OK. How much is this one?
S: This one? It worth nothing, you can have it [for free].
C: Haha, well, please...
S: The price is 500 tomans.
C: Oh! 500 tomans? It is very expensive? Do you have anything cheaper?
S: This is from a very good brand, we have the no-name ones, you can see
up there, and they are cheaper.
C: But the colors of those do not match my table. I would love to buy this if
you give me a very good discount. You know, I am about to marry,
marriage itself costs a lot, so it would be nice if you would do that.
S: It is your marriage, you should treat us! … Because it is a happy event I
can give it for 450.
C: 450 is still a little bit expensive. Why not we make a deal for 400?
S: 400 is too low, but since it is my first sell of the day, and you are getting
married, and everything looks very great I can give it for 430, but you
know, I haven’t sold it less than 450 so far.
C: You have been so nice so far, why not getting rid of that 30 and make it
round for 400. I am sure you are making lots of purchases today with
other customers and you can compensate for that. And I’m going to be a
frequent customer from now on, I will come to your shop again.
S: We would love to have you again, but the price is somehow at its
margin. But, I can guarantee it for one year for you, and I can give it to
you for 420, final! This is very good brand, you would never regret it, you
would love it, you can have my word that it works for you perfectly and
for one year if anything happens to that you can bring it to us and we can
fix it for you for free. Ok, I’ll write 420 for you.
C: Ok, thank you, you did me a good favor. Thanks a lot.
S: Oh please. We enjoy you being here and please come back soon.
C: Yeah I will. I’m actually going to do more purchases from now on from
your shop. I had a good experience.
S: Sure! Have a good time.
C: Hi. How are you?
S: Hi. Welcome to this shop.
C: Thank you. How much are these handicrafts here?
S: Oh, these are called Khaatam .The big ones are 120, [middle
sized ones for] 80 tomans and [small ones are] 50 tomans.
C: Um, this is Termeh, right?
S: Yes!
C: What about the Termeh?
S: Actually we have different types of Termeh. We have the red
ones there starts from 50 and the brown ones start from 150
and these special ones starts from 300.
C: You know what? I am about to marry, and I am looking for
something very nice to decorate my table. Its color is brown.
So, what do you suggest?
S: Oh! Congratulations for your marriage!
C: Thank you.
S: We usually go for the red ones for brown tables, but because
you are a great customer and you have good taste, probably
you would go for this one, this is very beautiful and it is from a
very famous designer.
Ostensible offer
[Mirzaei+ 2017]
![Page 43: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/43.jpg)
How much
is this one?
I expect him to
understand that
I want to know
the price.
Immediate quote
is not good.
Successful Scenario: Extract Analysis
I am asking him to
give me the
information on the
price so that I can
decide whether to
buy it or not.
She is asking
how much I
expect her to
pay for this item.
She does not expect
me to quote the price
immediately. It is
against convention.
[Mirzaei+ 2017]
![Page 44: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/44.jpg)
Have it
for free
Look at his body
language.
He shows his
sincerity.
Successful Scenario: Extract Analysis
Two arms extended forward with palms up
He is being kind
and polite but his
offer is not real.
He expects me to
decline the offer to
balance the equity.
She will be happy
by my ostensible
offer and she will
ostensibly decline.
I show her my
sincerity. I’m making
a non-real offer.
[Mirzaei+ 2017]
![Page 45: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/45.jpg)
Haha
Successful Scenario: Extract Analysis
She enjoyed it
and she knows
the offer is not
real.
I expect him to
be relieved by
receiving my
declination.
I understand you are
being very nice and I
acknowledge your
politeness.
She acknowledged
my offer. She is nice
and the atmosphere
of the conversation
is good.
She expects me to
continue with the
convention (i.e.,
going with 2nd offer)
[Mirzaei+ 2017]
![Page 46: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/46.jpg)
Synthetic evidential study
Synthetic evidential study (SES) combines dramatic role play and group discussion to help people spin stories by bringing together partial thoughts and evidence.
Componentize
Reuse
SES session Interpretation archive
Structured collection of {story, background, critique}Agent Play
Dramatic role play
Group discussions
[Nishida et al 2015]
![Page 47: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/47.jpg)
At the beginning of the 18th century, a feudal lord named Asano Takumi-no-kami
Naganori was in charge of a reception for envoys from the Imperial Court in Kyoto.
Another feudal lord, Kira Kozuke-no-suke Yoshinaka, was appointed to instruct
Asano in the ceremonies. On the day of the reception, while Kira was talking with
Yoriteru Kajikawa, a lesser official, at “Matsu no Roka” (“Hallway of Pine Trees”) in
Edo Castle, Asano came up to them screaming “This is for revenge!!” and slashed
Kira twice with a short sword. Soon after the incident, Kajikawa restrained Asano,
who was then imprisoned. The reason for the attack was not known, though it was
widely believed that Kira had somehow humiliated Asano. Ultimately Asano was
sentenced to commit seppuku, a ritual suicide, but Kira went without punishment.
Hallway of Pine Trees (from Chushingura)
Kira Kozuke-no-suke Yoshinaka
Asano Takumi-no-kami Naganori
Yoriteru Kajikawa
Why was it possible?
How did it happen?
What did each think?
![Page 48: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/48.jpg)
Dramatic Role Play
![Page 49: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/49.jpg)
Group play capture
![Page 50: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/50.jpg)
Agent play
![Page 51: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/51.jpg)
Discussion phase
T. Ookaki, M. Abe, M. Yoshino, Y. Ohmoto and T. Nishida. Synthetic Evidential Study for Deepening
Inside Their Heart. IEA/AIE 2015.
Asano
Kira
Kajikawa
Third person view First person view
Discussions
![Page 52: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/52.jpg)
1. Common ground for society and AI
2. Conversation as a contiguous update of common ground.
3. Conversation envisioning as visualization of conversation and common ground
4. Conversation envisioner as a platform for conversation envisioning
Conversation as
contiguous update
of common ground
Analysis
Training
Assistance
Conversation envisioner
Vision — Conversation Envisioner
![Page 53: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/53.jpg)
TSEiA: The Story Envisioning Agent
[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]
![Page 54: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/54.jpg)
TSEiA: The Story Envisioning Agent
[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]
![Page 55: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/55.jpg)
FACSvatar: An Asynchronous Open Source Modular Framework From Face to FACS Based Avatar Animation
Flow of data in framework
Facial expression data can be used with any character[van der Struijk+ 2018] Stef van der Struijk, Hung-Hsuan Huang, Maryam Sadat Mirzaei, Toyoaki Nishida. FACSvatar: An Open Source Modular Framework for Real-Time FACS based Facial Animation, IVA 2018 (accepted)
![Page 56: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/56.jpg)
Role-play Envision
L2 Learners New L2 Learners Teacher
AI Actor AI Assistance
Conversation Envisioning for CALL Purposes
CALL: Computer-Assisted Language Learning
[Mirzaei+ EutoCALL 2018]
![Page 57: Conversational Informatics - 京都大学 · Driving experiences data base Interactive presentation Driving experience recording system Scenes+GPS info [Nishida 2008] User behavior](https://reader034.vdocument.in/reader034/viewer/2022042805/5f611969c1af0166d663d406/html5/thumbnails/57.jpg)
Agenda
CreditsWill be awarded based on a report on subjects given at the class. Due date (January 31st, 2019)
Agenda (planned)
1. October 3: Introduction by Nishida2. October 10: History of Conversational Systems by Nishida 3. October 17: Methodologies for Conversational System Development (1) by Nishida4. October 24: Methodologies for Conversational System Development (2) by Nishida5. October 31: Methodologies for Conversational System Development (3) by Nishida
(November 7: no class)6. November 14: Cognitive Interaction Design by Ohmoto7. November 21: Smart Conversation Space by Ohmoto
(November 28: no class)8. December 5: Measurement, Analysis and Modeling by Ohmoto9. December 12: Affective Computing, Theory of Mind by Nishida10. December 19: Learning by Imitation by Nishida 11. December 26: Aspects of Conversation – 1 by Nishida12. January 9: Aspects of Conversation – 2 by Nishida13. January 16: Function of Nonverbal Behaviors in Conversation by Nishida14. January 23: Synergy by Nishida
Course materials available from: http://www.ii.ist.i.kyoto-u.ac.jp/?page_id=5881