interacting with ai module 3 - universitetet i oslo · interaction scenarios, randomly assigned...
TRANSCRIPT
![Page 1: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/1.jpg)
INTERACTING WITH AIMODULE 3Session 3, November 10, 2020
Amela Karahasanović, SINTEF and UiO
![Page 2: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/2.jpg)
Module 3Living and working with AI
Objectives
Understanding of challenges related to use of AI infused systems in everyday life and at work
How to evaluate them?
When and how to use them?
What do we know about living and working with them?
2
![Page 3: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/3.jpg)
Module 3Overview
Evaluation of interaction with AI [27th of October]
Human - AI partnership [3rd of November]
Lessons learned from studies of human – AI interaction [10th of November]
3
![Page 4: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/4.jpg)
EXAMPLES
![Page 5: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/5.jpg)
Human – human interaction -> Human - AI interactionHow to create patterns (building blocks)
for human interactions that can be used for human-robot interactions?
Interaction design patterns Interaction blocks - a visual prototyping
environment Evaluating usability of the environment
Sauppe, A., Mutlu, B., Design Patterns for Exploring and Prototyping Human-Robot Interactions, In Proc. of CHI 2014, 1439-1448, ACM
![Page 6: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/6.jpg)
6
![Page 7: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/7.jpg)
Human – human interaction
A lot of studies in linguistic and psychology on behaviour, but design patterns that can be translated in human-robot interactions were missing
16 participants from UoWM (19-62) assigned to 5 different interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation – discuss their educational experiences and
goals Collaboration – sorting the grocery bags Instructions – assembling some pipeline constructions Interview – a job interview, 14 questions Story telling – retelling a video, 7 minutes video
![Page 8: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/8.jpg)
Human – human interaction
Analysis – coding (one researcher, 10% also by the other researcher to confirm reliability)
Codes for states – important events in interaction (question, answer) and the transition between the states to discover patterns
Identified 7 patterns
![Page 9: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/9.jpg)
Design patterns for exploring and prototyping human-robot interaction
9
• Introductory monologue "Hi Welcome…"
• Questions and answer pair Storyteller: "Do you know who Marvin the Martian is?", Listener: "Oh yeah,…"
• Generic comment and personal comment "Interesting" "I had a similar experience once…"
• Monologue and generic commentStoryteller: "…and then, all of the sudden, thousands of these aliens appear at the Earth." Listener:"That's a lot"
![Page 10: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/10.jpg)
Design patterns for exploring and prototyping human-robot interaction
10
• Instructions and actionInstructor: "Now connect the long pipe with the one shaped like an S" Student: <locates the pipe and connect it>
• Finished commentParticipant: "I think that's it, so we should be done"
• Wait (while the other participant is talking)
![Page 11: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/11.jpg)
11
![Page 12: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/12.jpg)
Implemented and evaluated an authoring environment based on these patterns with interaction designers and developers
![Page 13: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/13.jpg)
13
![Page 14: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/14.jpg)
Your turn Where studies of human-human interaction systems can
be useful? For which kind of systems? Which methods can be used for data collection?
1-2 minutes note for yourself
14
![Page 15: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/15.jpg)
EXAMPLES
![Page 16: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/16.jpg)
Evaluating chatbots
Evaluating and Informing the Design of Chatbots, Jain M. et al. 2018, DIS 2018
Evaluation of 8 chatbots with 16 first-time chatbot users over multiple sessions on the Facebook Messenger platform
Three days interactions with chatbots
Face-to-face semi-structured interview with the participants to elicit their understanding of the chatbots, perceived benefits/limitations, interesting conversations/experiences and areas of improvements
Quantitative data analysis: total interaction time, message count, interactive elements (composition of the chatbots' and participants' messages)
16
![Page 17: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/17.jpg)
Chatbot evaluation
17
Chatbots: a trave agent, entertainment, a shopper, news, weather forecast, chit-chat, social chat, a quiz game
Quantitative data• similar amount of time with each chat-bot, the total
number of messages exchanged significantly different (chit-chat > shopper)
• game and chit-chat bots had more chatbot messages
![Page 18: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/18.jpg)
Qualitative data
Three researchers developed together iteratively a coding schema
4 high-level themes:functionalityconversational intelligence
personality interface
18
![Page 19: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/19.jpg)
Design implications
Chatbot designers
Clarify capabilities at the start and on-demand
Evaluate application-interface match
adding links to external webpages is not recommended; clear added value compare to search engine
Enable dialog efficiency through context resolution
Remain context from earlier conversations, intelligent questions
Consistent personality with small-talk and humor
Design for dialog failure – admitting failure
19
![Page 20: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/20.jpg)
Design implications
Chatbot platform UX designer
Combine text-based interface with buttons and media
participants did not like when the chatbot, for example, opened a news article in a new browser window
Enable efficient input from users
Auto-suggestion buttons
Provide persistent view on chatbot capabilities and context
In the beginning, but also later, keeping the chatbot and the user in the same state of mind
20
![Page 21: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/21.jpg)
Your turn What have you found in you chatbot evaluations? What
would you recommend to chatbot designers?
1-2 minutes note for yourself
21
![Page 22: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/22.jpg)
EXAMPLES
![Page 23: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/23.jpg)
ATM exampleEvaluation of DAC with ATCOs
Karahasanovic et al., 2019, Supporting Air Traffic Controllers During Sector Configuration Changes in Dynamic Airspace Configuration, SID 2019
23
![Page 24: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/24.jpg)
Data collection
Observations
Log files (simulator, UI)
Screen captures
Video recording of screens
Interviews (audio records)
Audio records of the communication between the ATCOs and pilots
Questionnaires
24
![Page 25: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/25.jpg)
Results overview
The overall workload - acceptable and the same or slightly reduced for DAC scenario
Few errors - mostly due to the simplifications of the protype's UI
Overall feedback positive
"Good", "almost perfect", "it enabled us to easily maintain the picture of the traffic"
Solutions for notifying and visualizing both horizontal and vertical changes were useful
Some improvements proposed
25
![Page 26: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/26.jpg)
Results overview cont.
Situation awareness was very good DAC concept potentially useful for supervisors There is a need for a better alignment of the
change with the ATCOs workload
26
MED MIN MAX STDEV
All sessions 8 5 10 1,43
Ref. scenario 8 5 10 1,31
Sol. Scenario 8 5 9 1,56
![Page 27: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/27.jpg)
EXAMPLES
![Page 28: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/28.jpg)
RoboCare
• A system that assists elderly at home• Evaluation
• Video based trials, 8 short movies with different domestic scenarios
• User initiative and system initiative• 100 participants (59 – 90 years)• Questionnaire (background, evaluation)
Cesta et al. Monitoring elderly people with the robocare domestic environment: interaction synthesis and user evaluations, Computational Intelligence, Vol 27, 2011
![Page 29: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/29.jpg)
Findings
High usefulness and acceptability
Useful for personal and environmental safety, reminding to take medications and finding objects
Not useful for providing suggestions
Appreciated help with cognitive problems (reminding them to do something, finding things)
![Page 30: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/30.jpg)
Your turn What have you read/seen (news, articles, videos) about
validations of robots in care of elderly? You can check it quickly now.
4-5 minutes note for yourself
30
![Page 31: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/31.jpg)
EXAMPLES
![Page 32: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/32.jpg)
Can robots manifest a personalityAIBO (robot dog) 48 participants based on the personality scores Randomly assigned to the introvert or extrovert robot(4 conditions) Verbal cues Asked to interact with AIBO (verbally and tactile commands) AIBO respond (extrovert turn faster and wider) Questionnaire on personality, intelligence and attractiveness of AIBO Participant enjoyed the interaction more with a compatible robotLee, K. M., Peng, W., Jin, S., & Yan, C. (2006). Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human-robot interaction. Journal of Communication, 56(4), 754-772.
![Page 33: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/33.jpg)
EXAMPLES
![Page 34: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/34.jpg)
Trust in automation An experiment to investigate how the experience of automated
driving change trust in automation and attitudes towards automation 72 participants different ages 19 – 79 (a half under 30) Questionnaire before and after a driving experience in the simulator Gaze recording Highly automated driving on a three line highway, 120 km/h Three take over scenarios, warned 7 seconds in advance, different
traffic densities Older participants more positive Driving increased trust
Trust in Automation – Before and After the Experience of Take-over Scenarios in a Highly Automated Vehicle, Gold et al., AHFE 2015
![Page 35: INTERACTING WITH AI MODULE 3 - Universitetet i oslo · interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation](https://reader035.vdocument.in/reader035/viewer/2022071511/612fd4621ecc51586943b3f2/html5/thumbnails/35.jpg)
Module 3
35
The End