Towards robotic assistants in nursing homes:Towards robotic assistants in nursing homes:challenges and resultschallenges and results
Joelle PineauMichael MontemerloMartha Pollack *
Nicholas RoySebastian Thrun
Carnegie Mellon University*University of Michigan
The Nursebot Project Joelle Pineau
Introducing Pearl – A mobile robotic assistant for elderly people and nurses
cameras
sonars
handle bars
mobile base
carrying tray
LCD mouth
touchscreen
microphone& speakers
laser
ROLE:
Moving thingsaroundMoving thingsaround
Management support of ADLsManagement support of ADLs
Providing physical assistance
Providing physical assistance
Remotehealth servicesRemotehealth services
Supportingcommunication
Supportingcommunication
Calling for helpin emergenciesCalling for helpin emergencies
Monitoring Rxadherence & safety
Monitoring Rxadherence & safety
Providing info(TV, weather)Providing info(TV, weather)
Reminding to eat,drink, take meds
Reminding to eat,drink, take meds
Linking caregiverand resources
Linking caregiverand resources
The Nursebot Project Joelle Pineau
The Nursebot project in its early days
The Nursebot Project Joelle Pineau
Architecture
Cognitive supportNavigation Communication
High-level controller
The Nursebot Project Joelle Pineau
• Localization and map building(Burgard et al., 1999)
• People detection and tracking(Montemerlo et al., 2002)
Architecture
Cognitive supportNavigation Communication
High-level controller
The Nursebot Project Joelle Pineau
• Autominder system (Pollack et al., 2002)
Architecture
Cognitive supportNavigation Communication
High-level controller
The Nursebot Project Joelle Pineau
• Speech recognition: Sphinx system(Ravishankar, 1996)
• Speech synthesis: Festival system(Black et al., 1999)
Architecture
Cognitive supportNavigation Communication
High-level controller
The Nursebot Project Joelle Pineau
The role of the top-level controller
• Établir les priorités parmi les objectifs des différents modules
• Négocier entre plusieurs objectifs ayant des coûts/gains variés
• Négocier entre l’acquisition d’information et la rencontre des objectifs
• Passer d’une tâche à l’autre en partageant l’information sensorielle
• Planifier malgré la présence d’incertitude
Cognitive supportNavigation Communication
ACTION SELECTION - based on the trade-off between:
- goals from different modules;
- goals with varying costs / rewards;
- reducing uncertainty versus accomplishing goals.
High-level controller
The Nursebot Project Joelle Pineau
Speech recognition with Sphinx
The Nursebot Project Joelle Pineau
Robot control under uncertainty
Belief StateP(st=weather-today)=0.5
P(st=appointment-today )=0.5
USER
Action={ say-weather,update-appointment,clarify-query}
Speech=“today”
Stateweather-today
The Nursebot Project Joelle Pineau
Robot control using Partially Observable MarkovPartially Observable Markov Decision Processes Decision Processes (POMDPs)
Belief state
USER + ENVIRONMENT + WORLD
Actions
ObservationsCosts / Rewards
State
Problem: Which action allows the robot to maximize its reward?
P(s1)P(s2)
The Nursebot Project Joelle Pineau
Methods to solve POMDPs
Objective: Find a policy, (b), which maximizes reward.
Complexity
Performance
QMDP
MDP
FIB
UMDP
AMDP
O(S2A) O(S2AT) O(S2AO )O(S2AO) O(S2AB) T
POMDP
New methods?
The Nursebot Project Joelle Pineau
New approach: A hierarchy of POMDPs
Idea: Exploit domain knowledge to divide one POMDP into many smaller ones.
Motivation: Complexity of POMDP solving grows exponentially with # of actions.
Assumption: We are given POMDP M = {S,A,,b,T,O,R} and hierarchy H
Act
ExamineHealth Navigate
MoveVerifyPulse
ClarifyGoal
North South East West
VerifyMeds
subtask
abstract action
primitive action
The Nursebot Project Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action setNavigate
Move ClarifyGoal
South East WestNorth
AMove = {N,S,E,W}
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
The Nursebot Project Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
Navigate
Move ClarifyGoal
South East WestNorth
AMove = {N,S,E,W}
SMove = {X,Y}
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
The Nursebot Project Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
Step 3: Choose parameters
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
Navigate
Move ClarifyGoal
South East WestNorth
AMove = {N,S,E,W}
SMove = {X,Y}
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
PARAMETERS
{bh,Th,Oh,Rh}
PARAMETERS
{bh,Th,Oh,Rh}
The Nursebot Project Joelle Pineau
PolCA: Planning with a hierarchy of POMDPs
Step 1: Select the action set
Step 2: Minimize the state set
Step 3: Choose parameters
Step 4: Plan task h
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
STATE FEATURESX-positionY-position
X-goalY-goal
HealthStatus
Navigate
Move ClarifyGoal
South East WestNorth
AMove = {N,S,E,W}
SMove = {X,Y}
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
ACTIONSNorthSouthEastWest
ClarifyGoalVerifyPulseVerifyMeds
PLAN
h
PLAN
h
PARAMETERS
{bh,Th,Oh,Rh}
PARAMETERS
{bh,Th,Oh,Rh}
The Nursebot Project Joelle Pineau
First study in simulation
• 20-questions domain:
Agent: “Is it an animal?”User: “No.”Agent: “Is it a vegetable?”User: “Yes.”Agent: “Is it green?”User: “No.”Agent: “...?”
Agent: “Is it an animal?”User: “No.”Agent: “Is it a vegetable?”User: “Yes.”Agent: “Is it green?”User: “No.”Agent: “...?”
Actions
Objective: Plan a sequence of questions allowing the agent to identify the chosen object.
Small complication…. the user can change objects at any time (Pr = 0.1), without telling the agent.
Observations
The Nursebot Project Joelle Pineau
The hierarchy
Animals
Begin
Vegetable?Vegetables Minerals Mineral? Animal?
Mammal? RABBITHerbivore? TURTLE …?… …
The Nursebot Project Joelle Pineau
Methods to solve POMDPs
Complexity
Performance
POMDP
QMDP
MDP
FIB
UMDP
AMDP
PolCA
The Nursebot Project Joelle Pineau
Results
Domain: |S|=12, |A|=20, |O|=3
-120
-100
-80
-60
-40
-20
0
0.01 0.1 1 10 100 1000 10000 100000 1000000
Time (secs)
R
POMDP
PolCA-D1
FIB
QMDP
The Nursebot Project Joelle Pineau
A new decomposition
Animals
Begin
Vegetable?Vegetables Minerals Mineral? Animal?
… … …
The Nursebot Project Joelle Pineau
A new decomposition
Fruits Plants
……
Animals
Begin
Vegetable?Vegetables Minerals Mineral? Animal?
…
Fruit?
…
The Nursebot Project Joelle Pineau
Results
-120
-100
-80
-60
-40
-20
0
0.01 0.1 1 10 100 1000 10000 100000 1000000
Time (secs)
R
POMDPPolCA-D1
PolCA-D2FIB
QMDP
Domain: |S|=12, |A|=20, |O|=3
The Nursebot Project Joelle Pineau
Methods to solve POMDPs
Complexity
Performance
POMDP
QMDP
MDP
FIB
UMDP
AMDP
PolCA
The Nursebot Project Joelle Pineau
PolCA in the Nursebot domain
• Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments.
• Domain: |S|=512, |A|=20, |O|=19 • Hierarchy:
The Nursebot Project Joelle Pineau
Sample scenario
The Nursebot Project Joelle Pineau
Results for dialogue system
0.1 0.1
0.18
POMDP policy MDP policy
The Nursebot Project Joelle Pineau
Summary
• We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people.
• The top-level controller uses a hierarchical variant of POMDPs to select actions.
• This allows it to acquire necessary information and successfully complete assigned tasks.
• Probabilistic techniques have been found to be very useful to flexibly model and track individuals.
The Nursebot Project Joelle Pineau
For more details: www.cs.cmu.edu/~nursebot
The Nursebot team
CMU - Robotics:Greg Armstrong
Michael MontemerloJoelle PineauNicholas RoyJamie Schulte
Sebastian Thrun
CMU - HCI/Design:Francine Gemperle
Jennifer GoetzSarah KieslerAaron Powers
U. of Pittsburgh - Nursing:Jacqueline Dunbar-Jacobs
Sandra EngbergJudith Matthews
U. of Pittsburgh - CS:Don Chiarulli
Colleen McCarthy
U. of Freiburg - CS:Maren BennewitzWolfram Burgard
Dirk Schulz
U. of Michigan - CS:Laura BrownDirk ColbryCheryl OroszBart PeintnerMartha PollackSailesh Ramakrishnan
Standard Robotics:Greg Baltus
The Nursebot Project Joelle Pineau
Our vision of robotic healthcare
Moving thingsaround
Moving thingsaround
Enabling use of remote
health services
Enabling use of remote
health services
Supportinginter-personal
communication
Supportinginter-personal
communication
Calling for helpin emergencies
Calling for helpin emergencies
Monitoring Rx adherence
& safety
Monitoring Rx adherence
& safety
Providinginformation
(TV, weather)
Providinginformation
(TV, weather)
Management support of
ADLs
Management support of
ADLsReminding
to eat, drink, & take meds
Reminding to eat, drink, & take meds
Providing physical
assistance
Providing physical
assistance
Linking the caregiver to resources
Linking the caregiver to resources
The Nursebot Project Joelle Pineau
Profile of an aging population
450,000 nurses to recruit before 2008 (USA)
The Nursebot Project Joelle Pineau
Localization and map building
The Nursebot Project Joelle Pineau
People tracking
The Nursebot Project Joelle Pineau
Analysis of movements
The Nursebot Project Joelle Pineau
Autominder System
The Nursebot Project Joelle Pineau
The family of Markov models
Markov Chain Hidden Markov Model(HMM)
Markov DecisionProcess (MDP)
Partially ObservableMarkov Decision
Process (POMDP)
State ambiguity?
no yes
Choice ofaction?
yes
no
The Nursebot Project Joelle Pineau
The POMDP is a septuple { S, A, , b, T, O, R }
Problem: Which action allows the robot to maximize its reward?
bt-1 bt
at-1
ot
st-1 st
...
...
??ot-1
...
...
rt-1 rt
The POMDP model
Belief:
State:
Action:
The Nursebot Project Joelle Pineau
Execution with a hierarchy of POMDPs
At each time step, traverse the hierarchy from top to bottom.
For each subtask, consult the policy at h(bt)
If at is an internal node > move to that subtask.
Act
ExamineHealth Navigate
MoveVerifyPulse
ClarifyGoal
North South East West
VerifyMeds
The Nursebot Project Joelle Pineau
How to choose parameters?
• Consider for example TNavigate(s,a,s’)
• Case #1: a is a primitive action.
TNavigate(s,ClarifyGoal,s’) T(s,ClarifyGoal,s’)
Navigate
Move ClarifyGoal
South East WestNorth
S
S’South
S’
North
S’
S’
ClarifyGoal
0.7
0.3
The Nursebot Project Joelle Pineau
How to choose parameters?
• Consider for example TNavigate(s,a,s’)
• Case #1: a is a primitive action.
TNavigate(s,ClarifyGoal,s’) T(s,ClarifyGoal,s’)
• Case #2: a is an abstract action.
TNavigate(s,Move,s’) T(s, Move(s) ,s’)
Navigate
Move ClarifyGoal
South East WestNorth
S
S’South
S’
North
0.9
0.8
0.1
0.2
S’
S’
ClarifyGoal
The Nursebot Project Joelle Pineau
States: 1 per object (e.g. tomato, cucumber, rabbit, turtle, tulip, ….)
Observations: “yes”, “no”, “”
Actions: 1 guess per objet Rewards: question = -1+ many questions correct guess = +5
incorrect guess = -20
The details
Animals
Begin
Vegetable?Vegetables Minerals Mineral? Animal?
Mammal? RABBITHerbivore? TURTLE …?… …