assistance for the elderly in robocare

Planning & Scheduling TeamIstc-Cnr

Assistance for the Elderly in RoboCare

Riccardo Rasconi

ISTC-CNR [PST]

Institute for Cognitive Science and Technology

National Research Council of Italy

Planning and Scheduling Team

http://pst.istc.cnr.it

Project “Project “RoboCareRoboCare: A multi-agent system with fixed and robotic : A multi-agent system with fixed and robotic intelligent components” MIUR Law 449/97(yr 00) – 2003-2006intelligent components” MIUR Law 449/97(yr 00) – 2003-2006

Joint work withAmedeo Cesta,Gabriella Cortellessa, Federico Pecora

Workshop on Telematics and Robotics

for the Quality of Life of the Elderly

28/09/2009

Planning & Scheduling TeamIstc-Cnr

• A distributed system– Software Agents– Robotic Agents– Human Agents

• All cooperate to provide services for human assistance

http://robocare.istc.cnr.it

The RoboCare Project’s Goal

Planning & Scheduling TeamIstc-Cnr

The Different Research Aspects Involved

Active Supervision Framework

Human-RobotInteraction

Personalized Intelligent Assistance

Acceptability Issues

Distributed H/S Infrastructure

Robust MobileRobotic Skills

http://robocare.istc.cnr.it

Planning & Scheduling TeamIstc-Cnr

The RoboCare Domestic Environment

People Localization & TrackingPosture Recognition

Robot Mobility+

User Interaction

PDA

ADL Monitoring

Planning & Scheduling TeamIstc-Cnr

Multiple Intelligent Systems

A number of base services provide the building blocks for higher-level assistive behavior

• Posture recognition

• Person localization

• Mobile robotic platform

• User interaction front-end

• PDA interface

• Daily activity monitor

Planning & Scheduling TeamIstc-Cnr

The robotic platform evolution

2004 2005 2006

Planning & Scheduling TeamIstc-Cnr

Vision Sensors: People Tracking System

Luca Iocchi and G. Riccardo Leone work from Univ. Rome “La Sapienza”

[Bahadori et al, Applied AI, 2007]

Planning & Scheduling TeamIstc-Cnr

Caregiver vs. Active SupervisionFramework interface

Contextual Knowledge Component: Non-Intrusive Activity Supervision

Caregiver specifications compiled into scheduling problems

(a temporal constraint network)

compilation

T-REXscheduling problem

physician

family members

behavioral requirements(in terms of daily activities)

[Pecora et al, ISSEJ, 2006]

Planning & Scheduling TeamIstc-Cnr

Representing temporal prescriptions as a schedule

• Activities and their mutual temporal constraints represented as a Simple Temporal Network

• Dispatched for execution and monitored

• Constraint violation triggers interaction

Planning & Scheduling TeamIstc-Cnr

Schedule Execution Monitoring

• Data are continuously retrieved from the stereo cameras (more about this issue later in the talk);

• Activity status is updated at each execution step;• All constraint violations are detected;

behavioral pattern

time nowtime nowtime nowtime nowtime nowtime nowtime nowtime nowtime nowtime now

tmaximum allowed distance

time nowtime nowtime nowtime now

Planning & Scheduling TeamIstc-Cnr

Using the Scheduler’s Temporal Knowledge to generate contextualized dialogues

breakfast cooking lunch

time now

t

medicine

You should hurry up taking your after-lunch medicine!

executed executed executed

Planning & Scheduling TeamIstc-Cnr

breakfast cooking lunch

time now

t

medicine

You should wait a little longer before having lunch!

executed executed

Using the Scheduler’s Temporal Knowledge to generate contextualized dialogues

Planning & Scheduling TeamIstc-Cnr

breakfast cooking lunch

time now

t

medicine

Maybe you should cook yourself something warm to

eat!

executed NOT executed

Using the Scheduler’s Temporal Knowledge to generate contextualized dialogues

Planning & Scheduling TeamIstc-Cnr

The Proactive Interactor

Talking head

Speechrecognition

Interaction Manager

• What is behind the interaction?

A set of active services ….

Simple I/O Engine

Input/Output Channels

Knowledge for Interaction

Planning & Scheduling TeamIstc-Cnr

The User Interaction Agent (1/2)

Speech Recognition

Planning & Scheduling TeamIstc-Cnr

The User Interaction Agent (2/2)

Verbalizations Synthesis

• Simple synthesis of Speech Acts is performed by analyzing the information contained inthe Constraint Violation DB and in the Environment Status DB

Planning & Scheduling TeamIstc-Cnr

Generating environment-coherent behavior

• Coordination of multiple services is achieved by solving a Multi-Agent Coordination (MAC) problem

• The MAC problem is cast as a Distributed Constraint Optimization Problem (DCOP)

• The DCOP is solved by the ADOPT-N algorithm, an extension of the ADOPT (Asynchronous Distributed Optimization) algorithm for dealing with n-ary constraints

Planning & Scheduling TeamIstc-Cnr

Agents, variables and soft constraints

• Through cost functions, soft constraints are used to prefer (for the monitored person) healthy states and avoid dangerous states

• Cost functions are modeled so as to reflect the desiderata of system behavior

• Detailed description in [Pecora & Cesta, Comp. Int. 2007]

Planning & Scheduling TeamIstc-Cnr

Possible Assistant/assisted interactions

• On-demand interaction(Person takes initiative)– Question / answering

• Proactive interaction(RoboCare takes initiative)– Danger– Warning

The RoboCare Environment as a Mixed-Initiative System

Planning & Scheduling TeamIstc-Cnr

Managing interaction in RoboCare

ProactiveOn-demand

Planning & Scheduling TeamIstc-Cnr

Some examples of interaction

Planning & Scheduling TeamIstc-Cnr

Proactive Warning

• Feedback from sensors is a key activator• Explanation triggered by T-REX temporal knowledge

Planning & Scheduling TeamIstc-Cnr

Proactive Alarm for Danger

• A reactive routine is activated with a precompiled plan– (go-to-place; try-interaction; call-emergency)

Planning & Scheduling TeamIstc-Cnr

On Demand Question-Answering

• Query to the temporalized knowledge in T-REX

• Very simple additional internal query capabilities

Planning & Scheduling TeamIstc-Cnr

Related work

• Intelligent assistants– Same domain and similar technologies:

• Autominder [Pollack et-al, 2003], PEAT [Levinson 1997], PEARL [Pineau et al. 2003; Pollack 2005], I.L.S.A. [Haigh, Kiff, & Ho 2006]

– Capability integration: • Similar problems addressed with CALO, CMradar,

etc. … although project scale quite different!

Planning & Scheduling TeamIstc-Cnr

Conclusions

• RoboCare has addressed (among others) – the open challenge of integrating diversified intelligent capabilities to create a

proactive monitoring assistant for everyday life in a domestic environment

• Highlighted in this work– particular use of the internal knowledge of a constraint-based scheduler (the

temporal constraint network) as well as its capability of reasoning on changes in the environment

– constraint violations determine when the system has to interact. The analysis and interpretation of the violation contribute to determine how to interact with the user

– the use of a distributed coordination algorithm to create a coherent behavior of multiple “active agents”

Planning & Scheduling TeamIstc-Cnr

THANK YOU!QUESTIONS?