![Page 1: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/1.jpg)
Technology to Support Individuals with Cognitive Impairment
Martha E. PollackComputer Science &
EngineeringUniversity of Michigan
![Page 2: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/2.jpg)
Challenges for Older Adults
• Physical
• Social/emotional
• Sensory
• Cognitive– Example: Alzheimer’s
65-74: 5%75-84: 20%> 85: 50%
intelligent wheelchairs
elder-friendly email and chat rooms
programmable digital hearing aids
Some of the technology also useful for younger individuals with cognitiveimpairment (e.g., TBI patients, developmentally disabled people)
![Page 3: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/3.jpg)
Why Build Cognitive Orthotics?
• Cognitive impairment can impact performance of daily activities
• Can lead to decreased quality of life, and potentially institutionalization– Costly
– Further decreases quality of life
• Goals– Improve performance of routine functional activities and thereby
support longer aging-in-place
– Reduce caregiver burden
![Page 4: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/4.jpg)
Activity Cueing
• Guide an individual through steps in a sequential or conditional-branching process
• Work done both on ADLs/IADLs (e.g., handwashing, cooking) and on functional job tasks (e.g., janitorial)
Handwashing Assistant [courtesy A. Mihailidis, U. Toronto]
![Page 5: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/5.jpg)
Prospective Memory Aids
• Tend to be designed for less severely impaired individuals
• Provide them with personalized, adaptive reminders about daily activities
• On the market: glorified alarm clocks!– Exception: PEAT
![Page 6: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/6.jpg)
Autominder
• Model, update, and maintain the client’s plan– Including complex temporal and causal constraints
• Monitor the client’s performance– Updating the plan as execution proceeds
• Reason about what reminders to issue, and when– To most effectively ensure compliance, without sacrificing
client independence
![Page 7: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/7.jpg)
Autominder Example
Req/Opt Activity Allowed Expected Observed
R toilet use 10:45-11:05
R lunch 12:00-12:45
O TV 14:00-14:30
10:55
R toilet use13:55-14:15
REMIND 12:25
REMIND 13:55
12:28
![Page 8: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/8.jpg)
Robot Platform
• Nomadic Technologies Scout II
w/custom-designed head
– Multiple sensors: lasers, sonars, microphone, touchscreen, camera vision, wireless ethernet
– Effectors: motion, speakers, display screen, facial expression
“Pearl”[courtesy Carnegie MellonUniv. Robotics Institute]
![Page 9: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/9.jpg)
“Ubicomp” Platform
• Handheld or wearable device– Currently: HP iPaq
• Deployed in a “smart” environment with multiple sensors (ubiquitous computing environment)
![Page 10: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/10.jpg)
Client Client ModelerModeler
Plan Plan ManagerManager
IntelligentIntelligentReminderReminderGeneratorGenerator
ClientPlan
Activity Info
Inferred Activity
Sensor Data
Reminders
Client Model Info
Activity Info
Preferences
Plan Updates
ClientModel
Autominder ArchitectureWhat should the client do?
Technologies: Automated Planning, Constraint-Based Temporal Reasoning
What is the client doing?
Technologies: Dynamic Bayesian InferenceIs a reminder needed?
Technologies: Iterative Refinement Planning, Reinforcement Learning
![Page 11: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/11.jpg)
Plan Manager: What should the client do?
• Maintains up-to-date record of client’s planned activities– Eating, hydrating, toileting, medicine-taking, exercise, social activities,
doctor’s appointments, etc.
• Updates plan and propagates constraints when– New planned activity added.
– Existing activity modified or deleted.
– Planned activity performed.
– Critical time bounds passed.
• Models plans as Disjunctive Temporal Problems and uses AI planning and CSP technology for updating.
![Page 12: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/12.jpg)
Client Modeler: What is the client doing?
• Given information:– Sensor input: client moved to kitchen– Clock time: at 7:23 a.m.– Client plan: breakfast should be eaten between 7 and 8
– Model of previous actions: client has not yet eaten breakfast– Learned patterns: 82% of the time, client starts breakfast between 7:10 and
7:25– Reminder information: we issued a reminder at 7:21
• Infers probability that various events have occurred– that the client has begun breakfast
• Uses Bayesian reasoning technology, addressing limitations of previous approaches to handle complex and dynamic temporal relations
![Page 13: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/13.jpg)
Intelligent Reminder Generation: What should Autominder do?
• Given a client’s plan and its execution status:– Easy to generate reminders
• Remind at earliest possible time of each action
– Harder to “remind well”• Maximize likelihood of appropriate performance of ADLs and
other key activities• Facilitate efficient performance• Avoid annoying client• Avoid making client overly reliant
• Uses local search tools to incrementally refine reminder plans; also investigating reinforcement learning for adaptive interaction policies
![Page 14: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/14.jpg)
Current Status
• System fully implemented• Early version “tested” on Pearl at Longwood Elder
Care Facility in Oakmont, PA• Later version currently being tested on handhelds,
without sensing/ with simple (RFID-based sensing), with TBI patients from U of M Med Rehab Clinic
• Larger scale wireless sensing technology being developed and integrated into Autominder in the lab, for field testing later this year
![Page 15: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/15.jpg)
Key Challenges for Cognitive Orthotics
• Technological– Advanced AI Techniques
– HCI
– Sensor Networks for Inference of Daily Activities
– Mechanisms to Ensure Privacy and Security
• Policy– Mechanisms to Ensure Privacy
– Reimbursement Policies
![Page 16: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/16.jpg)
For More Information…
www.eecs.umich.edu/~pollackm
![Page 17: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/17.jpg)
Extra Slides Follow….
![Page 18: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/18.jpg)
The Plan Manager
• Maintains up-to-date record of client’s planned activities and their execution status– Eating– Hydrating – Toileting– Medicine-taking – Exercise – Social activities – Doctors’ appointments– etc.
![Page 19: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/19.jpg)
How Does it Work?
• Models constraints on future actions– Lunch takes between 25 and 35 minutes – Take meds within one hour of finishing lunch – Watch the news at either 6pm or at 11pm
• Performs efficient constraint processing when key events occur:– New planned activity added.– Existing activity modified or deleted.– Planned activity performed.– Critical time bounds passed.
![Page 20: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/20.jpg)
Small Example
ClientPlan
1. New Activity2. Mod/Deletion3. Activity Execution4. Passed Time Bound
PLAN MANAGER
:0 MS – LE :60“Take meds within 1 hour of lunch”
LE = 12:15“Lunch ended at 12:15”-----------------------------12:15 MS 13:15“Take meds by 1:15”
![Page 21: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/21.jpg)
Client Client ModelerModeler
Plan Plan ManagerManager
IntelligentIntelligentReminderReminderGeneratorGenerator
ClientPlan
Activity Info
Inferred Activity
Sensor Data
Reminders
Client Model Info
Activity Info
Preferences
Plan Updates
ClientModel
Autominder Architecture
![Page 22: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/22.jpg)
CM: Client Modeler
Given what can be observed• Sensor input: client moved to kitchen • Clock time: at 7:23 a.m.• Client plan: breakfast should be eaten between 7 and 8• Model of previous actions: client has not yet eaten breakfast• Learned patterns: 82% of the time, client starts breakfast between 7:10 and 7:25• Reminder information: we issued a reminder at 7:21
Infers what has been done• Client Activity: probability that client has begun breakfast
![Page 23: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/23.jpg)
How Does it Work?
• Models probabilistic relations among observations and actions
• Performs Bayesian update, extended to handle temporal relations• Asks for confirmation when needed!
started
breakfast
breakfastreminder issued
went tokitchen
reminder kitchen start-breakfast Y Y .95 Y N .10 N Y .8 N N .03
![Page 24: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/24.jpg)
Client Client ModelerModeler
Plan Plan ManagerManager
IntelligentIntelligentReminderReminderGeneratorGenerator
ClientPlan
Activity Info
Inferred Activity
Sensor Data
Reminders
Client Model Info
Activity Info
Preferences
Plan Updates
ClientModel
Autominder Architecture
![Page 25: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/25.jpg)
Intelligent Reminders
• Decides whether and when to issue reminders• Given a client’s plan and its execution status:
– Easy to generate reminders• Remind at earliest possible time of each action
– Harder to “remind well”• Maximize likelihood of appropriate performance of
ADLs and other key activities• Facilitate efficient performance• Avoid annoying client• Avoid making client overly reliant
![Page 26: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/26.jpg)
How Does it Work (Now)?
LB D
TV
Midnight
8:00 16:0012:00
12:00
LB D
TV
Midnight
8:00 16:0012:00
12:00
LB D
TV
Midnight
8:00 16:0012:00
12:00
8:30 12:32
![Page 27: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/27.jpg)
How Will it Work?
• Use reinforcement learning to deduce an optimal reminding strategy
• Model the system as a Markov decision process that– Senses the environment
– Decides what action to perform
– Receives a “payoff”
and then “learn” the best policy after repeated interactions
![Page 28: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/28.jpg)
Current Status of Autominder
• V.0 (Autominder + Pearl) field-tested for client acceptability on Pearl at Longwood Elderly Care Facility in Oakmont, PA, summer, 2001
• V.1 of Autominder implemented – Java, Lisp on Wintel machines
• Data collection with three Oakmont residents completed summer 2002; with Ann Arbor TBI patient summer 2003
• Systematic field-testing to begin momentarily with TBI patients
![Page 29: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/29.jpg)
• Many projects going on to develop technology to support (older) individuals with cognitive impairment
• With the potential to have a huge impact • But still lots of issues to resolve:
– A host of scientific questions and engineering challenges
• Sensor interpretation
• Interface design
• . . .
– Question of cost and reimbursement structure
– Privacy, privacy, privacy!
Conclusions
![Page 30: Technology to Support Individuals with Cognitive Impairment Martha E. Pollack Computer Science & Engineering University of Michigan](https://reader035.vdocument.in/reader035/viewer/2022062417/5514c765550346b0338b4bd1/html5/thumbnails/30.jpg)
Acknowledgements
Autominder• DTP/PM:
– Ioannis Tsamardinos– Sailesh Ramakrishnan – Cheryl Orosz
• CM:– Dirk Colbry– Bart Peintner
• IRG:– Colleen McCarthy– Matt Rudary
• System Integration:– Laura Brown– Martina Gierke– Peter Schwartz– Joe Taylor
Funders•National Science Foundation•Intel Corporation[Supporting Technology: DARPA, AFOSR]
PearlSebastian Thrun, Mike Montemerlo, Joelle Pineau, Nick Roy
Rest of the Nursebot TeamJacqueline Dunbar-Jacob, Sandra Engberg, Judy Matthews, Sara Keisler, Don Chiarulli, Jennifer Goetz