Mobile Medication Assistant for the Elderly

António Teixeira, Flávio Ferreira, Nuno Almeida, Ana Filipa Rosa

S4S Presentation

Department of Electronics, Telecommunications and Informatics (DETI)Institute of Electronics Engineering and Telematics of Aveiro (IEETA)

Health School (ESSUA)University of Aveiro

Portugal

2

Method

• An iterative/cyclic method composed by– Requirements Analysis

• Get Personas, Problem and Vision• Brainstorming (with people from different domains)• Identify Persona expectations• Construct Context Scenarios• Identify requirements

– Developing the application– Testing the application– Evaluation with users

S4S Presentation

3

Motivation

• The elderly present high levels of non-adherence to medication

• High number of errors in taking medication, concerns about side effects, lack of motivation, memory issues

Goal: Mobile application for seniors to:• improve adherence to medication and minimize errors

• serve as a testbed for research in the best uses of different modalities with a particular focus on speech.

S4S Presentation

4

Persona & Persona Expectations

S4S Presentation

Persona Expectations

•Able to use the same application with arthrisis

• Allow speech interaction

• Help prevent gaps in medication

• Report the medication and side effects

• Could use the App even with poor ability to work with electronic devices

Persona: PrimaryName: Emília RodriguesAge: 62 years old

Mrs. Emília lives in Coimbra with her husband Filipe Rodrigues. She’s housewife. Mrs.Emília doesn’t know very well to work with electronic devices.She is right handwriter.She is diabetic and has arthritis in her right superior member. Her health condition requires a regular and diary medication.Mrs. Emília has the habit to call her daughter during the dinner preparation. However she has some difficulty in doing the two activities at the same time due to her limitation in the right superior member.She has weekly appointments in her local health centre for surveillance.Mrs. Emília would like to buy equipment that facilitates the contact with her daughter. Something that allow her to control her medication that she forgets to take.

5

Requirements

S4S Presentation

Data Requirements• When to take a medication

• Why to take a medication

• Medication side effects

• Medication expiration date

• Maximum dose

• Distributed data

• User profile• Clinical information

User Requirements• Inform in a casual way

• Use speech for everything

• Credible and reliable

• Avoid content overload and small icons

• Personalization

• Adaptation

• Extra care with language, dimension and touch

6

System Architecture• Windows Phone 7• Web services to overweight features• Several modules to control modalities and intelligence

S4S Presentation

Face Detection

User

7

Speech Service• Add Grammar: allow the

input of a new grammar for the service in GRXML or ARPA format

• Get Grammar: gets the list of all grammars in the service and their names

• ASR: translates na audio stream to text, using the microsoft speech platform

• TTS: converts text to speech

S4S Presentation

TTS

ASR

AddGrammar

GetGrammars

SpeechWS

GrammarRepository

MicrosoftSpeech

Platform

8

Image Service• Aims to generate a synthesized image of a pill/packages

– Inestead of having a database with all the images

• Based on pill/package discription it return an image

• Information needed:– Pill/Package shape– Pill/Package pattern– Pill/Package colors– Pill/Package name

S4S Presentation

9

User, NLG and FaceDetector Service• The User service that manage the user medication data and plan• The User service provide the push notification service to display

alerts

• The NLG (natural language generation) service that automatically generates sentences with the provided information

S4S Presentation

• The Face Detector service that detects faces in a image and returns the size of the closer face

10

Main Features• Alerts

– to remember users about medication schedules

• Help – on common medication related questions

• I forgot to take my medication?• What are the side effects of this medication?

• Medication insertion and management by third parties.• Configurable interaction• How to use application guide and tips• User knowledge adaption

S4S Presentation

Alerts Help

Guide Options

11

Multimodal Input

• Touch– Accepted for all operations

• Speech– MSFT Speech Plat. (MSP)– Can be more intuitive and

faster than touch

Choose:

1 ) “I forgot my Medication” 2) “I forgot a medication" 3) Select the forgotten medicine

User says:“I forgot to take X”

What to do if a medication intake was forgotten?

TOUCH

VOICE

S4S Presentation

Choose:

1 ) Select the right option

User says:“What are the side

effects?”

What are the side effects of this medication?

TOUCH

VOICE

12

Multimodal Output• Graphics and Text– Panoramic view– Medication images generation– Medication details

• Speech– TTS (e.g. “The side effects of medication are…”)

– Use of Natural Lang Generation

“You should take one tablet of Aspirin”

MOBACC 2013

Medication nameQuantityApplication methodIntake interval

13

Adaption to Context

• User distance to mobile phone– Relevant display elements resized if user

moves phone closer/farther– Deal with vision problems (it will depend

of user – myopia or astigmatism)

• Ambient noise level– Sound sampled while user is not

speaking– Volume level adjusted accordingly

MOBACC 2013

14

Further work• Improve zoom adaption algorithm and add it to more views

• Implement the dark mode of the application

• Improve the auto voice detect algorithm

• Improve the speech interaction

• Improve “how to use” features

• Add features– Auto adaptable help system– Advice for more daily problems about medication intake

S4S Presentation

15

Thank you for your attention.

S4S Presentation