Development of Web-based Platform for Privacy Protective Avatar Mediated Distant-Care

Development of Web-based Platform for Privacy Protective Avatar Mediated Distant-CareYu Kobayashi, Dai Hasegawa, Shinichi Shirakawa,Hiroshi Sakuta, Eijun NakayamaAoyama Gakuin University, JAPAN1

Today, I am going to talk about our work on the development of Web-based platform for privacy protective avatar mediated distant-care.1

Social Requirements for the ElderlyDifficult to notice the change of elderlys health statusUrgent:Falling down, Heart attack, Long-Term Change:Blood pressure, Posture,

Needs of daily support and watching over for elderlyUsed to be supported by family members and/or local communitySupporting business by human workersIncrease of social welfare expense2*Cabinet Office, Government of Japan, Annual Report on the Aging Society, 2015

I will begin with the background of our work.

The number of households in which an elderly lives alone has increased.Social isolation of the elderly in such households has been acknowledged as a problem because it becomes difficult to notice the change of their health status.

Additionally, daily support and watching over the elderly, which used to be supported by the mutual aid of family members and local community, are now supported by human workers involved in nursing and welfare services, offered by local governments or private sectors.This is also recognized as one of the problems of this isolation, because it inevitably leads to the increase of social welfare expense.

2

Distributed ICT Care ServicesCurrent statusSimple and Low-tech(e.g. Emergency call services with push button devices)

Advanced ICT devices are not widely used(e.g. cameras, sensors, )

Trade-off problem of convenience in care services and privacy invasion for Elderly and family members3

Because of this isolation problem, various ICT distant-care services are provided. However, in the current state of popularization of ICT based care services, only simple and low-tech services are widely used, such as an emergency call with push button, locating services with wearable IC tags.Also, the care services that utilize advanced ICT device such as cameras and sensors are not widely used.

Additionally, the elderly and family members avoid such advanced ICT technologies because of the concern about privacy invasion.

3

PrivacyThe persons right to control personal information revealed to othersShould be opt-in (especially in Europe)

Opt-in policyPermission-basedOf a selection, the property of having to choose explicitly to join or permit something4

it is said that privacy, the persons right to control personal information revealed to others, should be opt-in especially in Europe.

also, opt-in policy is the property of having to choose explicitly to join or permit something.

4

Observation MethodsCamera-based observations are often appliedNo effective solutions to protect ones privacy in capturing video(e.g. Filtering and automatic blurring cannot conceal ones nudity)Capturing video is not suited for opt-in(Difficult to filter the information)

Sensor-based observationsNo video capturingOnly collects the information which is permitted to be seen by the elderly (Suited for opt-in)Difficult to understand the meanings of the value of sensor data

Observe an avatarOnly collects the articular angle data by motion-capturing system (Kinect)Observe an elderly as an avatar which is rendered by sensor dataCan be recognized elderlys behavior

5

When designing distant-care systems to observe the elderly, it is required to continuously observe the elderly for a long-time.For this reason, observations that are based on video capturing are often used. To protect ones privacy in capturing video, many researchers developed technological solutions such as filters, automatic blurring and so on.However, such solutions are not enough to protect ones privacy completely.For example, Filtering and automatic blurring cannot conceal ones nudity.

Also, capturing video is not suited for opt-in because it is difficult to filter the information provided.

On the other hand, in sensor-based approach, it collects only the kinds of information that are permitted to be seen by the elderly, but it is difficult to understand the meanings of the value of sensor data.

Then, we proposed a solution that render an avatar of the elderly by using sensor data which is collected by motion capturing system.By doing so, it can be recognized elderlys behavior without capturing video.5

PurposePropose a platform for distant-care systemSensor-based observation (motion capturing with Kinect)observe the avatar of an elderly

AdvantagesPrivacy protectiveNo video capturing requiredSuited for both opt-in and opt-outReal-time and remote accessible via Internet6

Therefore, our purpose is to propose a platform for distant-care system.In the platform, the elderly is observed as an avatar which is rendered by articular angle data of the Kinect.The system is privacy protective and suited for both opt-in and opt-out. Using this system, caregivers can observe the elderly real-time, and can be remote accessed via internet.

6

System Overview

PC

Server

Multi DevicesSender Program

Acquire DataProcess Data

WebSocket Server

Web Browser

Renderwith WebGLKinect, Optional SensorsHTTPWebSocket

Browse

HTTP ServerRenderingProgramAvatarModelWebSocketArticular AngleArticular AngleAvatar ModelRenderer7Caregiver SideServer SideElderly Side

I will explain about the overview of our platform.

The system is composed of a three-sided client-server model: the server, caregiver side client, and elderly side client.The client of the elderly side consists of a KINECT sensor, programs for articular angle data acquisition and data processing, and a program to send the processed data to the server.

The server sends the data [click] received from the elderly side client [click] to the caregiver side program.The caregiver side program runs on a web browser. The program receives the articular angle data from the server and applies it to the avatar model data for rendering.

7

Avatar Animation8

OpenGL, OpenGL ES, DirectXNeed to developfor each device or OSInstalling application needsusers operationDifficult to support

WebGLA JavaScript APIRuns on modern browsersPCs and Mobiles

Generally, OpenGL, OpenGL ES for mobile, and DirectX are used in the development of applications using 3D Graphics. However, one of the problems in using these APIs is that we have to develop and to support applications compatible with each device and OS used. Another problem is the application has to be installed into the devices by users.

On the other hand, WebGL, a JavaScript API, runs on modern browsers such as Google Chrome, Firefox, Safari, and so on, without installing any other software.For this reason, the same program can be run on a web browser in both of PCs and mobiles without change.

[click]Therefore, we used WebGL for rendering avatar animation.8

Client-Server CommunicationHTTP ProtocolA traditional communication protocol between a server and a web browserCannot send the data to web browser activelyEvery request / response needs a relatively large amount of HTTP header(high-load)

WebSocket ProtocolReal-time, two-wayOnly one HTTP header is needed (low-load)Supported in almost all modern web browsers9

The sensor data is sent from the elderly side client through the server to the caregiver side client, 15-30 times per second.

HTTP, a traditional communication protocol between a server and web-browser, requires a large amount of HTTP header in every request and response communication which leads to a significant delay for such kinds of communication.Additionally, the server cannot send the data actively to the web browser with this protocol.

On the other hand, WebSocket, a extended HTTP protocol, can realize real-time two-way communication between a server and web-browser.And also, WebSocket protocol is supported in almost all modern web-browsers.

[click]Therefore, we used WebSocket protocol for client-server communication. 9

Sensor IntegrationVarious types of wearable sensors have been developed(e.g. blood pressure, heartbeat-rate, oxygen level)

10

Apple WatchSamsung Galaxy Gear

Recently, various types of wearable sensors such as Apple Watch and Galaxy Gear have been developed.Using these sensors, we can continuously monitor ones vital information such as blood pressure, heartbeat-rate, oxygen level, and so on.

The platform we developed can be customized by integrating these wearable devices to adapt to a huge variety of health-care situations.

10

Example of Sensor Integration

Temperature / HumiditySensor ModuleHeartbeat-rateSensor ModuleArduino UnoRaspberry PiWi-Fi AdapterBattery11{ temperature: 25, humidity: 65, heartbeat-rate: 80}

This is an example device of sensor integration.

We prepared a sensing device including [click] a temperature and humidity sensor, and [click] heartbeat-rate sensor.These sensors are connected to a micro computer Arduino Uno.[click] and Arduino Uno is connected to a one-board computer, Raspberry Pi[click]. The Raspberry Pi is connected to a Wi-Fi adapter[click] and Battery.

The sensor data is sent from Arduino Uno to Raspberry Pi via serial communication, then to the server via Wi-Fi with WebSocket protocol.The sensor data is processed and formatted in JSON [click].11

Example

12

Let me show you an example of how our system works by using the platform.

This is a screen capture of caregiver side client running on the web browser.The avatar which reflects the elderlys behavior is rendered on the left side.The location of sensed joints are depicted in the box at lower right.

The data of integrated sensors such as temperature, humidity, and heartbeat-rate, are shown at the green boxes at upper right.When the value of sensors goes beyond the set threshold, the green box turns red.12

Required Bandwidth TestData amount (15 FPS)Articular angles: 6 Kbytes (JSON)Sensors: 50 Bytes (JSON)

Measure the performance 10 seconds after communication was established

IndexesThe average frame-rateThe average time lapse after the server sent the data (latency)

13

It is expected that delays will occur in the animation of the caregiver side client.

We explored the minimum bandwidth for the system to properly work without delays.In our test, we set the frequency of data acquisition/sending to 15frames per second, which is the lower limit for the animation to be recognized as ones behavior.

The data amount of the articular angles was 6Kbytes a frame. And the total data amount of example sensing device was 50Bytes a frame. Both of these data amount are after processed to JSON format.

We measured the performance of the system ten seconds after communication was established.

We used two indexes:The average frame-rate, and the average time lapse after the server sent the data.13

Result of Required Bandwidth TestAbove 800Kbit/s :the animation was rendered smoothly without delay

Below 800Kbit/s :the delay was accumulated with each passing moment(e.g. 350Kbit/s, 6FPS, 11s latency)

The animation of the avatar rendered without delayvia mobile network.

14

This graph shows the frame-rate and the time lapse when the bandwidth was controlled from 350Kbit/s to 850Kbit/s.The results shows that when the bandwidth was above 800Kbit/s, the animation was rendered smoothly without delay.

However, when the bandwidth was below 800Kbit/s, delay was seen and was accumulated with each passing moment.For example, when the bandwidth was 350Kbit/s, the 11 seconds delay was occurred.

From the result, assuming that mobile telecommunication bandwidths of late years are not under multiple Mbit/s, caregivers can observe the animation of the avatar without delay via mobile network.14

Conclusion and Future WorksConclusionWe proposed a web-based platform for privacy-protected avatar mediated distant-care.We measured the bandwidth required for avatar animation without delay.

Future WorksWe will evaluate the systems UsabilityPsychological burdenAdditional ResearchReveal how to detect any other particular health problems.Test demonstration at a nursing home for elderly.15

Psychological burden such as how elderly react by having their behaviors tracked and showed in the form of virtual avatar of both being the observedand the caring by introducing our system into the nursing and personal care facility.

We held test demonstration at a nursing home for the elderly, we will continue doing so as additional research.15

AcknowledgementThis research was partially supported by Strategic Information and Communications R&D Promotion Programme (SCOPE) .16

16