Augmented Reality Tactile Map with Hand Gesture Recognition

Ryosuke Ichikari1, Tenshi Yanagimachi2 and Takeshi Kurata1

1: National Institute of Advanced Industrial Science and Technology (AIST), Japan

2: Toyohashi University of Technology, Japan

1

13 July 2016, ICCHP 2016@ University of Linz, Austria

Walking Assists for the visually-impaired people

Assisting technologies when they are walking • White canes & guide dogs • Assisting technologies with computer

– Remote Sensing by RGB(-D) cameras or LRF – Voice navigation system based on localization

technology such as GNSS Pre-Journey learning and re-viewing before/after walking • Tactile Map

2 Tactile trajectory

Tactile Map

3 UV Offset printing PIAF

• Maps with tactile presentation for the visually-impaired people

• UV offset printing：Printing transparent plastic on the color maps

• PIAF(Picture in a flash)：Using heat sensitive papers which can swell black-printed area (lines/images)

• Casio’s 2.5D printer：All-in-on solution for color and detailed tactile expression.

Casio’s 2.5D printer

Problems about tactile map

• Difficulties for constructing mental image of the contents with haptic sense – Needs experience and training

• Limitation of the amount of contents • Production issues

– Cost: UV offset printing is expensive – Needs know-how to design contents ⇒Tactile Map Automated Creation System (TMACS) [Watanabe et. al, ICCHP2014 ]

4

Proposed system：Utilizing AR for tactile map

• Augmented Reality; AR – Seamless integration of real

world and virtual world – Visual augmentation is main

stream, but AR concept can be applied for audio sense

• AR Tactile Map（Proposed System）: – Provides audio-visual feedbacks in response to

interactive operations – Enables dynamic and multimodal contents – Can be utilized for learning tactile map

5

Audio-visual AR [Higa et. al 2007]

Characteristics of AR tactile map • Augmentation for the tactile map by audio representation

(for completely blind people) • Enlarging/enhancing display for people with week sight • Providing POR(Point of reference) information • Intuitive interaction by actually touching tactile maps • Configuration with off-the-shelf and varsatile products • Hands-free arrangement

6 Showing enlarged map with voice

Providing POR(Point of reference) information

• Target contents of the AR tactile map: POI（Point of Interest）and POR(Point of Reference)

• POR：Specific point location where it is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor) [Okuno et. al, CSUN2015] [Denoncin,CSUN2014] [Theberge, CSUN2015]

• Integration with POR/POI data base – Voice navigation system with POR/POI data base [Okuno et. al,

CSUN2015]

7

System configuration System configuration of the prototype system • Tablet PC ⇒ For Visual-Audio feedbacks • RGB-D camera：DepthSense DS325 （ToF, Range：15㎝

~100㎝） ⇒For finger gesture recognition and robust extraction of hands area, measurement of the distance between the map and finger

• PC for recognition • Arm for fixing tablet • Tablet mount parts

8

Related Works • ATMaps (Audio-Tactile Maps) using

IVEO’s Interactive tactile graphics. (ICCHP2016 workshop)

• Image to Sound Converter [Kuroda] – Image is converted to sound by changing key

and difference of L/R sound – Transparent tactile sheet can be overlaid – Display size and UI are limited by touch display

• Kiko-AR by cybernet systems – Mobile AR system based on character recognition and audio

presentation – Collaborating with a printing company to produce printed material

with tactile presentation and AR augmentation. – Need to shoot with smartphone’s camera

9

Characteristics of AR Tactile Map： - Using versatile products - Various sizes and types of tactile maps can be used - Intuitive interface for the system are focused

Recognition of gesture • Recognizing “Tapping”, “Double

Tapping”, “Flat-hand gesture” gestures as interaction commands

• Algorithm for detecting “Tapping” – Extracting hand area by RGB-D camera with background differencing

of depth image and distance threshold – Detecting state transition of distances within time window （①->②->①)

10

①Finger on the map ②Finger in the air for tapping time

Distance between map and finger ②

①

Size of time window

• Flat-hand gesture: Tracking flat-hand position when user touches maps by gradients and distances

Automatic identification and tracking of tactile map

11

• ORB Feature point detector/local feature descriptor is used for identifying tactile map with RGB image

• Estimating homography matrix between rectified image templates and input image

Mapping party

fo

Previewing by printing as tactile map

Walking assistance by voice navigation

Gathering acctual sites for making map (Mapping Party)

Virtual Mapping Party with AR Tactile Map

Omni-directional image and 3D-Sound Recording

Pre-learning of the walking route with the AR Tactile Map

Utilized at voice navigation

Mapping Party Anytime, Anywhere

Requests for Mapping

Workshop on Virtual Mapping Party • Held at Miraikan, Tokyo, Japan • March 18th-19th, 2016. 1h x 6 times (total 42 participants) • Collecting POR/POI by experiencing omnidirectional

images(360 deg.) and 3D sounds with HMD(Oculus, GearVR. Etc.), PC and headphone

• Confirming and requesting POR/POI by AR tactile Map • The number of the registered POR/POI: 598

Usage of the AR tactile map for virtual mapping party • Allowing the visually-impaired people to join

the mapping party by gesture – Flat-hand: Confirming the existence of POR/POI where

user touches – Tap: Confirming POR/POI

with Text-to-Sound when user taps the specific point

– Double Tap: Requesting the POR/POI for the specific points of the map

Summery of a questionnaire for the AR tactile map • Accuracy of the gesture recognition has a room for

improvement • Requesting function is good because the it can reflect

visually-impaired people's demand. • Scale size of the map was not fitted. Considering to

accuracy of the gesture recognition, the scale size of tactile maps and the density of the distribution of the contents need to be carefully considered.

• It seems to be interesting if the voice recognition can be combined with gesture interface.

• A participant with visual impairment like annotations printed in Braille, but another participant with visual impairment does not like them because they disturbed perception of the shapes of roads and buildings.

17

On going works • Implementing another configuration of the AR

tactile map with HP’s Sprout. • Sprout is a PC which has

– A projector – A touch pad – depth and color cameras

• Sprout enables – touch recognition through

the map – Projection onto map – Sheet tracking – 3D scanning

18

AR tactile map with Sprout

• Tapping: Checking registered POR/POI

• Dragging: Searching • Double Tapping:

Requesting POR/POI • Projecting POR/POI

points on the map

No other device or PC is required

19

The proposed system AR tactile map with Sprout

Gesture recognitions

hand gesture recognition by RGB-D camera

touch detection by touch pad

Information augmentation

audio + enlarging image audio + enlarging Image + projection onto map

System configuration & cost

PC($500~) + RGB-D camera($200) → can be replaced with a tablet PC with RGB-D camera

Sprout($1500)

Limitation: no visual augmentation onto map

Size of map and UI for the map

Advantages mobility, flexibility of UI and size of map, moderate price

visual augmentation, accurate gesture recognition

Comparison with the proposed system

Conclusions • AR tactile map: Audio-visual augmentation for tactile map

by utilizing hand gesture interface with RGB-D camera. • Virtual Mapping Party with the AR tactile Map • AR tactile map by HP’s Sprout • Future works

– Increasing accuracy of the gesture recognition – Covering individual difference of gesture – Noise and unclear situation (sunlight, close to paper） – Other configurations of AR tactile maps (Sprout etc.)

• Acknowledgement – This research is conducted as a part of the project sponsored by

JST RISTEX.

20

Thank you!

• Ryosuke Ichikari, Ph.D. – Postdoctoral Researcher at AIST – Contact: r.ichikari@aist.go.jp

21