multi-lingual and deviceless computer access for disabled users
DESCRIPTION
MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS. C.Premnath and J.Ravikumar S.S.N. College of Engineering TamilNadu. Abstract. Hand forms one of the most effective interaction tool for HCI. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/1.jpg)
MULTI-LINGUAL AND DEVICELESSCOMPUTER ACCESS FOR DISABLEDUSERS
C.Premnath and J.Ravikumar
S.S.N. College of Engineering
TamilNadu
![Page 2: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/2.jpg)
Abstract Hand forms one of the most effective
interaction tool for HCI. Currently, the only technology that
satisfies the advanced requirements is glove-based sensing.
We present a new tool for gesture analysis, by a simple gesture-symbol/character mapping esp. suited to the disabled users.
![Page 3: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/3.jpg)
AIM :To use the hand gestures of the user to help them control/access the computer easily
OBJECTIVE : To provide simple and cheap system of
communication to people with single or multiple disabilities
To overcome the language problems in communication/computer access.
![Page 4: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/4.jpg)
INTRODUCTION
Difficulties and impairments reduce computer use .
Direct use of the hand as an input device is an attractive method for providing natural human-computer interaction (HCI) .
![Page 5: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/5.jpg)
Currently, the only technology that satisfies the advanced requirements of hand-based input for HCI is glove based sensing.
Hinders the ease and naturalness with which the user can interact with the computer controlled environment
Requires long calibration and setup procedures.
![Page 6: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/6.jpg)
Glove-Based Approach
Basic operation is to sense the gesture by electric/magnetic contact or by monitoring threshold values in chemical/electrode based sensors.
The sensors are connected to a control unit to find the gesture.
Cost, dexterity and flexibility.
![Page 7: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/7.jpg)
Hinders the ease and naturalness with which the user can interact with the computer controlled environment
Requires long calibration and setup procedures.
![Page 8: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/8.jpg)
Computer vision has the potential to provide much more natural, non-contact solutions.
![Page 9: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/9.jpg)
Gesture recognition methods Model-based approach Appearance-based approach
![Page 10: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/10.jpg)
Model-Based Approach Model based approaches estimate the
position of a hand by projecting a 3-D hand model to image space and comparing it with image features.
The steps involved are: Extracting a set of features from the input
images Projecting the model on the scene (or back-
projecting the image features in 3D) Establishing a correspondence between
groups of model and image features
![Page 11: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/11.jpg)
Appearance-Based Approach Appearance based approaches estimate
hand postures directly from the images after learning the mapping from image feature space to hand configuration space.
The feature vectors obtained is compared against user templates to determine the user whose hand photograph was taken.
![Page 12: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/12.jpg)
Require considerable research on mapping and other relevant work.
Actually allow us to create simple and cost effective systems.
![Page 13: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/13.jpg)
Systems providing computer-access for people with disabilities JavaSpeak
Parse the program and "speak" the program’s structure to a blind user
ViaVoice, which has a published API, is used as the speech reader.
![Page 14: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/14.jpg)
Emacspeak Provides functions geared
directly to programming. Only for someone familiar with a
UNIX environment.
![Page 15: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/15.jpg)
METHODOLOGY Novel approach of mapping the
character set of the language with the possible set of hand gestures and executes the actions mapped for the particular gesture. Capture the user’s gesture Manipulate and create a 5-digit code Execute required system operation
User-friendliness – providing audio request.
![Page 16: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/16.jpg)
The phases involved IMAGE CAPTURING PRE-PROCESSING EDGE DETECTION EDGE TRACKING CODE GENERATION ACTION EXECUTION
![Page 17: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/17.jpg)
Image Capturing
Setup and capture
![Page 18: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/18.jpg)
Pre-processing
Synthetic image An arithmetic operation is performed
with the different channels
![Page 19: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/19.jpg)
a) Sample input image
b) Synthetic image
![Page 20: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/20.jpg)
Edge Detection
Need for edge detection Edges Edge detection methods
Fourier domain Spatial domain
![Page 21: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/21.jpg)
GradientMagnitude operation Spatial Domain Method Performs convolution operations on
the source image using kernels.
![Page 22: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/22.jpg)
Sample Output
a) Synthetic image
b) Edge detection output
![Page 23: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/23.jpg)
Edge Tracking
Find critical points.
![Page 24: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/24.jpg)
![Page 25: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/25.jpg)
![Page 26: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/26.jpg)
Lets us see in detail how we trace fingertip shown below .
In-depth finger tip image
![Page 27: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/27.jpg)
Tracing for finger valley shown below is done in the exact reverse manner as discussed for finger tip
In-depth finger valley image
![Page 28: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/28.jpg)
Output after edge tracking
a) Critical points marked with red dot.
b) Finger length using Pythagoras Theorem.
![Page 29: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/29.jpg)
CODE GENERATION
Using phalanx information
![Page 30: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/30.jpg)
Information about the phalanxes of the right hand
![Page 31: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/31.jpg)
Values to be assigned 1 if the finger is open. 0 if the finger is half-closed i.e., only
the proximal phalanx is visible. Already have data about the full finger
length information of the user During code generation,
1 assigned when approximate matches with the stored value
0 when the obtained finger length is half that of the corresponding one in the database.
![Page 32: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/32.jpg)
5 fingers - 2 values each Overall 32 (2×2×2×2×2) action
gestures.
![Page 33: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/33.jpg)
![Page 34: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/34.jpg)
Mulitilingualism Map the gestures currently associated
with only English characters, to the characters in other languages by analyzing the phonetics and their translation to English.
![Page 35: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/35.jpg)
For example, the words involving Tamil characters , Hindi characters , Telugu characters , Malayalam characters , Can all be mapped with the English
letter ‘A’,
![Page 36: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/36.jpg)
European languages the alphabet is almost similar.
Voice engine support important Latin Non-Latin languages where we have
no space between words (Hindi and Arabic), are supported by tailoring the
Run-time speech engine Free TTS
![Page 37: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/37.jpg)
ACTION EXECUTION
The tree panel Acquires the path information of a
file/folder whenever that particular file/folder gets selected by the user’s input.
The filename is passed to the speech synthesizer unit and verification done by the user.
![Page 38: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/38.jpg)
JMF player controls the browsing work For example, if character ‘A’ is passed to
the file manager then it passes the next file/folder name starting with the letter ‘A’ to the JMF player.
File operations Type of the file selected (media/text)
and the user’s input gesture. Pass the file to the JMF player unit Execute appropriate operations
![Page 39: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/39.jpg)
Features
Minimized cost and user friendliness of the project.
Flexibility to change the gesture mapping based upon user’s comfort
Ambidexterity
![Page 40: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/40.jpg)
Limitations
Gesture mapping for languages with large character sets like Chinese and Japanese.
Voice support from the speech engine
![Page 41: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/41.jpg)
Conclusion
Novel approach for providing computer access to disabled user with a multilingual method.
Overcomes the problem of user’s age involved and physical measures.
Support for the illiterate users.
![Page 42: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/42.jpg)
FUTURE WORK
Both the hands as input aided with touch pad technology for the computer access.
1024 (210 values - taking 2 values for each of the ten fingers) Assume the ten bit code 10000 10010 is
associated with the word “Pause”, then the system would type the word “Pause” if the environment is a text editor and PAUSE the current music track if the environment is a music player.
![Page 43: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/43.jpg)
Map the gestures with system commands.
Other applications currently inaccessible for disabled users.
![Page 44: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/44.jpg)
The project has proposed an ambidextrous system where the computer access is all within your 5 fingers and the proposed enhancement has the potential to bring the world in your hands.
![Page 45: MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS](https://reader035.vdocument.in/reader035/viewer/2022062422/56813259550346895d98e792/html5/thumbnails/45.jpg)