a robust method of detecting hand gestures using depth sensors yan wen, chuanyan hu, guanghui yu,...
TRANSCRIPT
1
A Robust Method of Detecting Hand Gestures Using
Depth Sensors
Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang
Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on
2
Outline
Introduction
Related Works
The Proposed Method
Experimental Results
Conclusion
3
Introduction
4
Introduction
In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. Medical system, computer games, and human-robot
Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.
Goal: detect hand gestures with color and depth information
5
Related Works
6
Related Works
Body [4][5] V.S. Hand
Hand Superiority: simple
Inferiority: small scale, low resolution
Strict condition: cluttered background, lighting variation
7
Related Works
Hand gesture recognition Only color[12]
Data glove[7]
Training process[9][10]
Earth Mover’s Distance(EMD)[11]
8
References [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp.
203–205, 1993.
[9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov. 2011.
[10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.
[11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.
[12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.
9
The Proposed Method
10
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands
by k-meansFind palm center
11
Hand Segmentation (I)
Find hands through color
Train skin-color[12], detect face[15], image filtering[16], color threshold
L*a*b color space b
Operate AND on the two images
Minimum depth (10cm)
12
Hand Segmentation (I)--Find hands through color
RGB image Depth image L*a*b color space where b = 2
L*a*b color space where b = 3
Skin color images after ANDoperation
Binary image of hand segmentation
13
Hand Segmentation (II)
Separate hands by k-means k=2
Assignment:
Update:
Threshold of distance between 2 clusters
14
Hand Segmentation (II)--Separate Hands By K-means
15
Hand Segmentation (III)
Find palm center Inscribed circle
Minimum inner distance
Maximum element of inner distances set
16
Finger Recognition (I)
Find convex hull Graham’s scan algorithm
P: the lowest y-coordinate
Sort in increasing order of angle
Point to point is left/right turn
Left-turn: O ; right-turn: X
17
Finger Recognition (II)
Detect fingertip and direction Fingers are long and narrow
Find an isosceles triangle with V V: Every vertex on the convex hull
Set a maximum threshold to the vertex angle
The direction vector is paralleled with the median length of an isosceles triangle
18
Finger Recognition
Blue point : cluster centroid Green point : palm centerRed points : fingertipsYellow curves : hand contourLong lines : finger directions Structures around the hand : convex hull
19
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands
by k-meansFind palm center
20
Gesture Representation
All information about hands Palm center location
Finger number
Fingertips location
Finger direction vectors
Gestures Rock-paper-scissors game
Drag images
Grasping, releasing
21
Experimental Results
22
Experimental Results
Use Kinect as input of depth and color images
The detection successful rate can reach 95%.
No matter the hand is horizontally or vertically placed.
23
24
Experimental Results
25
Experimental Results--Shadow Puppetry
26
Conclusion
27
Conclusion
Present a new method to detect hands’ positions and gestures
NO training, NO database
Future works Set a threshold to the distance between the palm center
and the fingers
Add additional sensor devices to overcome no palm detection
Shadow Puppetry project