knowledge systems lab jn 1/15/2016 facilitating user interaction with complex systems via hand...
DESCRIPTION
Knowledge Systems Lab JN 1/15/2016 Motivation Gesturing is a natural form of communication Interaction problems with the mouse –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouseTRANSCRIPT
JN 05/03/23 Knowledge Systems Lab
Facilitating User Interaction with Complex Systems via
Hand Gesture Recognition
MCIS DepartmentKnowledge Systems Laboratory
Jacksonville State University
Joshua R. New, Erion Hasanbelliu, andMario Aguilar
JN 05/03/23 Knowledge Systems Lab
Outline
• Motivation• System Architecture• Implementation Overview• Proposed Approach• Demonstration• Future Directions
JN 05/03/23 Knowledge Systems Lab
Motivation
• Gesturing is a natural form of communication
• Interaction problems with the mouse– Have to locate cursor– Hard for some to control (Parkinsons or
people on a train)– Limited forms of input from the mouse
JN 05/03/23 Knowledge Systems Lab
Motivation (2)
• Interaction Problems with the Virtual Reality Glove– Reliability– Always connected – Encumbrance
JN 05/03/23 Knowledge Systems Lab
System Architecture
StandardWeb Camera
Rendering
User InterfaceDisplay
Hand
M
ovem
ent
User
Gesture Recognition
System
ImageCapture
Update Object
Image Input
JN 05/03/23 Knowledge Systems Lab
Implementation Overview• System:
• 1.6 Ghz AMD Athlon• OpenCV and IPL libraries (from Intel)
• Input:• 640x480 video image• Hand calibration measure
• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in response to
gesturing
JN 05/03/23 Knowledge Systems Lab
Implementation Overview (2)• Hand Calibration Measure:
• Max hand size in x and y orientations in # of pixels
JN 05/03/23 Knowledge Systems Lab
Implementation Overview (3)
Saturation Channel Extraction (HSL space):
Original ImageOriginal Image Hue
Lightness
Saturation
JN 05/03/23 Knowledge Systems Lab
Proposed Approach
JN 05/03/23 Knowledge Systems Lab
Proposed Approach (2)
JN 05/03/23 Knowledge Systems Lab
Proposed Approach (3)
)(*19.0 HandSizeYHandSizeXRadius
• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses
• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)
• Total fingers is number of fingers minus 1 for the hand itself
JN 05/03/23 Knowledge Systems Lab
Proposed Approach (4)
System Runtime:
• Current time – 41 ms for one image from camera
• Processing Capability on 1.6 Ghz Athlon:• 24 fps
ProcessSteps
Time (ms)Athlon
XP 1900(1.6 Ghz)
1) Extract Sat Channel 9
2) Threshold 3
3) Connected Contour Fill
14
4) Centroid 2
5) Segment Hand From Arm
9
6) Refined Centroid 4
7) Count Number of Fingers
0
Total Time 41
JN 05/03/23 Knowledge Systems Lab
Demonstration
SystemConfiguration
SystemGUI Layout
JN 05/03/23 Knowledge Systems Lab
Demonstration (2)Gesture to Interaction Mapping
Number of Fingers:
2 – Roll Left3 – Roll Right
4 – Zoom In5 – Zoom Out
JN 05/03/23 Knowledge Systems Lab
Demonstration (3)
JN 05/03/23 Knowledge Systems Lab
Demonstration (4)
JN 05/03/23 Knowledge Systems Lab
Future Directions
• Optimization• Calibration Phase• Defining Hand Orientation• Learning System• Interface Extensions
For additional information, please visit http://ksl.jsu.edu.