gestix: a doctor-computer sterile gesture interface … a doctor-computer sterile gesture interface...
TRANSCRIPT
Gestix: A Doctor-Computer Sterile Gesture Interfacefor Dynamic Environments
Juan Wachs, Helman Stern, Yael EdanBen Gurion University of the Negev, Be’er Sheeva, Israel
Michael Gillam, Craig Feied, Mark Smith, Jon HandlerInstitute for Medical Informatics, Washington Hospital Center
Washington DC, US
Wachs et al. 2006 WSC 11 2
Presentation Overview
Introduction and MotivationGesture interfaces in Medicine“Gestix” - System OverviewDetecting and Tracking the HandOperation ModesConclusions
Wachs et al. 2006 WSC 11 3
Maintain Sterile Environments
Motivation: Sterile Environments
Study by Clorox
Toilets CleanerThan Keyboards
Toilets Cleaner Than Mice
Wachs et al. 2006 WSC 11 4
Hand Gestures in Medical Environment
• Access to critical information in hospitals, in case of epidemic crisis eruption, while maintaining total sterilization.
• Surgeon can control the different tools and visualization devices by simply making the appropriate gesture.
• Portable gesture recognition systems can be used to control home devices, and wheelchair transportation vehicles for the handicapped population.
• Sterile exploration and manipulation of high-quality images of the human anatomy, produced by 3D scanning techniques (MRI, CT).
Wachs et al. 2006 WSC 11 5
A robotic wheelchair based on the integration of human and environmental observations [Kuno et. al 2003]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 6
Interactive Manipulation of Real time Visualization from Medical Volume Data by using 2-Handed VR-Techniques [Köchy et al. 98]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 7
The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring [Starner et. al 2000]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 8
Staying Alive: A Virtual Reality Visualization Tool for Cancer Patients [Becker and Pentland 96]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 9
Facilitating User Interaction with Complex Systems via Hand Gesture Recognition [New et. al 2003]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 10
FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope [Nishikawa et. al 2003]
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 11
FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope (Cont’d)
Gesture Interfaces in Medicine
Wachs et al. 2006 WSC 11 12
Summary: Why Hand Gestures?
• Sterile
• Rapid Reaction
• Unencumbered
• Navigational Richness
• Noisy Environments
• Support of multimodal inputs
• Natural communication
Wachs et al. 2006 WSC 11 13
GESTIX: System Overview Screen Navigation MapHand Gesture Control
Wachs et al. 2006 WSC 11 14
System Overview - Architecture (Cont’d)
State machine for the gesture-based medical browser
Start
Calibration
Mode Detector
Stop
FlickGestures
Left Right Up Down
SweepGesture
Closer Further
Track SSI
Rotate
Browse Zoom Rotate
Wachs et al. 2006 WSC 11 15
Hand DetectionCalibration
The background is updated using ‘Background Differencing’
),(),()1(),( 1 jifjiBccjiBcc kk ∗+∗−= − ααWhen a significant change occurs, the background is not updated anymore, and the CDM is created
Wachs et al. 2006 WSC 11 16
Camshift Algorithm to track skin colorBuilding the color model
Hue
Sat
Sat
Hue
Wachs et al. 2006 WSC 11 17
Color Cue (The Back Projection Image)
Wachs et al. 2006 WSC 11 18
Motion Cue (Frame Differencing)
Wachs et al. 2006 WSC 11 19
Fusion of Color and Motion Cues
),(*)1(),(}).,(,1min{),( jipjidjipjiI kkkkkk αφα −+∗=
),( jiIk
kα
),( jipk
d
),( jikφ
Skin color probability (color cue) at frame k
Motion image (motion cue) at frame k
Amplifying factor
Motion assessment variable
Fused intensity image
Wachs et al. 2006 WSC 11 20
Operation Modes - Browsing
When a doctor wishes to browse the image database, he/she moves the hand rapidly out from a ‘neutral area’ to any of the 4 directions, and then back to the neutral area, hence evoking a ‘flick’ gesture.
Wachs et al. 2006 WSC 11 21
Switching to Zoom Mode
f1 f2 f3 f4
)f,f( jiα
)f,f( jiα
εααε ∗+≤−≤∗− 290290 )f,f()f,f( 2143
εαεα ∗≤∨∗≤ 5.1)(std5.1)(std )f,f()f,f( 2143
(3)
(4)
90
180
Frame number
Average angle
Instant angle
Hand Orientation – Major Axis
Wachs et al. 2006 WSC 11 22
Operation Modes – Zoom Mode
Wachs et al. 2006 WSC 11 23
Operation Modes - RotationStraight lines are detected using the Probabilistic Hough Transform and the straight sterile instrument is detected by Weighted Rule Based Algorithm
*
Wachs et al. 2006 WSC 11 24
Operation Modes – Rotation (Cont’d)
Wachs et al. 2006 WSC 11 25
Conclusions
‘Flicking’ gestures for interface control.Rotation of images are controlled by a straight sterile instrument type of objects.Dynamic gestures are detected by a color-motion fusion approachThe gesture recognition system was implemented in a sterile medical data-browser.
