Hand Movement Hand Movement RecognitionRecognition

By:By:Tokman NivTokman Niv

Levenbroun GuyLevenbroun Guy

Instructor:Instructor:Todtfeld AriTodtfeld Ari

Goals and objectivesGoals and objectives

Project goal was to create a system Project goal was to create a system that analyses colored glove that analyses colored glove movements and translate them into movements and translate them into speech.speech.

This project is based on the work This project is based on the work done by Alex Gluzman And Lior done by Alex Gluzman And Lior Neuman At spring 2004, who created Neuman At spring 2004, who created a system that detects the position of a system that detects the position of the colored glove.the colored glove.

System overviewSystem overview

The system is composed of the The system is composed of the following units:following units: Color video cameraColor video camera

Colored gloveColored glove

Double talkDouble talk

PC+Rio frame grabberPC+Rio frame grabber

System overview – basic System overview – basic operationoperation

The camera shoots the The camera shoots the glove.glove.

The data if grabbed by The data if grabbed by the rio FrameGrabber and the rio FrameGrabber and placed in memory.placed in memory.

Rio routines inform the Rio routines inform the program that there is a program that there is a new frame waiting, by new frame waiting, by calling OnDraw().calling OnDraw().

The image is processed.The image is processed. The result is transferred The result is transferred

to the DoubleTalk device to the DoubleTalk device via the computers COM via the computers COM port.port.

System’s user interfaceSystem’s user interface

Movement recognition Movement recognition flowflow

Movement is derived from a series of still Movement is derived from a series of still frames, for each we have colors centroids frames, for each we have colors centroids locations and the recognized positionlocations and the recognized position

For each frame, we compare the color’s For each frame, we compare the color’s location to the previous frame and extract location to the previous frame and extract the direction each color has moved.the direction each color has moved.These directions are the 8 prime direction, These directions are the 8 prime direction, no movement or missing color.no movement or missing color.

The list of direction, with the positions The list of direction, with the positions compose a movementcompose a movement

Y

es

NoN

o

Extracting color Extracting color locationslocations

(x, y, Color)Yes

Update:Sum of X values

Sum of Y values

Number of pixels with that color.

For every scanned pixel

Mark pixel as black

For each color

Number of pixels <

threshold ?

1

1

pixels

pixelsN

i Centroid nN n

P P P

Color match one

of the preset colors (in

HSV values)?

Mark color as missing

1 2( , )meas meask P Pdistance f

ExampleExample

Result : LEFTResult : LEFT

Calculating directionsCalculating directions Each color CM is compared to the CM in Each color CM is compared to the CM in

the previous frame.the previous frame. Δx and Δy are calculated.Δx and Δy are calculated. Conclude the direction.Conclude the direction.

Δx = -10Δx = -10

Recognition flowRecognition flow

Yes

New frame isready in Rio memory buffer

Process framefor color’s centroidslocation

Identify handposition

Calculate colordirection inreference to previous frame

30 frames without

movement?

Identifymovement

Start

Below is a regular main loop Below is a regular main loop iteration:iteration:

Low frame rateLow frame rate Scanning each frame is time/CPU consuming.Scanning each frame is time/CPU consuming. Low frame rate cause movement information Low frame rate cause movement information

lossloss Noise in frame samplingNoise in frame sampling

Camera shakes, lighting variesCamera shakes, lighting varies Non linear hand movementNon linear hand movement

Analyzing movement (directions list)Analyzing movement (directions list) Frame rate influence length of listFrame rate influence length of list List comparing is time consuming (affecting List comparing is time consuming (affecting

frame rate)frame rate)

Challenges of flowChallenges of flow

Hand movement

desired movement

To reduce scan time, only every 4To reduce scan time, only every 4thth pixel is scanned, until a color is found. pixel is scanned, until a color is found. Then scan is done on every pixel until Then scan is done on every pixel until color disappears. color disappears.

This allows for faster frame processing This allows for faster frame processing with still accurate CM calculations.with still accurate CM calculations.

HSV transformation is done during the HSV transformation is done during the scan.scan.

Pixels are transformed to HSV Pixels are transformed to HSV minimize lighting effects.minimize lighting effects.

Frame processing solutionFrame processing solution

http://en.wikipedia.org/wiki/HSV_color_space

Scanning a black line takes ¼ of the time since Scanning a black line takes ¼ of the time since we sample only every 4we sample only every 4thth pixel (most lines are pixel (most lines are black).black).

When we come across a colored pixel we scan When we come across a colored pixel we scan every pixel, so we won’t miss any of them, to every pixel, so we won’t miss any of them, to improve CM accuracy.improve CM accuracy.

Sometimes we might miss the edges of the Sometimes we might miss the edges of the color, since we don’t start to scan at the first color, since we don’t start to scan at the first appearance.appearance.

Frame processing solutionFrame processing solution

Noise filteringNoise filtering

First task is to distinguish between a real glove First task is to distinguish between a real glove movement and noise (camera shakes, light movement and noise (camera shakes, light changes).changes).

This is accomplished by defining a distance This is accomplished by defining a distance threshold (in pixels).threshold (in pixels).

This threshold can be set automatically at run This threshold can be set automatically at run time: 120 frames are collected while glove is time: 120 frames are collected while glove is still, and worst deviation between any two still, and worst deviation between any two frames is set as threshold.frames is set as threshold.

Result : LEFTResult : LEFT

Δy = 3Δy = 3

Δx = -10Δx = -10TH = 4TH = 4

Whenever a new CM is calculated, Whenever a new CM is calculated, we try to guess it’s position by time we try to guess it’s position by time derivation in x and y axis.derivation in x and y axis.

We than calculate the ‘real’ CM by:We than calculate the ‘real’ CM by:

sampledguessed CMWCMW )1(CMnew

Non-linear movement Non-linear movement solutionsolution

http://www.cs.unc.edu/~welch/kalman/

We collect the movements history until a We collect the movements history until a series of 30 no-movement frames are series of 30 no-movement frames are accumulated.accumulated.

Next, same adjacent movements are Next, same adjacent movements are collapsed into one, so in the end the history collapsed into one, so in the end the history contains only changes in direction. (The contains only changes in direction. (The Mozilla gesture recognizer uses this model)Mozilla gesture recognizer uses this model)

There is also a threshold for movements in There is also a threshold for movements in the same basic direction, below which they the same basic direction, below which they are discarded as noise.are discarded as noise.

In order to speed up the comparison In order to speed up the comparison process, we convert the collapsed list into a process, we convert the collapsed list into a decimal number by enumerating the decimal number by enumerating the direction.direction.

Movement recognition Movement recognition solutionsolution

http://blog.monstuff.com/archives/000012.html

Down = 1

Up = 2

Left = 3 321321

left left left Up-left up up up up down down

left up down

First the list is collapsedFirst the list is collapsed The new list is translated to a decimal The new list is translated to a decimal

number according to the enumeration number according to the enumeration The number is than compared against the The number is than compared against the

movements in the Database.movements in the Database.

Movement recognition - Movement recognition - exampleexample

SummerySummery

We have created a system that tracks and We have created a system that tracks and identifies glove movements, and translate identifies glove movements, and translate it into speech according to user defined it into speech according to user defined DB.DB.

Main features:Main features: An intuitive graphical user interface that allows An intuitive graphical user interface that allows

easy configuration and operation.easy configuration and operation. Data base support to store user’s movements Data base support to store user’s movements

and positions, with the ability to modify, add and positions, with the ability to modify, add and delete entries through the GUI.and delete entries through the GUI.

Calibration of environment: camera’s height, Calibration of environment: camera’s height, noise reduction etc.noise reduction etc.

Additional uses – UI demoAdditional uses – UI demo

This recognition engine can be used for This recognition engine can be used for other uses.other uses.

For example: turn the glove into an input For example: turn the glove into an input device that can replace the mouse.device that can replace the mouse.

possible use: possible use: Catheterization RoomCatheterization Roomsince the room must be since the room must be sterile regular IO devicessterile regular IO devices can’t be used. can’t be used.

http://www.clarontech.com/

Example of use: UIExample of use: UI

Added values – what we Added values – what we learnedlearned

Movement tracking in noisy Movement tracking in noisy environment.environment.

Challenges of real time image Challenges of real time image processingprocessing

MFC programming skillsMFC programming skills Windows IPC (Inter Process Windows IPC (Inter Process

Communication)Communication)

Ideas for future Ideas for future developmentdevelopment

Use USB camera, and MS speech SDK.Use USB camera, and MS speech SDK. Saving history of recognized movements Saving history of recognized movements

for more complex sentences.for more complex sentences. Dynamically setting different movement Dynamically setting different movement

threshold for each color.threshold for each color. During frame scanning, when encounter a During frame scanning, when encounter a

color pixel jump back and rescan.color pixel jump back and rescan. Using the number of frames in a continues Using the number of frames in a continues

directiondirection Use neural network for better recognition.Use neural network for better recognition.

http://www.codeproject.com/cpp/gestureapp.asp