sign language classification process by neural network

A Contour Based Approach to Classify A Contour Based Approach to Classify Hand Posture using Neural NetworkHand Posture using Neural Network

Presented byMd.Tunvir Rahman

ID:0704026

Supervised byAnik Saha

Lecturer , CSE CUET

2

Department of CSE, CUET

Motivation

Touch less interaction with devices require fast, robust method to classify hand posture.

Autonomous driving require to classify the hand gesture shown by traffic police or passengers.

Home appliances like TV, Microwave oven etc. need posture classification.

Giving command to a robot can be done by hand gesture which needs a good gesture classifier.

3


Previous Work and Limitation

In[6] template matching approach which require hand band in the hand to normalize the image.

In [1] orientation Histogram based approach some times map same posture in different class.

In [2] gesture classification by presence of number of fingers and their respective distance with palm center limit the number gesture to be classified

4


Goal

Classify gesture in an dynamic background.

No special marker in the hand.

Noise reduction from image frame.

Implementation of neural network as classifier.

Implement this approach to classify Bangla Sign character.

5


Our Proposed Methodology

Hand Region Segmentation

RGB Image

Preprocessing

Connected Component labeling and Noise Removal

Normalization and Contour Detection

Feature Extraction

Neural NetworkTraining Set

Classified Posture

Train

Test

6


Recognition System

Detected sign

7


ROI Detection

ROI Detection Based on Skin Color

Some unwanted region appears in the frame

8


Preprocessing

Erosion and Dilation on Binary image to smooth the image contour and remove small holes.

255

( , )( , ) if

d I i j ifO i j at least one neighbor is 255

at least one neighbor is 255 Dilation

9


Preprocessing

Erosion

255

( , )( , ) ife I i j ifO i j all 8 neighbors are 255

at least one neighbor is 0

10


Noise Reduction

Label the Connected Component using Flood Fill and Consider two big region containing maximum binary data. Other will be considered as noise.

Color Segmented Image After Removing Noise

11


Flood Fill Algorithm

2

P1 P2 p3 p4 p3 p4

2

p3 p4 P4 p5

2

p5 p6 p7

2 2

2 2

A Connected region label

by 2

12


Normalization

Hand Forearm follow an Non-increasing radius shape up to wrist of the hand .

Forearm part is unwanted for classification.

Contour pixel of Hand shape

13


Feature Extraction

90

0

Summing Up Number

of Pixel’s lie in this Angle

Total 19 histogram is extracted from the image

180

14


Feature Extraction

Each Bin Contain count of Contour pixel .

Taking ratio of bins count and pass this ratio as the feature vector to the neural network.

First train the network by feature(input) and response (output).

Then test gesture with the trained network.

15


F1

F2

1

2

3

N

5

2

1

Our Proposed Network

Hidden layer

F3

Fn

Input Output

16


1

X1

x2

w11= 3

w21= 4

w12= 6

w22= 5

w10= 1

w20= -6

w21= -1

w22= 1

1

0

w01= -3.93Input

Target

Neural Network and Back propagation

1

w02= 1

17


I1

1

I2

H1

H2

O1

X1

x2

-6

1

0

Input Target


Activation a=1*3+0*4+3*1*1=4

3

41

6

5

2

4

Output Y1=0.982

Activation a=0

Output Y2=0.50

-3.93

1 1

Output 0.51

Target-Output=0.49

O2 1

2

4-3.

Output 0.72

Target-Output=0.27

Total Error=0.49+0.27=0.76

18


I1

1

I2

H1

H2

O1

X1

x2

-6

1

0

Input Target

Calculating the Delta values for Output and Hidden Neuron

3

41

6

5

2

4

-3.93

1 1

O2 1

2

4-3.

out= out*(1-out)*(target-out)

o1 =0.51*(1-0.51)*(1-0.51)=0.1225

o2 =0.27*(1-0.27)*(1-0.27)=0.14

= out*(1-out)*W*oi

h11 =0.982*(1-0.982)*2*0.1225=0.0043h12 =0.982*(1-

0.982)*2*0.14=0.0049

h21 =0.51*(1-0.51)*4*0.1225=0.1225

h22 =0.51*(1-0.51)*4*0.14=0.139

19


I1

1

I2

H1

H2

O1

X1

x2

-6

1

0

Input Target


3

41

6

5

2

4

-3.93

1 1

O2 1

2

4-3.

Wij= η*Yi*jη= Learning Constant=0.1

o1 =0.1225

o1 =0.14

h11 =0.0043

h12 =0.0049

h21 =0.1225

h22 =0.139

w(03)= 0.1*1*0.0043=0.0004

1.0004

w (ij)new =wij(old)+w(ij)

20


I1

1

I2

H1

H2

O1

X1

x2

-5.987

1

0

Input Target


3.0004

41

6.0123

5

2.012

4.006

-3.918

1 1

O2 1

2.013

4.012-2.98

1.0004

21


I1

1

I2

H1

H2

O1

X1

x2

-5.987

1

0

InputTarget

Network response after Weight adjustment

3.0004

41

6.0123

5

2.012

4.006

-3.918

1 1

O2 1

2.013

4.012-2.98

1.0004

New OutH1=0.9820New OutH2=0.5063New OutO1=0.5214New OutO2=0.736

1-0.5214=0.4786

1-0.736=0.264

Total Error=0.4786+0.264

=0.7426In First Iteration Error reduced from 0.76 to 0.74

Iteration Continues until the desired error goal is achieved

22


Experimental Analysis

• Performance depends on the no of training set

• Train: Test ratio significantly effects successful classification.

• Defining 100 neurons in

hidden layer

requires around

450 epochs to reach

the error goal.

23


Performance Analysis

Train

SuccessfulClassification

Rate

10 Sample for each Sign character

24


Limitation

Background fully skin-colored the classification system fail.

Noise component is larger than the hand ROI.

Angular distortion cause the system failure.

25


Future Works

Shape based hand region segmentation can make the classification independent of background.

Dynamic hand gesture can be extracted from video and make the system user friendly.

26


References

[1] William T. Freeman and Michel Roth, “Orientation Histograms for Hand Gesture Recognition ” IEEE Intl. Workshop on Automatic Face and Gesture Recognition, Zurich, June ,2006

[2] S.M Hassan Ahmed Todd C Alexender, “Real Time static and dynamic hand gesture recognition for human computer Interaction”-Electrical Engineering, University of Miami, FL.

[3]Priyanka Mekala, “Real-time Sign Language Recognition based on Neural Network Architecture”, Florida International University, FL, U.S.A

[4] Klimis Symeonidis “Hand Gesture Recognition Using Neural Networks”, School of Electronic and Electrical Engineering, August 23, 2009.

[5]Bowden & Sarhadi ” Building Temporal models for Gesture Recognition” in preceding British Machine Vision Conference, pages 32-41,2002.

[6] Dr. Kaushik deb, Helena Parveen Mony & Sujan Chowdhury “Two Handed Sign Language Recognition for Bangla Sign Character using Cross Correlation” Global journal of Computer Science and Technology, Volume 12, Issue 3, February 2012.

27


Thanks