interactive dialogue technique based computer vision with palm tracking

INTERACTIVE DIALOGUE TECHNIQUE BASED COMPUTER VISION WITH PALM TRACKING

GIA MUHAMAD AGUSTA, NANDANG SUNANDAR, QURROTUL AINI

FACULTY OF SCIENCE AND TECHNOLOGY, STATE ISLAMIC UNIVERSITY SYARIF HIDAYATULLAHJAKARTA - INDONESIA

• Introduction about Interaction and Palm Tracking• Haar Cascade• Controlling Mouse Pointer Method• Controlling Cursor Key Method• Experimental Result• Conclusion and Future Work

PRESENTATION SUMMARY

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

• The needed of HCI Development

INTRODUCTION

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

• Why Palm Tracking?

INTRODUCTION

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

• Why Haar Cascade– Gaussian Mixture Model– SVM (Support Vector Machine)– Pyramidal Lucas Kanade– ANN (Artificial Neural Network)

• Haar Cascade– Integral Image– Haar-Like Feature– Ada Boost Algorithm– Cascade Classifier

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Computational Time(Andrew King, Survey of of Methods for Face Detection, 2003)

• Integral Image

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

𝐷=( 𝐴+𝐵+𝐶+𝐷 )− ( 𝐴+𝐵 )− ( 𝐴+𝐶 )+𝐴

• Haar-Like Feature

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

1 3 3 2

1 4 4 51 7 5 2

2 3 4 6

1 3 2 2

1 3 5 81 2 3 1

3 5 3 1

1 4 7 9

2 9 16 233 17 29 38

5 22 38 53

6 26 44 60

7 30 53 778 33 59 84

11 41 70 96

• Ada Boost Algorithm

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Weak or basic classifierLearning rateFinal classifier

𝑓 (𝑥 )=∑𝑡=1

𝑇

𝛼 𝑡h𝑡 (𝑥)

• Cascade Classifier

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Filter 1

Filter n

BukanTelapak Tangan

...

Telapak Tangan

Filter 2

Telapak Tangan

Citra Digital

Telapak Tangan

Telapak Tangan

Bukan Telapak Tangan

Bukan Telapak Tangan

Bukan Telapak Tangan

• How we do it• Training and Running(Detecting)

HAAR CASCADE

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Filter 1

Filter n

BukanTelapak Tangan

...

Telapak Tangan

Filter 2

Telapak Tangan

Citra Digital

Telapak Tangan

Telapak Tangan

Bukan Telapak Tangan

Bukan Telapak Tangan

Bukan Telapak Tangan

Training Detecting

.xml (Knowledge Based)Open and Close Palm

• Take x and y coordinate from detected palm (Hook)

CONTROLLING MOUSE POINTER

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Start

Is Object detected

?

Take coordinate from dectected

object

Use coordinate as parameter to

controlling pointer

End

CONTROLLING CURSOR KEY

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Start

End

Menghitung Frame

Jika Frame pada posisi 0

Jika Frame pada posisi 3

Set posisi frame 0 Mencatat koordinat akhir

Menghitung jarak Euclide

Distance >=40

Koordinat x awal < x akhir

Left Cursor Key Direction

Right Cursor Key Direction

Set Waiting Frame = 15 frame

Jika Waiting frame = 0

Mencatat koordinat awal

Count distance from last frame to current frame with Euclidean Distance

𝑑=√¿¿

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Haar Cascade Detection

Palm Position

ExperimentPalm Position

Straight ± 250 ± 450 ± 900 Reverse1 ü ü ü û û2 ü ü û û û3 ü ü ü û û4 ü û û û û5 ü ü û û û6 ü û û û û7 ü ü û û û8 ü û û û û9 ü ü û û û

10 ü ü û û ûAverage

(%)100 70 20 0 0

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Haar Cascade Detection

Palm Position

Tegak ± 25° ± 45° ± 90° Terbalik0

20

40

60

80

100

120

100

70

20

0 0

Measurement with Different Positions

Palm Position

Aver

age

(%)

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Haar Cascade Detection

Distance Measurement

ExperimentDistance (cm)

40 60 80 100 120 140 160 180 200 220 240

1 ü ü ü ü ü ü ü ü ü ü û2 ü ü ü ü ü ü ü ü ü ü û3 ü ü ü ü ü ü ü ü ü û û4 ü ü ü ü ü ü ü ü ü û û5 ü ü ü ü ü ü ü ü ü û û6 ü ü ü ü ü ü ü ü ü ü û7 ü ü ü ü ü ü ü ü ü û û8 ü ü ü ü ü ü ü ü û û û9 ü ü ü ü ü ü ü ü ü û û

10 ü ü ü ü ü ü ü ü ü û ûAverage

(%)100 100 100 100 100 100 100 100 90 30 0

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Haar Cascade Detection

Distance

40 60 80 100 120 140 160 180 200 220 2400

20

40

60

80

100

120

100 100 100 100 100 100 100 100

90

30

0

Measurement Result through Distances

Distances

Aver

age

(%)

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Haar Cascade Detection

1 37 73 109145 181 217253289325 361397 433 469 505541 577613649 685721757 793 829 865 901937 9730.00

100.00

200.00

300.00

400.00

500.00

600.00

700.00

Processing Time of Palm Detection

Frame Number

Proc

essin

g Ti

me

(ms)

𝐹𝑃𝑆=1000𝑚𝑠139,5 =𝟕 ,𝟏𝟔

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Timing Process of Palm Detection Mouse PointerDistance (cm) Detection Left Click Function

40 ü ü60 ü ü80 ü ü

100 ü ü120 ü ü140 ü ü160 ü ü180 ü û200 ü û220 ü û240 û û

Controlling Process

EXPERIMENTAL RESULT

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Timing Process of Palm DetectionControlling Process Cursor Keys

Distances (cm) DetectionCursor Keys Function

Left Right

40 ü ü ü60 ü ü ü80 ü ü ü

100 ü ü ü120 ü ü ü140 ü ü ü160 ü ü ü180 ü ü ü200 ü û û220 ü û û240 û û û

CONCLUSION

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

The results of a series test (position angle and distance between user's palm and screen) shows that the more upright position of hand movement, the detection results would be better.

Tracking process can run on 7.16 fps at maximum distance 200 cm with accuracy rate 90.9%.

The left click event (mouse pointer) enables to function at a maximum distance 160 cm with accuracy reaching 63.6%, while left and right cursor keys will be function at maximum distance 180 cm with accuracy 72.7%.

FUTURE WORK

GiaMuhammad | Singapore, January 11th, 2014International Conference on Computer Science and Human Computer Interaction (ICCSHCI) 2014 | Changi Village Hotel

Use Kalman Filter method for soflty mouse movement.

Adding more positive and negative image sample data training to increase it accuration.

Impementing a whole system with a good interface and add some option for a better tracking (brightness, contrast, etc.)

THANK YOU