04/13/2023 1

A Unified Framework for 3D Hand Tracking

Rudra PoudelJose A S FonsecaJian J ZhangHammadi Nait-Charif

ISVC 2013

04/13/2023Next Introduction 2

Overview1. Introduction2. Applications3. Related Work4. Skin Colour Detection5. Hand Region Segmentation6. Hand Pose Estimation7. 3D Hand Tracking8. Conclusions

04/13/2023Continue … 3

1. Introduction• Aim: detect 3D

position of hand joints• Task in details:

o Hand detection (skin colour, hand shape)

o Joints detectiono Estimate 3D positions of the

joints

• Problems: appearance similarity, occlusion, random motions, 27 degree-of-freedom/dimensions

04/13/2023Next Applications 4

1. Introduction

Overview of the system

04/13/2023 5

2. Applications

Next Related Work

• Human machine/computer interaction- TV remote controller

• Sign language recognition• Gamming• Health and Medicine• Many more …

04/13/2023Continue … 6

3. Related Work• Computer vision is

holy grail of Artificial Intelligent/Robotics

• 3D Hand tracking deals with three major problems- detection, recognition, tracking

• Research started two decades back

Starner et al. (PAMI, 1998)

04/13/2023Next Skin Color Detection 7

3. Related Work• Oikonomidis, I., Kyriazis, N., Argyros, A.A.:

Efficient model-based 3d tracking of hand articulations using kinect. In: BMVC. (2011): template matching so computationally very expensive

• Keskin, C., Kirac, F., Kara, Y., Akarun, L.: Real time hand pose estimation using depth sensors. In: ICCV Workshops. (2011) 1228–1234: needed extra occlusion handling technique, hasn’t utilizes temporal information

04/13/2023 8

4. Skin Colour Detection

Next Hand Region Segmentation

• Learned two skin and non-skin histograms using- 4,700 skin and 9,000 non-skin images

• Manually labelled pixels is nearly 1 billion

• Includes all lighting conditions and races

04/13/2023 9

5. Hand Region Segmentation

Next Hand Pose Estimation

• Kinect depth image • Segmented hand using skin colour, depth and hand at previous frame

04/13/2023Continue … 10

6. Hand Pose Estimation

Learned hand parts classifier using 450 thousands artificial images

Example of hand parts detection

04/13/2023Continue … 11

6. Hand Pose Estimation

• Regression of joint positions using hough voting technique

04/13/2023Continue … 12

6. Hand Pose Estimation

Comparison: our vs. state-of-the-art Keskin (2011)

04/13/2023 13

6. Hand Pose Estimation

Example of hand pose estimation

Next 3D Hand Tracking

04/13/2023 14

7. 3D Hand Tracking

Next Conclusion

• Tracking using- hand joints estimation at current frame (t), temporal information and kinematic constraints

• 3D hand tracking demo: http://www.youtube.com/watch?v=xqyfWWlAnVI

04/13/2023 15

8. Conclusion• Depth sensor Kinect: RGB and depth images• Skin colour detection• Hand segmentation using skin colour, depth and

hand position at previous frame• Hand joints regression• 3D hand tracking using joints detection, temporal

information and kinematic constraints

• Used techniques- Naive Bayes for skin detection, Random Forest for joints detection, Mean-shift for joint vote aggression, Markov Random Fields for tracking

Next Questions

04/13/2023 16

Questions ?

Thank you !!!

9th International Symposium on Visual Computing