3d hand pose estimation a wrist camera via dorsum ... · presentation3 author: wu erwin created...
TRANSCRIPT
Back‐Hand‐Pose:3D Hand Pose Estimation for a Wrist‐worn Camera via Dorsum Deformation Network
Erwin WuPhD 1st StudentJSPS Research Fellow (DC1)Tokyo Institute of Technology
RGB Cam
Kris KitaniAssociate Professor,Carnegie Mellon UniversityResearch Fellow,University of Tokyo
Hideki KoikeProfessor,School of Computing,Tokyo Institute of Technology
Background
Introduction Motivation Approach Study Discussion Future Work
Touch Screen VR/ARGesture Control
Human beings get used to interact with object using their hands.
Motivation Approach Study Discussion Future Work
Limitations of Previous Work
Oculus Quest Hand Pose
Leap Motion
Glove Technologies Zhou et al. (CVPR2020)
Digits (UIST2012)
Camera
FingerInput (ISS2018)
Introduction
Related WorkCyclopsRing(Chan et al. UIST 2015)
Using ring‐like fisheye camera between the finger
Finger Angle‐Based Hand Gesture Recognition(Chen et al. Appl. Sci. 2018)
Using an elevated wrist‐worn camera to recognize finger angle
Motivation Approach Study Discussion Future WorkIntroduction
Do we really need to see the fingers?
Motivation Approach Study Discussion Future Work
Question
Introduction
Motivation Approach Study Discussion Future WorkIntroduction
Background
Camera ViewSome Commercial Products
Our Previous Work
Motivation Approach Study Discussion Future WorkIntroduction
Gesture Recognition from back hand image using depth camera.
OpisthenarWu et al. (UIST 2019)
Our Approach
Motivation Approach Study Discussion Future WorkIntroduction
Hardware Design
Motivation Approach Study Discussion Future WorkIntroduction
2.1mm 120° lens 30-FPS Camera
Image Preprocessing
Motivation Approach Study Discussion Future WorkIntroduction
Image Preprocessing
Motivation Approach Study Discussion Future WorkIntroduction
Network Architecture
Motivation Approach Study Discussion Future WorkIntroduction
Hand Representation
Motivation Approach Study Discussion Future WorkIntroduction
Demo
Motivation Approach Study Discussion Future WorkIntroduction
Study: Data Collection
Motivation Approach Study Discussion Future WorkIntroduction
User 1
Data Collection Environment
Ground Truth
Raw Data
Study: Data Collection
• Subject: 5 Male (age 25-32)
• Pose: 10 static gestures (ASL number)5 dynamic tapping (each finger)
• Length: 20FPS@30s * 5 sessions / gesture
• Number: 225,000 frames in total
Motivation Approach Study Discussion Future WorkIntroduction
User 1
Study: 3D Hand Pose Estimation Accuracy
Motivation Approach Study Discussion Future WorkIntroduction
Evaluation on each joint and finger:
Study: 3D Hand Pose Estimation Accuracy
Motivation Approach Study Discussion Future WorkIntroduction
Study: Different Lighting Condition
Motivation Approach Study Discussion Future WorkIntroduction
Study: Ablation Study
Motivation Approach Study Discussion Future WorkIntroduction
VGG16(RGB)
ResN18+LSTM (RGB)
ResN18+LSTM (TS)
Ours (TS)
Study: Grasp Recognition
Motivation Approach Study Discussion Future WorkIntroduction
Confusion Matrix of 50 test resultDifferent Grasp Types
Application
Motivation Approach Study Discussion Future WorkIntroduction
Summary
• Proposed a novel DorsalNet to reconstruct angle-basedhand pose from images of the back of the hand.
• Evaluations on 3D hand pose by comparing with severalbaselines and an ablation study.
• Evaluations on gesture and grasp recognition.
• Demonstrated several types of applications to show use-cases.
Motivation Approach Study Discussion Future WorkIntroduction
Future Work
• Implementation for full wrist actuation & rotation
• Solutions for completely dark environment
• More various subjects to study the generalization
• Embedded in real watches
Motivation Approach Study Discussion Future WorkIntroduction
