real time visual body feedback & ir tracking in hmd based virtual environments using microsoft...
Post on 20-Dec-2015
216 views
TRANSCRIPT
Real Time Visual Body Feedback & IR Tracking in
HMD Based Virtual Environments Using
Microsoft Kinects
Speaker: Srivishnu ( Kaushik ) Satyavolu
Advisor: Dr. Pete Willemsen
Thesis Problem
Analysis of using multiple kinects to implement an
inexpensive IR based tracking system, in addition to real
time 3D visual body feedback and natural interaction to the
user in HMD based Virtual Environments.
Motivation
• Absence of real time ( self ) visual body feedback in existing HMD based VEs
• Difficulties to study the perceptions / interaction of multiple users simultaneously in HMD based VEs
Full-body Tracking possible only by highly expensive tracking systems & Motion Capture suits
Issues
• Camera Calibration
• RGB Depth Mapping
• 3D User segmentation, smoothing & representation
• IR Interference between multiple kinects
• Relative Orientation between multiple Kinects
• Tracking User Positions and/or pose
Camera Calibration & RGB Depth Mapping
Issues: Intrinsic Camera Calibration
Incorrect mapping between RGB and Depth images
Why is thisinteresting?: Accurate RGB to Depth Mapping
Solution: OpenCV Camera Calibration using Standard
Chessboard Recognition Techniques
3D User Representation
Issue: 3D Point Cloud representation
Why is this
interesting?: Real time Visual body feedback to the user
Approaches
used: Triangulation, RGB/ IR / Depth Background
Subtraction, Basic Filtering Techniques
IR Interference
Issue: Interference of IR between multiplekinects
Why is this
interesting?: For many reasons, which you will see shortly
Solution: Not as bad as expected
Unresolved at the moment
Relative Orientation between multiple kinects
Issue: Extrinsic Camera Calibration
Why is thisInteresting?: Relative Orientation of Multiple Kinects
Solutions:Manual Orientation
Automatic Calibration using OpenCV’s Chessboard Recognition Techniques
Others?
User TrackingIssues: Tracking User Position, Orientation & Pose
across large VR Lab spaces
Why is it Interesting?: Real time 3D Natural Interaction by User
across VR Lab spaces
Solution: IR Based Kinect Tracking System?
IR based Kinect Tracking System
• What is it for?– Track User Head Position and orientation ( possibly ) in VR Lab
Spaces
• How is it done?– Tracking an IR marker across a VR Space ( how large??? )
• Why do we need it?– Interference issues ( again?? ) with existing Tracking Sytems like
World Viz etc.– Extending OpenNI and Microsoft SDK pose estimation
techniques( effects of interference?? )
Current Approach
• Principle of locality
• Multiple Kinects ( interference??? ) over the network
• Mean Filter
• Interpolation???
Experiments & Results
Experiment #1: Evaluation of noise/jitter of kinect's
IR based position tracking over a large space
Results:
Standard Deviation: 0.001 – 0.003 % of the mean
Future Scope & Conclusion• Photorealisitc and real time 3D Self Representation
– 3D Visual Body Feedback to the User
• Real time 3D Natural Interaction of the User across VR lab spaces– Kinect Based IR tracking System in conjunction with
Microsoft / OpenNI pose estimation techniques
• Increased perception and level of presence in VEs– Yet to be verified, but it's quite possible
• Better understanding of HMD based VEs with multiple users
References
• Libfreenect
• Nicholas Burrus’ work ( RGBDemov0.4)
• Oliver Kreylos’ Kinect Viewer
• OpenCV
• And all the other websites from which I downloaded some of the images used in this presentation