looking at people and image-based localisation roberto cipolla department of engineering research...
TRANSCRIPT
![Page 1: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/1.jpg)
Looking at people and
Image-based LocalisationRoberto Cipolla
Department of Engineering
Research team http://www.eng.cam.ac.uk/~cipolla/people.html
![Page 2: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/2.jpg)
1. Real-time hand detection and tracking
![Page 3: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/3.jpg)
Why is it hard?
• Highly articulated object, 27 model parameters
• Shape variation and self-occlusions
• Unreliable point features
• Ambiguities in single viewlead to multi-modal distributions (local minima)
![Page 4: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/4.jpg)
Why is it hard?
• Background clutter
• Potentially fast motion
• Lighting changes
• Partial / full occlusion
![Page 5: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/5.jpg)
A Solved Problem?
3D tracking, 6/7 DOF• Model: 3D quadrics• Cost Function:
Edges or colour-edges • Tracking: Unscented
Kalman filtering• Single or dual view• Single hypothesis
filter, no recovery strategy
![Page 6: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/6.jpg)
A Robust Tracker
• Should work in scenes with complex backgroundand varying illumination– Important: Cost function design– Optimization strategy
• Should handle multi-modality– Examples: Particle filters, multi-hypotheses filters
• Should have a recovery strategy when track is lost– Trigger search algorithm
![Page 7: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/7.jpg)
3D Pose Recovery
3D hand model constructed from cones and ellipsoids Contour projection, handling self-occlusions 27 motion parameters
![Page 8: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/8.jpg)
Hierarchy of classifiers
![Page 9: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/9.jpg)
Likelihood : Edges
Edge Detection Projected Contours
Robust EdgeMatching
Input Image 3D Model
![Page 10: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/10.jpg)
Chamfer Matching
Input image Canny edges
Distance transform Projected Contours
![Page 11: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/11.jpg)
Likelihood : Colour
Skin Colour ModelProjected Silhouette
Input Image 3D Model
Template Matching
![Page 12: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/12.jpg)
Tree-based bayesian filtering
![Page 13: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/13.jpg)
Matching Multiple Templates
Use tree structure to efficiently match many templates (>50,000) Arrange templates in tree based on their similarity Traverse tree using breadth-first search, several ‘active’ leaves possible
Search TreeGrid-based partitioning of parameter space
![Page 14: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/14.jpg)
Bayesian-Tree
• The search-tree is brought into a Bayesian framework by adding the prior knowledge from previous frame.
• The Bayesian-Tree can be thought as approximating the posterior probability at different resolutions.
State space partitioning Estimation of posterior pdf
![Page 15: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/15.jpg)
Experiments
Global Motion3D motions limited to hemisphere
Dynamics: First-order Gaussian process
3 level tree with 16,000 templates at leaf level
5 scales, divisions of 15 degrees in 3D rotation and
divisions of 10 degrees in image plane rotation
Translation search at 20, 5, 2-pixel resolution
![Page 16: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/16.jpg)
Tracking Results
![Page 17: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/17.jpg)
Tracking Results
![Page 18: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/18.jpg)
Experiments
Finger Articulation
• Opening and closing of thumb and fingers approximated by 2 parameters
• Global motion restricted to smaller range, but still with 6 DOF
• 35,000 templates at the leaf level
![Page 19: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/19.jpg)
Opening and closing
![Page 20: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/20.jpg)
Hand detection system
![Page 21: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/21.jpg)
Ongoing work
Large number of templates requiredExamples shown here show only constrained motionNumber of templates required for fully articulated motion?
Tracking rates at 5 fps to 0.2 fps For 400 - 35,000 templates (on a 2.4 GHz Pentium IV)
Error introduced by geometric model No palm deformation, no skin deformation, no arm model
![Page 22: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/22.jpg)
Detecting people
![Page 23: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/23.jpg)
2. Building 3D models of cities
![Page 24: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/24.jpg)
Trumpington Street Data
![Page 25: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/25.jpg)
Camera pose determination
![Page 26: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/26.jpg)
3D reconstruction
![Page 27: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/27.jpg)
Reconstruction texture mapped
![Page 28: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/28.jpg)
3. Where am I?
![Page 29: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/29.jpg)
Image-based localisation
......
![Page 30: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/30.jpg)
Image-based localisation
![Page 31: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/31.jpg)
Image-based localisation
……
![Page 32: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/32.jpg)
Image-based localisation
![Page 33: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/33.jpg)
Image-based localisation
![Page 34: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/34.jpg)
Image-based localisation
![Page 35: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/35.jpg)
Image-based localisation
![Page 36: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/36.jpg)
Image-based localisation
![Page 37: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/37.jpg)
Image-based localisation
![Page 38: Looking at people and Image-based Localisation Roberto Cipolla Department of Engineering Research team cipolla/people.html](https://reader036.vdocument.in/reader036/viewer/2022062500/5697bfb51a28abf838c9d643/html5/thumbnails/38.jpg)
Summary and deliverables
• Realtime hand detection in clutter
• 3D models from uncalibrated images
• Image-based localisation for augmented reality