![Page 1: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/1.jpg)
Mobile 3D Scanning and Applications Prof. Dr. Marc Pollefeys ETH Zurich
17 March, 2015 http://www.inf.ethz.ch/personal/marc.pollefeys/
![Page 2: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/2.jpg)
3D Scanning
![Page 3: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/3.jpg)
Time-of-flight
X
d =c
2Dt
Mesa Imaging’s SwissRanger
new MS Kinect
PMDTec
![Page 4: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/4.jpg)
3D -> 2D image projection
C X
x L
![Page 5: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/5.jpg)
2D -> 3D ?
C x
l’
X?
x’
X
C’
L’ Triangulation
L
![Page 6: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/6.jpg)
Structured light
P x
l’ x’
X
C’
L’
L
Replace one camera with projector/known laser pattern
PrimeSense
Mantis Vision
![Page 7: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/7.jpg)
(Pollefeys et al. ICCV98, VSMM98, 3DIM99, IJCV04, ...)
![Page 8: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/8.jpg)
Turning 2D street-side video into 3D models Street-Side Video Real-Time Stereo (Pollefeys IJCV08;
Gallup DAGM10; …)
real-time processing of video (30fps on PC, leveraging GPU)
![Page 9: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/9.jpg)
…
![Page 10: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/10.jpg)
Joint 3D reconstruction and class segmentation
joint reconstruction and segmentation (ground, building, vegetation, stuff)
(Haene et al CVPR13, …)
Building Ground Vegetation Clutter
reconstruction only (isotropic smoothness prior)
![Page 11: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/11.jpg)
(Lee et al CVPR13; Heng et al. ICRA14; Haene et al…)
cameras as main sensors: no lasers, no GPS omnidirectional visual simultaneous localization and mapping
3D mapping for autonomous driving robust image-based localization
depth from motion stereo live obstacle maps
![Page 12: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/12.jpg)
Computer Games
![Page 13: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/13.jpg)
Body tracking
S
S1 S2
regression tree
Randomized forest (Shotton et al.)
based on simple depth difference tests at each node
![Page 14: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/14.jpg)
Raw Kinect output: Color + Depth = RGBD
http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect
![Page 15: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/15.jpg)
3D Video with Kinect
![Page 16: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/16.jpg)
![Page 17: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/17.jpg)
Kinect-based 3D avatar scanning
CVPR 2014 demo (Medioni et al. USC)
![Page 18: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/18.jpg)
Sense 3D Scanner
![Page 19: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/19.jpg)
Home Scanning
(now Apple)
![Page 20: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/20.jpg)
Tablet + Depth Sensor
Structure Sensor
![Page 21: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/21.jpg)
Tablet + Depth Sensor
Low power (<1W) models for tablet and for PC
depth from 2 (or 1) cameras + (un)structured laser pattern
![Page 22: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/22.jpg)
![Page 23: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/23.jpg)
ETH Zurich as Project Tango partner
Some of our contributions…
![Page 24: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/24.jpg)
Today‘s mobile phones
Mobile phones are great for computational photography
• High-resolution digital camera
• Computing power (CPU+GPU+...)
• Additional sensors (accelerometers, gyroscopes, ...)
• Storage
![Page 25: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/25.jpg)
3D reconstruction services in the cloud
Catch http://www.arc3d.be/
![Page 26: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/26.jpg)
![Page 27: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/27.jpg)
![Page 28: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/28.jpg)
Recording cultural heritage
![Page 29: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/29.jpg)
(Delaunoy et al. CVPR14)
![Page 30: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/30.jpg)
3D Scan and Print
![Page 31: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/31.jpg)
3D Portraits
![Page 32: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/32.jpg)
The Selfie Authentication
Company®
+ =
• Use of individual smartphones is the only way to
massively conquer the identity management market
• Smartphones now embed enough CPU power for 3D face
computation
• 3D live-face identification extremely hard to hack (no
picture …)
![Page 33: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/33.jpg)
Metaio’s ikea catalogue AR
captured with mobile app
Towards 3D capture + augmented reality
Microsoft HoloLens
Future?
![Page 34: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/34.jpg)
Fully autonomous vision-based MAV
sensing: video cameras + IMU full on-board processing indoor and outdoor operation obstacle avoidance, mapping and exploration no laser, no GPS, no network
More on PixHawk: http://pixhawk.ethz.ch
(Fraundorfer et al. IROS12 best paper finalist)
(Meier ICRA11, ICRA15,...)
also Open Source software and hardware initiative
![Page 35: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/35.jpg)
Interactive Navigation of casually captured videos
Collection of videos of the same event from different angles
Flickr/Paul J Coles
![Page 36: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/36.jpg)
Navigation in space and time
Interactive Navigation of casually captured videos
![Page 37: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/37.jpg)
4D Video
Interactive viewer, more results & datasets available at: http://cvg.ethz.ch/research/unstructured-vbr/
![Page 38: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/38.jpg)
Conclusion
• 3D scanning capabilities will become ubiquitous
• Full 6DOF localization of devices in environment
– Games
– Navigation
– Augmented reality
• Will enable tight interaction between physical and digital world
Live demos @ Swiss Pavilion (Hall 9, Stand A28)
![Page 39: Mobile 3D Scanning and Applications - Deutsche Messe AGfiles.messe.de/abstracts/62032_CGC2015_ETH_Pollefeys.pdf · Mobile 3D Scanning and Applications . Prof. Dr. Marc Pollefeys](https://reader031.vdocument.in/reader031/viewer/2022013015/5b3317917f8b9ab5728d9610/html5/thumbnails/39.jpg)
Thank you!