![Page 1: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/1.jpg)
Davide Scaramuzza
Are We Ready for Autonomous Drones?
1
My lab homepage: http://rpg.ifi.uzh.ch/
Publications: http://rpg.ifi.uzh.ch/publications.html
Software & Datasets: http://rpg.ifi.uzh.ch/software_datasets.html
YouTube: https://www.youtube.com/user/ailabRPG/videos
![Page 2: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/2.jpg)
Drone Market Today: 130B$
22
Inspection Agriculture
Transport Search and Rescue
Davide Scaramuzza University of Zurich
https://www.pwc.pl/pl/pdf/clarity-from-above-pwc.pdf
![Page 3: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/3.jpg)
Swiss Drone Industry Map
3https://droneindustry.ch/swiss-drone-industry-map/
![Page 4: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/4.jpg)
How are current commercial drones controlled?
Remote control
Requires line of sight or communication link
Requires skilled pilots
GPS-based navigation
Drone crash during soccer match, Brasilia, 2013
![Page 5: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/5.jpg)
Flyability - Inspection drone (remote controlled)
5
![Page 6: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/6.jpg)
Wingtra - Autonomous Mapping (GPS-based)
6
![Page 7: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/7.jpg)
Cargo Delivery (GPS-based)
7
Matternet & Swiss Post delivering blood samples between two hospitals in Lugano
https://www.post.ch/en/companycars/post-company-cars-ltd/media/2017/swiss-post-drone-to-fly-laboratory-samples-for-ticino-hospitals
![Page 8: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/8.jpg)
Does not work indoors
Even outdoors it is not a reliable service
Satellite coverage
Multipath problem
What is GPS if obstructed or just not available?
![Page 9: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/9.jpg)
9
![Page 10: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/10.jpg)
![Page 11: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/11.jpg)
How do we localize without GPS?
11
![Page 12: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/12.jpg)
Ultra-Wide-Band Localization (UWB)
12A Ledergerber, M Hamer, R D'Andrea,
A robot self-localization system using one-way ultra-wideband communication, IROS 2015
![Page 13: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/13.jpg)
Ultra-Wide-Band Localization (Verity Studios AG)
13
A Ledergerber, M Hamer, R D'Andrea,
A robot self-localization system using one-way ultra-wideband communication, IROS 2015
![Page 14: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/14.jpg)
How can Autonomous Drones fly without external sensors
(no GPS, no UWB)?
14
![Page 15: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/15.jpg)
LIDAR-based Localization
15
![Page 16: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/16.jpg)
Lidar-based Localization
Google car
![Page 17: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/17.jpg)
Lidar-based Localization
17J. Zhang and S. Singh.
Laser-visual-inertial Odometry and Mapping with High Robustness and Low Drift, Journal of Field Robotics. 2018
![Page 18: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/18.jpg)
Vision-based Localization
18
![Page 19: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/19.jpg)
Vision-based Localization
19
Camera
Phone PC
![Page 20: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/20.jpg)
How Does it Work ?
Image 1Image 2
Scaramuzza, Fraundorfer, Pollefeys, Lee, Meier, Siegwart, Chli, Achtelick, Weiss, et al., Vision-Controlled Micro Flying Robots, IEEE RAM’14
Visual Simultaneous Localization and Mapping(VSLAM)
![Page 21: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/21.jpg)
DARPA FLA Program (2015-2018)
21Mohta, Watterson, Mulgaonkar, Liu, Qu, Makineni, Saulnier, Sun, Zhu, Delmerico, Karydis, Atanasov, Loianno, Scaramuzza, Daniilidis, Taylor, Kumar
Fast, Autonomous Flight in GPS‐denied and Cluttered Environments, Journal of Field Robotics, 2018. PDF
![Page 22: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/22.jpg)
Is then Autonomous Navigation Solved?
22
![Page 23: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/23.jpg)
Davide Scaramuzza – University of Zurich – [email protected]
My Vision (10 years?): Fast, Lightweight, Autonomous
WARNING! There are 50 drones in this video but 40 are CGI and
10 are controlled via a Motion Capture System
LEXUS commercial, 2013
23
![Page 24: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/24.jpg)
24
Perception is the biggest challenge:
Challenging illumination
Motion blur at high speeds
Open Challenges for Vision-based Drones
![Page 25: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/25.jpg)
Artificial Intelligence
• Currently solved by Machine Learning, i.e., programming by examples: Show the computer what to do, without explaining how to do it!
• The robot programs itself to imitate the examples at best.
• Continuous self improvement via more data/experience.
25
![Page 26: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/26.jpg)
Deep Networks Can Help Cope with Inaccurate Models
[Giusti, RAL’16], [Smolyanskiy, IROS 2017], [Loquercio, RAL’18], [Kaufmann, CORL’18, 4% Oral Acceptance Rate]
![Page 27: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/27.jpg)
[Giusti, RAL’16], [Smolyanskiy, IROS 2017], [Loquercio, RAL’18], [Kaufmann, CORL’18]
Learning to Fly by Mimicing Cars and Bicycles
![Page 28: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/28.jpg)
Reconfigurable Drones
Falanga, Kleber, Mintchev, Floreano, Scaramuzza, An Autonomous Vision-based Foldable Quadrotor that can Squeeze and Fly, IEEE Robotics and Automation Letters 2018, under review
![Page 29: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/29.jpg)
Davide Scaramuzza – University of Zurich – [email protected]
Future: Autonomous Drone Racing?
WARNING! There are 50 drones in this video but 40 are CGI and
10 are controlled via a Motion Capture System
LEXUS commercial, 2013
WARNING! This drone flown is NOT autonomous; it is operated by a human pilot.
Human pilots take years to acquire the skills shown in this video.29
![Page 30: Are We Ready for Autonomous Drones? · Davide Scaramuzza –University of Zurich –sdavide@ifi.uzh.ch Future: Autonomous Drone Racing? WARNING! There are 50 drones in this video](https://reader031.vdocument.in/reader031/viewer/2022011912/5fa334fd51d87529f1217bdb/html5/thumbnails/30.jpg)