MACHINE VISION FOR INTELLIGENT
DRONES: AN OVERVIEW
APRIL 27TH 2016
KEVIN HEFFNER SAMUEL FOUCHER
RESEARCH ASSOCIATE, CRIM DIRECTEUR, VISION AND IMAGING TEAM
PEGASUS RESEARCH & TECHNOLOGIES CRIM
2
Outline
Introduction to UAVs
Intelligent Drones
Machine Vision Technologies
Example Applications
Trends and Challenges
3
noun: drone; plural noun: drones a male bee in a colony of social bees, which does no work but can fertilize a queen.
Source: Steven Zaloga, Unmanned Aerial Vehicles: Robotic Air Warfare 1917-2007, Osprey Publishing, 2008
In 1935, U.S. Adm. William H. Standley saw a British demonstration of the Royal Navy's DH 82B Queen Bee. Commander D Fahrney was tasked to develop something similar for the US Navy. "Fahrney adopted the name ‘drone’ in homage to the Queen Bee”
DRONE TERMINOLOGY
4
TYPES OF DRONES BY SIZE
UAV ACRONYMS
Unmanned Aerial Vehicle (UAV)
Micro Aerial Vehicle (MAV)
Miniature Aerial Vehicle (MAV)
Small UAS (SUAS)
Medium Altitude Long Endurance (MALE)
High Altitude Long Endurance (HALE)
Remotely Piloted Aircraft Systems (RPAS)
5
Range (miles)
Flight d
uratio
n (h
ou
rs) M
ax O
per
atin
g A
ltit
ud
e (
feet
)
1 10 100 500 1K 10K
64
32
16
8
4
2
1
40K
20K
10K
5K
1K
Maximum Weight (lbs)
1 10 50 100 1K 2K 5K 10K 20K
UAV CLASSIFICATIONS
HOW FAR, HOW HIGH, HOW LONG, HOW HEAVY ?
6
Range (miles)
Flight d
uratio
n (h
ou
rs) M
ax O
per
atin
g A
ltit
ud
e (
feet
)
1 10 100 500 1K 10K
64
32
16
8
4
2
1
40K
20K
10K
5K
1K
Maximum Weight (lbs)
1 10 50 100 1K 2K 5K 10K 20K
UAV CLASSIFICATIONS: SOME EXAMPLES
7
The majority of private sector applications utilize drones as sensing platforms
FUNCTIONS AND MISSIONS
Site inspection (sensing)
Energy sector (sensing)
Geomatics (sensing)
Firefighting (sensing)
Search & Rescue (sensing)
Surveillance (sensing)
Agriculture (sensing)
Agriculture (intervention)
Transport (packages)
Recreational
PRIVATE SECTOR
Intelligence gathering
Weather
Weapons
Search & Rescue
Medical Evacuation
Communications
Transport
Target practice
Decoy
MILITARY
9
Which rules apply to me ?
http://www.tc.gc.ca/media/documents/ca-standards/Info_graphic_-_Flying_an_umanned_aircraft_-_Find_out_if_you_need_permission_from_TC.pdf
START
NO
I have read and can meet the exemption conditions
for UAVs < 2 kg
I have read and can meet the exemption conditions for UAVs from2 to 25 kg
You don’t need permission,
but you must meet the exemption
conditions
You must apply for a
Special Flight Operations Certificate
You don’t need permission, but
you do have to fly safely
You don’t need permission, but you must meet the exemption
conditions AND notify Transport Canada with:
1.Contact info 2.UAV model 3.Description of operation 4.Geographic boundaries of operation
NO
It weighs > 35 Kg
NO
I use my aircraft for WORK or RESEARCH
YES
YES NO
It weighs < 2 Kg
NO YES YES NO
YES YES
It weighs > 25 Kg
CANADIAN UAV REGULATION
10
http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html
• Fly your drone during daylight and in good weather (not in clouds or fog). • Keep your drone in sight, where you can see it with your own eyes – not only
through an on-board camera, monitor or SmartPhone. • Make sure your drone is safe for flight before take-off. Ask yourself, for
example, are the batteries fully charged? Is it too cold to fly? • Know if you need to apply for a Special Flight Operations Certificate • Respect the privacy of others – avoid flying over private property or taking
photos or videos without permission.
CANADIAN UAV REGULATION
• Closer than 9 km from any airport, heliport, or aerodrome. • Higher than 90 metres above the ground. • Closer than 150 metres from people, animals, buildings, structures, or vehicles. • In populated areas or near large groups of people, including sporting events,
concerts, festivals, and firework shows. • Near moving vehicles, highways, bridges, busy streets or anywhere you could
endanger or distract drivers. • Within restricted and controlled airspace, including near or over military bases,
prisons, and forest fires. • Anywhere you may interfere with first responders.
Do
Don’t fly
11
http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html
Currently there is no requirement for a Pilot or Operator License. Transport Canada has communicated their intent to institute new regulatory requirements that will address licensing and training of drone operators for drones up to 25 kg.
These requirements also establish how the UAV should be marked and registered.
CANADIAN UAV REGULATION
13
DRONE EDUCATION
1. Make drones easier and safer to operate
2. To increase their autonomy and decrease human supervisory control requirements
3. So that they can perform complex functions
4. So they can collaborate with other systems (including drones)
Why do we want drones to become more intelligent ?
Making decisions
Reasoning
Prioritizing tasks
Detecting differences
Finding similarities
Learning from experience
Characteristics of Intelligence
14
*See Ingrand and Ghallab - http://homepages.laas.fr/felix/publis-pdf/aicom13.pdf
Cognitive Robotics – Breakdown of intelligence into functions
ROBOT INTELLIGENCE
Planning off-line prediction of feasible actions; Acting task execution; refines planned actions; Observing detects and recognizes; Monitoring comparison of predicted versus observed; Goal reasoning monitoring at mission level; Learning acquire, adapt and improve through experience;
DELIBERATION FUNCTIONS*
15
ENVIRONMENT
Task Execution
Low-level instructions Hardware Interface
Robot Platform
Sensory Functions
Motor Functions
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Low-level instructions Hardware Interface
Sensory Functions
Motor Functions
Robot Architecture Cognitive Robot Architecture
ROBOT ARCHITECTURES
16
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Low-level instructions Hardware Interface
Sensory Functions
Motor Functions
Symbolic reasoning
Sub-symbolic processing
Adjust speed to 40 knots. Change heading to 75 deg. …
Follow route A. Return to base. …
Robot Architecture Cognitive Robot Architecture
COGNITIVE ROBOT ARCHITECTURES
17
ENVIRONMENT
Deliberation
Robot Platform
Task Execution
Low-level instructions Hardware Interface
Symbolic reasoning
Sub-symbolic processing
Adjust speed to 40 knots. Change heading to 75 deg. …
Follow route A. Return to base. …
Robot Architecture Cognitive Robot Architecture
COGNITIVE ROBOT ABSTRACT MACHINE
CRAM Kernel
ROS Interface
Computable Predicates
Perception Navigation Manipulation Code OWL
LISP
SWI Prolog
CPL - CRAM Plans Belief State - Object designators - Entity designators - KnowRob knowledge base
KnowRob Reasoner
CPL Extension Modules
KnowRob Extension Modules
20
COMPUTER VISION
Object or place recognition Visual odometry 3D reconstruction Target tracking Anomaly detection
Many applications
OpenCV PCL ArrayFire …
Open-source libraries
The main goal of Computer Vision is to enable a computer to analyze, process and understand one or more images taken by a vision system.
Development of MV Applications has exploded recently thanks to the advent of high-res digital cameras along with increased computing power & machine learning.
21
PLATFORM Ubiquity of low-cost prosumer
drones
SENSORS Smarter, Smaller,
Integrated Processors
PERFECT STORM OF TECHNOLOGIES
Size Weight and Power Price
(low) SWaPP
COMPUTING RESOURCES
Embedded, cloud Machine Vision
Libraries
22
ON-BOARD COMPUTER VISION
The combination of computer vision and IMU data (altitude, acceleration, etc.) can improve the precision and robustness of drone navigation.
Vision-based solutions are interesting for small drones (< 2.0 kg) for indoor navigation, obstacle avoidance and other problems.
Navigation Path Planning Localization Geo-fencing
Environment GPS-denied Obstacle avoidance Swarming
Automatic Landing and takeoff Toward increasing autonomy
Mission critical aspects
Environment mapping Object detection Anomaly detection Video enhancement:
De-hazing Image stabilization Super-resolution Contrast enhancement
Data compression
Application related processing
23
SIMULTANEOUS LOCALIZATION AND MAPPING (SLAM)
Example of an application for an on-board vision system
Precise drone localization Environment mapping around the drone Sensing with a sonar, a laser or a camera Object and place recognition
Problems to solve
Motion Initial
position prediction
Image Keypoint
Extraction Matching
Revised Prediction
24
KEYPOINTS DETECTION
Robustness Appearance (illumination, occlusion, etc.) Viewpoint
Local and distinct description Quantity Low computation cost
Ideal properties
Keypoints are sailient points in an image
Harris SURF SIFT ORBE FAST
Many different techniques
Detection
Description
Matching
25
VISUAL SLAM
Camera + SLAM = monocular visual SLAM (vSLAM) exploits visual information
(keypoints)
RGB-D or stereo cameras directly minimize the photometric error
Matching between successive images Mapping
26
VISUAL SLAM FOR GPS-DENIED ENVIRONMENTS
https://www.youtube.com/watch?v=VWWvjSHZCNo
High precision intertial navigation
units are costly and heavy
Laser range finders (costly and
heavy)
Cameras (visual odometry)
Possible solutions
Precision: +/- 2.5m
Not accurate in urban environment
Not possible to use indoors
GPS navigation limitations
28
A LIST OF DRONE APPLICATIONS
Analysis of first 3136 exemption requests authorized by the FAA for small drones (<25 Kg)
29
NATURAL RESOURCE MANAGEMENT
RESEARCH APPLICATIONS
Source: Shahbazi, M., Théau, J. et P. Ménard (2014) Recent applications of unmanned aerial imagery in natural
resource management. GIScience & Remote Sensing 51
Based on review of 150 articles about the use of UAV imagery
30
Improve water use
Pest management
Weed-identification
Irrigation and water distribution
Manned aircraft: ~$1,000/hour
Satellite Imagery: $$$
Multispectral sensors:
• Visible (R,G,B)
• Near-Infrared (NIR)
• Thermal Infrared (TIR)
• Resolution < 1 cm, > 12 Mpixels
Hyperspectral: 640 bands
LIDAR
AGRICULTURE APPLICATIONS
31
PRECISION AGRICULTURE
Source: http://www.pole-pegase.com/documents/Actualites/6_-_presentation_Exametrics.pdf Source : http://www.icv.fr/conseil-viticulture-oenologie/oenoview
Re
solu
tio
n (
m)
10 1 0.1 0.01 0.001
$$$
$$
$
32
AGRICULTURE APPLICATIONS
Information gathering
NDVI
Multispectral
Hyperspectral
Intervention
Irrigation control
Spraying
Harvesting
AGRIBOTICS APPICATIONS
33
GLACIER MASS BALANCE APPLICATION
Source: Christophe Kinnard, Atelier sur la sécurité civile, AQT’2015
Orthomosaic (10 cm) + MNE
~US$10,600
34
ENERGY SECTOR INSPECTION & SITE SECURITY
Crack detection in dams and reactors
A security drone that will chase down intruders.
36
HOTSPOT DETECTION USING OPTIMAL SEARCH PATTERN
Pegasus Research and Technologies Research Project
Rapid deployment Low-cost lightweight IR sensors Optimized search path Applications for
For missing persons Remnant forest fire detection Security applications
HOTSPOT DETECTION
37
OPEN-FIELD HOTSPOT DETECTION TRIAL
FOR LOCALIZED SEARCH AND RESCUE
1. Determine presence and approximate location of candidate “missing person”.
2. High resolution camera is then pointed at location and fed to SAR team.
3. System queries operator for next step 4. Future capability for automatic object
recognition.
HOTSPOT DETECTION
39
ROOF INSPECTION APPLICATION
1. Automated data collection a. 3D Model construction images b. High-resolution texture images
2. 3D Model construction 3. Roof extraction and zone detection 4. High-resolution texture generation 5. Inspection annotations 6. Report generation
ROOF INSPECTION PIPELINE
40
(define-policy start-execution()
"MISSION STARTED ?"
(:init (format t "Initializing policy to start the execution ~%") t)
(:check (setq execution_param (roslisp:get-param "/execute_mission"))
(print execution_param)
(cond ((eql execution_param 1))))
(:recover (format t "Running recovery mechanisms ~%"))
(:clean-up (format t "Running clean up ~%")))
(def-top-level-cram-function execute_misson ()
"MISSION EXECUTION"
(roslisp:start-ros-node "execute_mission")
(mav:init-ros-mav "hummingbird")
(roslisp:set-param "/execute_mission" 0)
(roslisp:set-param "/pause_execution" 0)
(roslisp:set-param "/roof_detected" 0)
(setf foo 1)
(setf counter 1)
(with-failure-handling
((policy-check-condition-met (f)
(print "starting execution") (return)))
(with-named-policy 'start-execution ()
(loop while (= counter 1)
do(print "waiting for execution")
(setq counter 1)
(sleep 3))))
(with-failure-handling
((policy-check-condition-met (f)
(print "test condition met")
(return)))
(with-named-policy 'my-policy ()
(defparameter *transform-listener* (make-instance 'cl-tf:transform-listener))
(sleep 5)
(setq trans (cl-tf:lookup-transform *transform-listener* :source-frame "/hummingbird/ground_truth" :target-frame "/world"))
AUTOMATED IMAGE COLLECTION USING DELIBERATION
48
Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016
TRENDS AND CHALLENGES
49
TRENDS AND CHALLENGES
Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016
50
?
Drone
? ?
?
TRENDS AND CHALLENGES
Airworthiness considerations
Platform health monitoring & failure prediction
Fault detection & failure recovery
Intelligent operator aids
Operator training
Navigation warnings and notifications
Obstacle avoidance
Night-time operations
Drone Safety
Rogue drones
Drone pirates
Drone data protection (Cybersecurity)
Privacy
Drone Security
51
Personal Drones
TRENDS AND CHALLENGES: DRONE HYPE CYCLE ?
Agrib
otics
Cin
em
a
Ge
om
atics
Pu
blic Safety
Ene
rgy Secto
r
Site In
spe
ction
52
http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html
Advances in remote sensing, embedded processing and cloud computing technologies have created opportunities for new applications. The availability of low-cost prosumer drones adds another dimension of opportunity. Currently most drones are flying sensors. Adding intelligence to drones can improve safety and allow for drones to participate in data gathering and processing as part of an information ecosystem, e.g. Internet of Things (IOT). Regulation and safety aspects will be main factors in the adoption and practical use of drones for commercial purposes.
CONCLUSIONS
WWW.CRIM.CA
Suivez-nous
Dialoguez avec nous
Suivez-nous
#CRIM_ca
wwwCRIMca
Tous droits réservés © 2016 CRIM. 405, avenue Ogilvy, bureau 101, Montréal (Québec) H3N 1M3/514 840-1234/1 877 840-2746
Kevin Heffner Research Associate, CRIM
President, Pegasus Research & Technologies
Samuel Foucher Équipe Vision et Imagerie
CRIM – Centre de recherche informatique de Montréal
Principal partenaire financier Le CRIM est un centre de recherche appliquée en TI qui développe, en mode collaboratif avec ses clients et partenaires, des
technologies innovatrices et du savoir-faire de pointe, et les transfère aux entreprises et aux organismes québécois afin de les
rendre plus productifs et plus compétitifs localement et mondialement. Le CRIM dispose de quatre équipes de recherche en TI
de calibre mondial, d’un centre de tests et d’interopérabilité considéré comme une référence neutre au Québec ainsi qu’un
centre de formations de pointe en TI. Le CRIM œuvre principalement dans les domaines des interactions et interfaces personne-
système, de l’analytique avancée et des architectures et technologies avancées de développement et tests. Détenteur d’une
certification ISO 9001:2008, son action s’inscrit dans les politiques et stratégies pilotées par le ministère de l'Économie, de
l'Innovation et des Exportations (MEIE), son principal partenaire financier.