Download - Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles
![Page 1: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/1.jpg)
Detecting and Avoiding Frontal Obstacles from a Monocular
Camera for Micro Unmanned Aerial Vehicles
1Robot Project
WakaWaka Group
Professor: Yvan Petillot
Team Members : H.Kidane , I.Sadek , M.Elawady
Heriot Watt University
School of Electrical and Physical Sciences
![Page 2: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/2.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 2
![Page 3: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/3.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 3
![Page 4: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/4.jpg)
Introduction
GoalDetecting and avoiding frontal obstacles using Ar-Drone2
Robot Project 4
![Page 5: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/5.jpg)
Introduction
Ar.Drone It is a rotating rigid structure with 6 degree of freedom. The two pair of rotors
rotate in different directions
Robot Project 5
HD Camera 720P , 30FPSVery Light and High
Resistance Plastic
Specific
Propeller
Ultrasound Sensor
Indoor Weight :420g
Price : $300
![Page 6: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/6.jpg)
• Applications:
MAVs play an important role in many applications (i.e.
search, monitoring, rescue, surveillance, etc)
- Able to maneuver rapidly and adequately.
- less dangerous for people.
- Provide real time data to the operator.
• Limitation:
- Limited payload to carry additional sensors.
- Depend on monocular camera.
- Obstacles can’t be observed directly using this camera.
Robot Project 6
Introduction
![Page 7: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/7.jpg)
Outline
• Introduction
• Related Work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 7
![Page 8: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/8.jpg)
Related Work
• Paper1: Learning Monocular Reactive UAV Control in Cluttered Natural EnvironmentsStephane Ross, Narek Melik-Barkhudarov, Kumar Shaurya , Shankar Andreas Wendel, DebadeeptaDey, J. Andrew Bagnell, Martial Hebert
• Paper2: First Results in Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned
Aerial VehiclesTomoyuki Mori , Sebastian Scherer
• Paper3: Autonomous quad rotor flight with vision-based obstacle avoidance in virtual environmentAydın Eresen, Nevrez Imamoglu, Mehmet Önder Efe
Robot Project 8
![Page 9: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/9.jpg)
Robot Project 9
Related Work-Paper1
The system observes a human expert driving the drone
Video Stream
Visual Features
Expert
Input
Unsupervised Learner
Control Command
http://robotwhisperer.org/bird-muri/
![Page 10: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/10.jpg)
Robot Project 10
Related Work-Paper1
Example shows the learning process where learner in this frame
gives wrong results (white line), while the expert provides the correct command
(red line).
![Page 11: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/11.jpg)
Robot Project 11
Related Work-Paper2
This method relies on the relative size change of an object in two
consecutive frames
![Page 12: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/12.jpg)
Robot Project 12
Related Work-Paper2
Position of obstacle
Confirm scale with template matching
Discard key-points (smaller or same size)
Discard mismatch (Distance threshold)
Matching in consecutive frames
Generate surf key points
![Page 13: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/13.jpg)
Robot Project 13
Related Work-Paper3
Take Snap Shot
Image Pre-processingGoal
Achieved
Object Detection
Generate PathYaw Angle
Landing
Controller
No
Yes
![Page 14: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/14.jpg)
Robot Project 14
Related Work-Paper3
• Image pre-processing: resizing and de-blurring
• Object detection: optical flow (Horn and Schunk)
Search
WindowTemplate
Google earth environment
![Page 15: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/15.jpg)
Robot Project 15
Related Work-Paper3
Result
![Page 16: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/16.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 16
![Page 17: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/17.jpg)
17
Methodology – Detection
Un-successful Works
Robot Project
![Page 18: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/18.jpg)
18
Methodology – Detection
Semi-Dense Optical Flow
Robot Project
![Page 19: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/19.jpg)
Outline
Check image de-blurring (variance of laplacian)
Robot Project 19
BlurredNot Blurred
Correct Incorrect
Not Blurred (7 Images) 6 1
Blurred (8 Images) 8 0
https://www.mathworks.co.uk/matlabcentral/fileexchange/27314-focus-
measure/content/fmeasure/fmeasure.m
![Page 20: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/20.jpg)
20
Methodology – DetectionBlock Diagram
Level 1
Robot Project
Image at
frame x
[1]
Image at
frame x+k
[2]
(Optional)
Pre-processing
image resizing
and sharpening
[1] Compute
symmetric
feature locations
within step range
Optical-flow
Algorithm
[Gunnar
Farneback]
Mismatch points
removal
[euclidean distance]
(Optional)
[2] Region of interest
(ROI) column selection
[25%]
[2] Split image into
five regions
[FL,NL,CN,NR,FR]
Calculate
average/median
euclidean distance
for each region
Find region with
maximum value
last 5 max ==
current max
&&
current max
>= threshold
yesObstacle
direction
[left/right]
No
No
obstacle
![Page 21: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/21.jpg)
21
Methodology – DetectionBlock Diagram
Level 0
Robot Project
Image at
frame x
Image at
frame x+k Obstacle
direction
[left/right]
No obstacle
Detection
Algorithm OR
![Page 22: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/22.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 22
![Page 23: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/23.jpg)
Robot Project 23
Takeoff
Fly Forward
Detected?
Process
video to detect obstacle
Fly sideways
Land/wait_joy_cmDestination/
joy_active?
Yes
Yes
No
No
Avoidance
![Page 24: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/24.jpg)
Robot Project 24
ROS driver for Parrot AR.Drone
Avoidance
• "ardrone_autonomy” developed in Autonomy Lab of Simon Fraser University
![Page 25: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/25.jpg)
Avoidance
• Information from Ar.Drone will be published in ardrone/navdatatopic
• ardrone/navdata
– Battery percent
– Drone state
– Orientations/tilt magnitudes
– pressure
– etc
Robot Project 25
Receiving data from AR.Drone
”ardrone_anatomy“
![Page 26: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/26.jpg)
Avoidance
• ROS camera interface topics to
capture Images/video from Drone
– ardrone/image_raw
– ardrone/front/image_raw
– ardrone/bottom/image_raw
Robot Project 26
Receiving data from AR.Drone
”ardrone_anatomy“
![Page 27: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/27.jpg)
Avoidance
• Drone will takeoff, land, or
emergency stop/reset by
publishing an Empty ROS
messages to the ff topics
– ardrone/ takeoff
– ardrone/land
– ardrone/reset
Robot Project 27
Sending commands to AR.Drone
![Page 28: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/28.jpg)
Avoidance
• To fly the Drone after takeoff,
publish a message of type
geometry_msgs::Twist to the
cmd_vel
• geometry_msgs::Twist expresses
velocity in free space broken into its
Linear and angular
Robot Project 28
Sending commands to AR.Drone
![Page 29: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/29.jpg)
Robot Project 29
Waka_Controller ardrone_driver
/cmd_vel
/ardrone/takeoff
/ardrone/land
/ardrone/reset
/ardrone/front/image_rawWaka_Image_Proce
Avoidance
Autonomous flying controller
![Page 30: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/30.jpg)
Robot Project 30
Avoidance
Integrating with Joystick
/joy
Waka_Controller ardrone_driver
/cmd_vel
/ardrone/takeoff
/ardrone/land
/ardrone/reset
/ardrone/front/image_rawWaka_Image_Proce
Joy_node
![Page 31: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/31.jpg)
Robot Project 31
Sideway velocity for 1s
Avoidance
geometry_msgs::Twist /cmd_vel
Forward Velocity
linear.x: 1m/s (move forward)
linear.y: 0
linear.z: 0
Obstacle in half left
linear.x: 0
linear.y: -2/s move right
linear.z: 0
Obstacle in half right
linear.x: 0
linear.y: 2m/s move left
linear.z: 0
![Page 32: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/32.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 32
![Page 33: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/33.jpg)
• Setup:
- Intel Core™ i7-3630 QM Processor
- Clock speed : 2.40 / 3.40 Turbo GHz
- 3rd level cache : 6 MB
- Running OS: Linux (Ubuntu)
Robot Project 33
Experiments
![Page 34: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/34.jpg)
Robot Project 34
Experiments
Control training
virtual obstacle
Online Obstacle
Avoidance test
Detection training
using offline video
![Page 35: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/35.jpg)
35Robot Project
Offline Online
Correct Incorrect Total Correct Incorrect Total
Indoor 7 5 12 7 3 10
Outdoor 8 4 12 - - -
Experiments
Results
![Page 36: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/36.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 36
![Page 37: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/37.jpg)
• Optical flow algorithm gives better detection results
comparing with feature-based algorithms
• Control part for avoidance reacts as expected
• Accuracy is reduced due to inaccurate measurement
of time to collusion
Robot Project 37
Conclusion
![Page 38: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/38.jpg)
Outline
• Introduction
• Related work
• Methodology–Detection
–Avoidance
• Experiments
• Conclusion
• Future Work
Robot Project 38
![Page 39: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/39.jpg)
39Robot Project
Multi-sensor data / multi-detectors for robust time-to-collision estimation
• Frontal camera with Optical flow is used
Optical flow comparisons across all frames
• One comparison at current frame
Find de-blurring kernel for wiener/lucy algorithm
• Neglect blurring images
Path planning and follow m-line to goal
• Fly forward
Future Work
![Page 40: Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned Aerial Vehicles](https://reader034.vdocument.in/reader034/viewer/2022042817/559c89391a28ab41488b4842/html5/thumbnails/40.jpg)
40Robot Project