detecting and avoiding frontal obstacles from a monocular camera for micro unmanned aerial vehicles

Detecting and Avoiding Frontal Obstacles from a Monocular

Camera for Micro Unmanned Aerial Vehicles

1Robot Project

WakaWaka Group

Professor: Yvan Petillot

Team Members : H.Kidane , I.Sadek , M.Elawady

Heriot Watt University

School of Electrical and Physical Sciences

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 2

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 3

Introduction

GoalDetecting and avoiding frontal obstacles using Ar-Drone2

Robot Project 4

Introduction

Ar.Drone It is a rotating rigid structure with 6 degree of freedom. The two pair of rotors

rotate in different directions

Robot Project 5

HD Camera 720P , 30FPSVery Light and High

Resistance Plastic

Specific

Propeller

Ultrasound Sensor

Indoor Weight :420g

Price : $300

• Applications:

MAVs play an important role in many applications (i.e.

search, monitoring, rescue, surveillance, etc)

- Able to maneuver rapidly and adequately.

- less dangerous for people.

- Provide real time data to the operator.

• Limitation:

- Limited payload to carry additional sensors.

- Depend on monocular camera.

- Obstacles can’t be observed directly using this camera.

Robot Project 6

Introduction

Outline

• Introduction

• Related Work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 7

Related Work

• Paper1: Learning Monocular Reactive UAV Control in Cluttered Natural EnvironmentsStephane Ross, Narek Melik-Barkhudarov, Kumar Shaurya , Shankar Andreas Wendel, DebadeeptaDey, J. Andrew Bagnell, Martial Hebert

• Paper2: First Results in Detecting and Avoiding Frontal Obstacles from a Monocular Camera for Micro Unmanned

Aerial VehiclesTomoyuki Mori , Sebastian Scherer

• Paper3: Autonomous quad rotor flight with vision-based obstacle avoidance in virtual environmentAydın Eresen, Nevrez Imamoglu, Mehmet Önder Efe

Robot Project 8

Robot Project 9

Related Work-Paper1

The system observes a human expert driving the drone

Video Stream

Visual Features

Expert

Input

Unsupervised Learner

Control Command

http://robotwhisperer.org/bird-muri/

Robot Project 10

Related Work-Paper1

Example shows the learning process where learner in this frame

gives wrong results (white line), while the expert provides the correct command

(red line).

Robot Project 11

Related Work-Paper2

This method relies on the relative size change of an object in two

consecutive frames

Robot Project 12

Related Work-Paper2

Position of obstacle

Confirm scale with template matching

Discard key-points (smaller or same size)

Discard mismatch (Distance threshold)

Matching in consecutive frames

Generate surf key points

Robot Project 13

Related Work-Paper3

Take Snap Shot

Image Pre-processingGoal

Achieved

Object Detection

Generate PathYaw Angle

Landing

Controller

No

Yes

Robot Project 14

Related Work-Paper3

• Image pre-processing: resizing and de-blurring

• Object detection: optical flow (Horn and Schunk)

Search

WindowTemplate

Google earth environment

Robot Project 15

Related Work-Paper3

Result

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 16

17

Methodology – Detection

Un-successful Works

Robot Project

18

Methodology – Detection

Semi-Dense Optical Flow

Robot Project

Outline

Check image de-blurring (variance of laplacian)

Robot Project 19

BlurredNot Blurred

Correct Incorrect

Not Blurred (7 Images) 6 1

Blurred (8 Images) 8 0

https://www.mathworks.co.uk/matlabcentral/fileexchange/27314-focus-

measure/content/fmeasure/fmeasure.m

20

Methodology – DetectionBlock Diagram

Level 1

Robot Project

Image at

frame x

[1]

Image at

frame x+k

[2]

(Optional)

Pre-processing

image resizing

and sharpening

[1] Compute

symmetric

feature locations

within step range

Optical-flow

Algorithm

[Gunnar

Farneback]

Mismatch points

removal

[euclidean distance]

(Optional)

[2] Region of interest

(ROI) column selection

[25%]

[2] Split image into

five regions

[FL,NL,CN,NR,FR]

Calculate

average/median

euclidean distance

for each region

Find region with

maximum value

last 5 max ==

current max

&&

current max

>= threshold

yesObstacle

direction

[left/right]

No

No

obstacle

21

Methodology – DetectionBlock Diagram

Level 0

Robot Project

Image at

frame x

Image at

frame x+k Obstacle

direction

[left/right]

No obstacle

Detection

Algorithm OR

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 22

Robot Project 23

Takeoff

Fly Forward

Detected?

Process

video to detect obstacle

Fly sideways

Land/wait_joy_cmDestination/

joy_active?

Yes

Yes

No

No

Avoidance

Robot Project 24

ROS driver for Parrot AR.Drone

Avoidance

• "ardrone_autonomy” developed in Autonomy Lab of Simon Fraser University

Avoidance

• Information from Ar.Drone will be published in ardrone/navdatatopic

• ardrone/navdata

– Battery percent

– Drone state

– Orientations/tilt magnitudes

– pressure

– etc

Robot Project 25

Receiving data from AR.Drone

”ardrone_anatomy“

Avoidance

• ROS camera interface topics to

capture Images/video from Drone

– ardrone/image_raw

– ardrone/front/image_raw

– ardrone/bottom/image_raw

Robot Project 26

Receiving data from AR.Drone

”ardrone_anatomy“

Avoidance

• Drone will takeoff, land, or

emergency stop/reset by

publishing an Empty ROS

messages to the ff topics

– ardrone/ takeoff

– ardrone/land

– ardrone/reset

Robot Project 27

Sending commands to AR.Drone

Avoidance

• To fly the Drone after takeoff,

publish a message of type

geometry_msgs::Twist to the

cmd_vel

• geometry_msgs::Twist expresses

velocity in free space broken into its

Linear and angular

Robot Project 28

Sending commands to AR.Drone

Robot Project 29

Waka_Controller ardrone_driver

/cmd_vel

/ardrone/takeoff

/ardrone/land

/ardrone/reset

/ardrone/front/image_rawWaka_Image_Proce

Avoidance

Autonomous flying controller

Robot Project 30

Avoidance

Integrating with Joystick

/joy

Waka_Controller ardrone_driver

/cmd_vel

/ardrone/takeoff

/ardrone/land

/ardrone/reset

/ardrone/front/image_rawWaka_Image_Proce

Joy_node

Robot Project 31

Sideway velocity for 1s

Avoidance

geometry_msgs::Twist /cmd_vel

Forward Velocity

linear.x: 1m/s (move forward)

linear.y: 0

linear.z: 0

Obstacle in half left

linear.x: 0

linear.y: -2/s move right

linear.z: 0

Obstacle in half right

linear.x: 0

linear.y: 2m/s move left

linear.z: 0

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 32

• Setup:

- Intel Core™ i7-3630 QM Processor

- Clock speed : 2.40 / 3.40 Turbo GHz

- 3rd level cache : 6 MB

- Running OS: Linux (Ubuntu)

Robot Project 33

Experiments

Robot Project 34

Experiments

Control training

virtual obstacle

Online Obstacle

Avoidance test

Detection training

using offline video

35Robot Project

Offline Online

Correct Incorrect Total Correct Incorrect Total

Indoor 7 5 12 7 3 10

Outdoor 8 4 12 - - -

Experiments

Results

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 36

• Optical flow algorithm gives better detection results

comparing with feature-based algorithms

• Control part for avoidance reacts as expected

• Accuracy is reduced due to inaccurate measurement

of time to collusion

Robot Project 37

Conclusion

Outline

• Introduction

• Related work

• Methodology–Detection

–Avoidance

• Experiments

• Conclusion

• Future Work

Robot Project 38

39Robot Project

Multi-sensor data / multi-detectors for robust time-to-collision estimation

• Frontal camera with Optical flow is used

Optical flow comparisons across all frames

• One comparison at current frame

Find de-blurring kernel for wiener/lucy algorithm

• Neglect blurring images

Path planning and follow m-line to goal

• Fly forward

Future Work

40Robot Project