vision-based autonomous control and navigation of a uavchaow17/research/urop.pdfobject tracking....
TRANSCRIPT
Undergraduate Research Opportunity Program(UROP) Project Report
Vision-based Autonomous Control and Navigation of a UAV
By
Wang Chao
Department of Computer Science
School of Computing
National University of Singapore
Nov 2014
Project No: U148150Advisors: Dr. Somchaya Liemhetcharat and Asst. Prof. Kian Hsiang LowDeliverables:
Report: 1 Volume
Abstract
In this report, we present the design and implementation of a real-time, vision-based naviga-tion system for an Unmanned Aerial Vehicle (UAV). The navigation system is integrated withautonomous flight control system, visual servoing, and visual detection of marker and object(a hula hoop). Based on the autonomous control system and visual servoing, the navigation isable to be done from an arbitrary initial position and orientation. Finally, the report concludesby presenting the flight trial performance, conclusions, and future work.
Keywords:Unmanned Aerial Vehicle, Autonomous control, Vision-based navigation
Implementation Software and Hardware:Parrot AR.Drone 2.0
Acknowledgement
This project is made possible through the help and support from everyone, including: my
advisors, parents and friends. Especially, please allow me to dedicate my acknowledgment of
gratitude toward the following significant advisors:
First, I would like to thank my advisor Prof. Low Kian Hsiang. Without his agreement,
this project will not be made possible. He kindly gives me valuable advice and makes me keep
consistent in this whole project.
Second, I would like to thank my co-advisor Dr. Somchaya Liemhetcharat for his most
support and encouragement. Without his scientific guidance and contributions, I would not be
able to finish this project. Moreover, he kindly reads my report and o↵ered invaluable detailed
advice.
Finally, I sincerely thank to my parents and friends, who provide moral support. Without
all of them the product of this project would not be possible.
List of Figures
3.1 Parrot AR.Drone 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.1 Drone manoeuvres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2 Performance of the Proportional Controller . . . . . . . . . . . . . . . . . . . . . 94.3 Performance of the PID Controller . . . . . . . . . . . . . . . . . . . . . . . . . . 114.4 UAV’s Control Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.1 Visual Servoing Marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.2 Blobs and Bounding boxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.3 Calculation of the relative world coordinate with marker . . . . . . . . . . . . . . 155.4 The state transition diagram for marker tracking . . . . . . . . . . . . . . . . . . 16
6.1 UAV is aligned with the hula hoop . . . . . . . . . . . . . . . . . . . . . . . . . . 176.2 Fish-eye lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.3 FOV increased after using fish-eye lens . . . . . . . . . . . . . . . . . . . . . . . . 186.4 Chessboard for calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.5 Distorted Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.6 Undistorted Image Using OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . 206.8 Image processing results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.9 The state transition diagram for hula hoop detection and flying through . . . . . 24
7.1 Orientation Indication Marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.2 Orientation Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.3 UAV and ground robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
8.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2 The state transition diagram for the final experiment . . . . . . . . . . . . . . . . 31
iv
List of Tables
7.1 Hoop angle detected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
v
Table of Contents
Title i
Abstract ii
Acknowledgement iii
List of Figures iv
List of Tables v
1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Related Work 4
3 UAV Platform 6
4 UAV Autonomous Control 84.1 Proportional Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 PID Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5 Visual Servoing Using Vertical Camera 135.1 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.2 Marker Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.3 Marker Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.4 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6 Hula hoop Detection and Flying Through 176.1 Usage of Fish-eye Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.3 Image Undistortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.4 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.5 Position Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226.6 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7 Hula hoop Detection While Not Aligned 257.1 Ellipse Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.2 Ground Robot Assistance Using Marker . . . . . . . . . . . . . . . . . . . . . . . 26
vi
8 Experiment Results 298.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2 UAV Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
9 Conclusion And Future Work 32
References 33
vii
Chapter 1
Introduction
In recent years, unmanned aerial vehicle (UAV) has become a major focus of robotics researches,
since it can extend human’s capabilities in a variety of areas especially for military application
like search-and-rescue and surveillance, or civilian application like transportation. Among all
kinds of UAVs, quadcopter is more stable and easier to control. For example, it is easier to
maintain a hovering position with a quadcopter than with a helicopter or airplane. We choose
quadcopter as a platform to design and implement an autonomous flight control system and
vision-based navigation system.
1.1 Motivation
Typical UAVs are controlled by humans and experiences are needed but still have low precision.
The missions of autonomous UAVs require flying at low speed to follow a path or to track an
object of interest and take a series of actions.
Imagine a scenario that there are some tunnels. Inside the tunnels are full of unknown
dangers. Since the ground environment inside the tunnels are unstructured, the best choice
would be dispatching a UAV to explore it. The UAV takes o↵ from a ground robot and starts
tracking it. Then ground robot will navigate the UAV to the entrance of the tunnel. Upon
arrival, the UAV will scan for the entrance and fly into the tunnel after confirmation of the
entrance using its on-board vision algorithm. Inside the tunnel, wireless signals are unreliable
1
and there may not be a consistent signal for a human operator to control the UAV. Therefore,
the UAV will perform autonomous flight using vision-based navigation system and complete a
series of missions. Finally, it flies out from the exit of the tunnel, then the ground robot will
navigate it to another tunnel and so on.
As you can see from the scenario, accurate autonomous flight control and vision-based
navigation are necessary to perform such tasks.
1.2 Challenge
UAVs are highly sensitive to control inputs. For example, the mechanical structurer of quad-
copter comprises four rotors attached to the four ends. To maintain a hover, the speed of each
rotor need to be same. Even a small disturbance can cause drifting and rotations. Feedback
is critical to stable the autonomous flight. As a result, a closed loop control system is needed
with low latency and precise prediction of future outputs.
UAVs can move quickly. Small and mid-sized quadcopters can accelerate in the range of 0.5
- 0.7g and can exhibit 40-60� per second angular velocity under normal operating conditions
(Amidi, 1996). To keep up with the quadcopters high degree of maneuverability, an on-board
vision system must be able to sample and process camera images at high frequency. On-board
image processing must be performed at frame rate (30 Hz) or higher for e↵ective vision-based
object tracking.
UAVs are always moving which can cause motion blur and sensor data is noisy. To make
the UAVs stable enough, some techniques must be used to filter noisy data and get usable data.
1.3 Contribution
Our UAV flight control system is fully autonomous. It is accurate and stable with high flexibility
as well. Our vision-based navigation system is fully on-board with fast and real-time image
processing (30 FPS). Furthermore, our system works both indoor and outdoor without relying
on any other external positioning devices like GPS and motion capture systems like VICON.
2
Moreover, a wide-angle lens is mounted on the front camera of the UAV which is an innovative
attempt. It successfully improves the FOV of our vision system and make it more reliable and
practical. All of this make our system promising and able to be developed further.
1.4 Report Organization
This report is divided into nine chapters. Chapter 2 presents related work that has been
done in autonomous control systems and vision-based navigation systems. Some issues are also
addressed. Chapter 3 provides some background information of the UAV platform we use.
Chapter 4 presents the design and implementation of the UAV autonomous control system.
Chapter 5 presents a visual servoing system using image processing and autonomous control.
Chapter 6 presents our navigation system and algorithm that allows the UAV to autonomously
detect a hula hoop and fly through it.
Chapter 7 addresses detection issues while the UAV and the hula hoop are not aligned. A
solution is provided as well.
Chapter 8 presents the final flight experiments using the vision-based autonomous control
and navigation system. Results are discussed and evaluated.
Chapter 9 presents conclusions and future work. Future work includes multi-UAV coordi-
nation and motion planning.
3
Chapter 2
Related Work
In recent years, unmanned aerial vehicle (UAV) and micro aerial vehicle (MAV) have been an
active area of research. They have been used for military applications but also are useful for
many civilian applications such as terrain and utilities inspection, disaster monitoring, environ-
mental surveillance, search and rescue, and tra�c surveillance. However, most of them are still
semi-autonomous and guided by human who is not on board. There are lots of challenges to
turn an UAV into fully autonomous control.
The control of UAV during autonomous flights relies on knowledge of variables like position,
velocity and orientation, which can be partly calculated using information provided by on-board
inertial sensors. However, the drift of inertial sensors leads to errors during time-discrete integra-
tion, making a steadily accurate estimation of the absolute pose nearly impossible. Therefore,
an absolute reference is needed.
An early autonomous navigation system for a model-scale helicopter (the Hummingbird)
was developed at Stanford University (Conway, 1995). The unique feature of this system was
the sole use of GPS as the navigation sensor replacing the Inertial Measurement Unit, which is
conventionally favored as the primary navigation sensor. However, indoors, the GPS will not be
available, where a visual tracking system could be applied. In (Gurdan, Stumpf, Achtelik, Doth,
Hirzinger, & Rus, 2007) a position controller for a quadcopter was implemented using an optical
tracking system by VICON in the Holodeck lab at MIT. This intelligent space is nevertheless
very expensive and not transportable. A notable vision-based technique used in autonomous
4
helicopter control, the visual odometer (Amidi, 1996), provides accurate navigational informa-
tion such as position and velocity which is then combined with inertial measurements. However,
outdoors, it still needs GPS for positioning.
While several techniques have been applied for vision-based control of UAVs, none of them
have shown an fast, fully on-board vision-based control and positioning system. Our approach,
presented here, di↵ers from prior approaches in three ways. First, our vision system is real-time
with low latency image processing (30 FPS). Hence, the UAV positioning can be done very fast
and accurately. Second, the front camera vision system has a FOV of 178� which can easily
detect object in a large range. Third, our autonomous control system works both indoors and
outdoors without relying on any other external positioning devices like GPS.
5
Chapter 3
UAV Platform
Figure 3.1: Parrot AR.Drone 2.0
Parrot AR.Drone is a radio controlled flying quadcopter built by the French company Parrot.
The drone is designed to be controlled by mobile or tablet operating systems such as the
supported iOS or Android within their respective apps or the uno�cial software available for
Windows Phone, Samsung BADA and Symbian devices.
Version 2.0 is the successor to the original drone, the AR.Drone 2.0 was unveiled at CES
Las Vegas 2012. A lot of improvements was made to its functionality. Technical specifications
are as follows:
• 1GHz 32 bit ARM Cortex A8 processor
6
• Linux 2.6.32
• 1GB DDR2 RAM at 200MHz
• Wi-Fi b g n
• 3 axis gyroscope 2000�/second precision
• 3 axis accelerometer +-50mg precision
• 3 axis magnetometer 6�/ precision
• Pressure sensor +/- 10 Pa precision
• Ultrasound sensors for ground altitude measurement
• Front camera: 720p sensor with 93�/ lens, recording up to 30 FPS
• Vertical camera: QVGA sensor with 64�/ lens, recording up to 60FPS
Thanks to the AR.Drone 2.0 on-board sensors, take-o↵, hovering, trimming and landing are
completely automatic. Some other basic control and automatic stabilization features are also
integrated. The AR.Drone 2.0 Library is also provided as an open-source library with high
level APIs to access the drone. However, the SDK provided is designed to run on an external
computer, but we adapted it to run on the drone. Our application is developed based on the
SDK and running on the drone as well.
7
Chapter 4
UAV Autonomous Control
Basically, manoeuvres are obtained by changing pitch, roll, and yaw angles of the AR.Drone
2.0.
Figure 4.1: Drone manoeuvres
The APIs of controlling pitch, roll, and yaw angles are provided by the SDK. Besides, some
other basic manoeuvres’ APIs are also provided including take-o↵, hovering, trimming, and
landing. However, these controls are too low-level to achieve autonomous flight. For example, if
the UAV finds a target ahead at one meter, the UAV need to move forward one meter precisely.
However, adjusting pitch angle directly is not su�cient to obtain the required performance.
As a result, some high-level controllers are needed for advanced manoeuvres. We consider two
frequently used controllers, namely Proportional Controller and PID Controller.
8
4.1 Proportional Controller
Proportional Controller is the most intuitive control method which is to adjust pitch angle
directly proportional to the distance to target. For example, if the UAV is far away from the
target, it can tilt a larger pitch angle so that it can move forward with large acceleration. While
it is approaching the target, the distance becomes smaller and smaller. At this time, it should
decrease accelerating rate. Hence, it needs tilt a smaller pitch angle. When it reach the target,
the pitch angle should be zero which means no acceleration. The experiment result is shown in
Figure 4.2.
0 2 4 6 8 10
0
20
40
60
80
Time(s)
Position(cm)
Move forward 50cm
target
UAV
Figure 4.2: Performance of the Proportional Controller
The manoeuvre being tested is to move forward 50 centimeters. The world coordinate was
updated using odometry from IMU sensor data. The experiment result shows that at first, the
UAV was approaching the target quickly. However, after it reached 50cm it still moving forward
but slow down. After it stopped, it started moving backward to 50cm but overshoot again. The
position of UAV is keep fluctuating around 50cm although the gap is getting smaller. It can be
seen that to make itself stop at 50cm will take a long time. Therefore, a more precise controller
is needed.
9
4.2 PID Controller
A proportional-integral-derivative controller (PID Controller) is a closed loop control based on
feedback mechanism. A PID controller calculates an error value as the di↵erence between a
measured process variable and a desired setpoint. The controller attempts to minimize the
error by adjusting the process through use of a manipulated variable.
u(t) = MV (t) = K
p
e(t) +K
i
Zt
0
e(⌧)d⌧ +K
d
d
dt
e(t) (4.1)
where Kp
is Proportional gain, a tuning parameter. Ki
is Integral gain, a tuning parameter.
K
d
is Derivative gain, a tuning parameter. e is the di↵erence between desired value and current
value. t is Time or instantaneous time (the present). ⌧ is Variable of integration; takes on
values from time 0 to the present t.
The proportional term produces an output value that is proportional to the current error
value. The proportional response can be adjusted by multiplying the error by a constant K
p
,
called the proportional gain constant. The Proportional Controller introduced in Chapter 4.1
is a PID controller with only proportional term. A high proportional gain results in a large
change in the output for a given change in the error. If the proportional gain is too high,
the system can become unstable. In contrast, a small gain results in a small output response
to a large input error, and a less responsive or less sensitive controller. Tuning theory and
industrial practice indicate that the proportional term should contribute the bulk of the output
change (Hjalmarsson, Gevers, Gunnarsson, & Lequin, 1998). However, a controller with only
proportional term will result in high instability with oscillation especially for a system with
inertia. As a consequence, derivative term needs to be considered.
The derivative of the process error is calculated by determining the change of the error over
time and multiplying this rate of change by the derivative gain K
d
. Derivative action can be
thought of as making changes slower as one changes too fast. Derivative control will slow the
response of the circuit. In doing so, it is able to partially compensate for the overshoot as well as
damp out any oscillations caused by integral and proportional control. Furthermore derivative
action predicts system behavior and thus improves settling time and stability of the system.
10
Now, the problem is how to define the process variable (PV) and the setpoint (SP) in our
system. Obviously, the manipulated variable (MV) is the pitch angle which is the only interface
provided. Basically, the change of pitch angle results in the change of the acceleration of UAV
proportionally rather than the speed. For the Proportional Controller in last chapter, even the
pitch angle is set to be zero, the UAV still has velocity caused by inertia. Thus, the PV and SP
should be in term of velocity. The way we define SP is proportional to the distance to target.
The larger the distance, the higher the SP (desired velocity). While the UAV reach the target,
the SP should be zero, so that it can stop right there without oscillation. The proportional gain
and derivative gain are tuned and optimized through hundreds of experiments. The experiment
result is shown in Figure 4.3.
0 2 4 6 8
0
50
100
150
200
250
300
Time(s)
Position(cm)
Move forward 300cm
target
UAV
Figure 4.3: Performance of the PID Controller
The improvements of PID controller is significant compared with Proportional Controller.
The UAV keeps approaching the setpoint and the overshoot and oscillation are reduced.
For moving left and right, we apply the same PID controller to adjust roll angle. For
turning and climbing, desired yaw and altitude are obtained by setting the AR.Drone’s yaw
and altitude changing rates. Therefore, P controllers were enough for these two degrees of
11
freedom. As you can see in the figure below, the control loop is fully autonomous now. With
this precise autonomous flight control, the UAV has gained the ability to perform some more
complicated tasks and higher level autonomous behaviors.
Figure 4.4: UAV’s Control Loop
12
Chapter 5
Visual Servoing Using Vertical
Camera
Visual Servoing is also known as vision-based robot control. It combines image processing and
control techniques, in such a way that the visual information is used within the control loop.
Basically, the task can be described as: given a target, we want to track a visual point feature
(centroid or a characteristic point on the target) in order to perform continuous monitoring
by keeping the UAV in flight above it. In other words, to control the position of the UAV
(hovering) autonomously, while keeping the target in the center of image plane. As the on-
board computational power of AR.Drone 2.0 is limited, the object must be easy to detect but
unique as well. What we use is a simple marker consists of two concentric circles with white
and black colors only (shown in Figure 5.1).
5.1 Image Processing
The first step is thresholding. Thresholding is to convert the original image to a dicrete color
image we want. For the marker which consists of only black and white color, the thresholded
image consists of only black and white.
The second step is segmentation. Segmentation is to find the contiguous regions in the
thresholded image with same color. Each of these region will be considered as a blob. The
13
Figure 5.1: Visual Servoing Marker
marker consists three blobs b
1
, b2
, and b
3
. Each blob will be detected with a bounding box
(Figure 5.2).
Figure 5.2: Blobs and Bounding boxes
5.2 Marker Detection
The coordinates of bounding boxes of each blobs on image will be calculated. The uniqueness
of the marker is that b1
is inside b2
, and b
2
is inside b3
. Additionally, they should have the same
center. The vision algorithm scans for the marker and compute the coordinate of marker’s
center in the image.
14
5.3 Marker Tracking
Figure 5.3: Calculation of the relative world coordinate with marker
Figure 7.1 explains how the relative position of the marker is calculated. Since the altitude
can be obtained from the UAV with ultrasound sensor, and the FOV (Field of View) of the
vertical camera is also given. Therefore the relative distance from the marker to the UAV in
real world can be easily calculated. Then the target position will be sent to control loop and
command will be sent to the UAV. The updating speed is around 30 FPS which is good enough
for tracking a low speed target.
5.4 UAV Behaviors
The overall tracking strategy is best described as a simple finite state machine (Figure 5.4) with
two states: search and track. Initially the UAV is in the search mode. The marker detection
algorithm scans for the marker. Once the marker is detected, the system transitions to the
track mode. In this mode, the marker detection algorithm sends the relative distance between
marker and UAV in the world coordinate. The marker tracking algorithm will send command
to UAV to make it track the marker. Similarly, if the target is lost during the track mode, the
15
system transitions back to the search mode.
Take o↵
Search
marker
Found
marker?
Track
marker
Lost
marker?
no
yes
no
yes
Figure 5.4: The state transition diagram for marker tracking
16
Chapter 6
Hula hoop Detection and Flying
Through
In addition to marker tracking, we want to make the UAV do some more interesting things
using the front camera and autonomous control techniques. Our aim is to make the UAV detect
a hula hoop and fly through the hula hoop autonomously and precisely. In this chapter, we
assume that the UAV and the hula hoop are aligned with each other (Figure 6.1). We will talk
about our solution while they are not aligned in the next chapter.
Figure 6.1: UAV is aligned with the hula hoop
17
6.1 Usage of Fish-eye Lens
Since the hula hoop is a large hoop with thin edge, the whole hoop must be visible in order to
detect it. However, the UAV’s front camera has only 90� FOV, which is too narrow and the
UAV will only be able to see a portion of the hoop while it gets close to the hoop. To increase
the FOV of the UAV, we mount a fish-eye lens on the original front camera (Figure 6.2).
Figure 6.2: Fish-eye lens Figure 6.3: FOV increased after using fish-eye lens
With the help of the fish-eye lens, the FOV of the UAV was increased to 178� (Figure 6.3).
The next task is to calibrate the camera and get the intrinsic parameters like focal length so
that the 3D position of object in the real world can be estimated.
6.2 Camera Calibration
For the camera calibration, we use OpenCV and classical black-white chessboard.
Figure 6.4: Chessboard for calibration
18
The aim of camera calibration is to find the camera matrix which is referred as
2
666664
f
x
0 c
x
0 f
y
c
y
0 0 1
3
777775(6.1)
The unknown parameters are f
x
and f
y
(camera focal length) and (cx
, c
y
) which are the
optical centers expressed in pixels coordinates. Basically, we need to take snapshots of the
chessboard with the front camera of UAV and let OpenCV solve the unknown parameters. To
get a good results, we need at least 10 good snapshots of the chessboard in di↵erent positions
from di↵erent angles.
6.3 Image Undistortion
However, using fish-eye lens causes a serious problem which is image distortion (Figure 6.5).
Figure 6.5: Distorted Image
From Figure 6.5, it can be seen that the chessboard in image was distorted. Especially
the distortion on the edge is more severe and apparent. Such significant distorted image will
result in wrong detection of object and estimation of position. Hence, an image undistortion is
necessary.
We use OpenCV to do undistortion and remapping. The formula given by OpenCV is as
follows:
x
corrected
= x(1 + k
1
r
2 + k
2
r
4 + k
3
r
6)
y
corrected
= y(1 + k
1
r
2 + k
2
r
4 + k
3
r
6)
(6.2)
19
So for any pixel point at (x, y) coordinates in the input image, its coordinates on the cor-
rected image will be (xcorrected
, y
corrected
) where k
1
, k
2
, k
3
are distortion coe�cients. However,
the undistortion seems not work well.
Figure 6.6: Undistorted Image Using OpenCV
From the image, it can bee seen that only the chessboard part was undistorted correctly.
Other part especially the edge is still distorted severely. After trying to tune some setup
parameters, we are still not able to get a good result. Then we do some researches regarding
to this problem. We find that the problem we have is probably because that the undistortion
model of OpenCV is based on pinhole camera which is di↵erent from cameras with wide-angle
lens such as fish-eye lens. Hence, in order to get a good undistortion result, we have to find
another way to do undistortion.
After doing a lot of experiments, OCamCalib (Omnidirectional Camera Calibration Toolbox
for Matlab) gives us a good result (Scaramuzza, Martinelli, & Siegwart, 2006a). This toolbox
is designed for catastrophic and fish-eye cameras up to 195 degrees (Scaramuzza, Martinelli, &
Siegwart, 2006b). We use OCamCalib toolbox to calibrate the fish-eye lens again and get
a mapping matrix which is able to correct distorted pixels with a mapping function (Rufli,
Scaramuzza, & Siegwart, 2008).
(xcorrected
, y
corrected
) = mapping(x, y) (6.3)
where (x, y) is the coordinates on distorted images, and (xcorrected
, y
corrected
) is the coordi-
nates mapped to the undistorted image.
20
(a) Distorted Image (b) Undistorted Image
(c) Distorted Image (d) Undistorted Image
Figure 6.7a is one of the chessboard image to do undistortion. Compared with Figure 6.7b,
the undistortion is acceptable and much better then the one using OpenCV. Figure 6.7c is the
image of hoop viewing from the UAV before undistortion. Compared with Figure 6.7d, the line
on the ground and the edge of white board become straight as expected. The improvement
using OCamCalib is significant. Although there is still distortion on the edge, but this is good
enough for us to proceed to the next step.
6.4 Image Processing
The goal of image processing is to locate and extract the hula hoop. Figure 7.3b shows an front
camera view of hula hoop.
(a) Image from Front Camera (b) Segmented Image
Figure 6.8: Image processing results
21
First, the original image is down-sampled to 180p. The reason to do down-sampling is that
object detection is real-time and the UAV always keeps moving. Thus the requirement of image
processing speed is high which must be at least 30FPS. The image we obtained from the camera
is 720p, its computational cost is too high to run on-board. Hence we reduce the image to 180p
to increase the e↵ectiveness of the algorithm.
Next, thresholding converts the color image to a discrete color image. As you can see from
the figures above, some bright orange color paper are stuck to the hula hoop. This makes the
UAV much more easier to distinguish between hula hoop and other objects present in the image.
At last, segmentation will find the contiguous regions with orange color. Figure 7.3a shows
that after segmentation, the orange hula hoop is extracted from the original image.
6.5 Position Estimation
To identify the center of the hoop in the image, we will do a circle fitting which aims to fit the
equation of a circle to the segmented image. The center coordinates and radius of the hoop in
pixels will be computed. Since the size of the hoop in real world can be measured and focal
length is given in camera matrix, the distance between any pixels in the image in real world is
able to be calculated. Thus the relative position of the hoop center to the UAV in real world
can be estimated.
However, in our experiments, we found that the position data is noisy. The noisy data
might due to the vibration of the UAV. Furthermore, the noisy data could cause more severe
vibration. My solution to this problem is to use a modified moving average filter. A moving
average filter smooths data by replacing each data point with the average of the neighboring
data points defined within the span. The equation is
y
s
(i) =1
N
(y(i) + y(i� 1) + ...+ y(i�N + 1)) (6.4)
where ys
(i) is the smoothed value for the ith data point, N is the number of neighboring data
points on left side of ys
(i) , and N is the span. My modified version is to remove the highest
and lowest part of data within the span before averaging. The highest and lowest part of data
22
have large variance which might caused by vibration. Another concern is that how large should
the span is. If the span is too large, the actual large change of position might be smoothed
wrongly. If the span is too small, the smooth e↵ect will not be obvious. As a result, we choose
the span to be same as the FPS rate which is 30. Since the UAV will not have a sudden change
of position in one second normally, it is reasonable to set the span to 30.
6.6 UAV Behaviors
The overall hoop detection and flying through strategy is described as a finite state machine
(Figure 6.9) with three states: search, approach, and fly through. Initially after taking o↵
the UAV is in the search mode. The image processing and hoop detection algorithm scans for
the hoop. Once the hoop is detected, the system transitions to the approach mode. In this
mode, the hoop detection algorithm sends the coordinates of the marker relative to the UAV’s
position and orientation to the hoop approaching algorithm. The hoop approaching algorithm
will send command to UAV to make it approach to the center of the hoop. If hoop is lost
during the approach mode, the system transitions back to the search mode. If the UAV finishes
approaching and is at the right position to fly through, the hoop approaching algorithm will
send command to UAV so that the UAV will fly through the hoop.
23
Take o↵
Search
hoop
Found
hoop?
Approach
hoop
Lost
hoop?
Ready to
go?
Fly
through
hoop
no
yesyes
no
no
yes
Figure 6.9: The state transition diagram for hula hoop detection and flying through
24
Chapter 7
Hula hoop Detection While Not
Aligned
The assumption we made in the last chapter is that the UAV is facing to the hoop plane
perpendicularly. However, most of the time, the UAV will not be aligned with the hoop. To
achieve the alignment, the UAV need to know the angle to the hoop plane.
7.1 Ellipse Fitting
Since the hula hoop is a circle from the view facing to it. If we view it from other directions,
the circle will become an ellipse. Hence, the angle to the hoop plane could be estimated using
ellipse fitting.
However, the experiment results show that the UAV does not always rotate to the correct
direction. This might because of angle detected is wrong. We collect some angle data while the
UAV is static (Table 7.1).
While the drone’s body is on each side of the hoop, three di↵erent sets of data on di↵erent
position are collected. The head of column is the position of the center of hoop in the image.
The angle detected is supposed to have no relationship with the position of the hoop center.
For each row, the angles detected are supposed to sum up to about 180�. However, only when
the center of the hoop is in the center of the image, the angle detected is correct. Our solution
25
Drone’s body
Center of hoop
in image
Left of hoop Middle of hoop Right of hoop
Left
31±2 30±2 27±2
27±2 21±2 16±2
17±2 13±2 11±2
Right
153±2 154±1 141±2
159±2 164±1 150±2
170±2 168±1 166±2
Center
65±2 90±2 109±2
31±2 90±2 154±2
14±2 90±2 168±2
Table 7.1: Hoop angle detected
to this problem is to make sure the angle detected is when the hoop’s center and image center
are match. To make sure this work we test it while the drone is flying. However, the angle
detected is incorrect although the two centers are match. After doing some research, we find
that the problem might be due to motion blur. Therefore, we have to find another solution to
this problem which is to use a marker to indicate the orientation of the hoop.
7.2 Ground Robot Assistance Using Marker
The orientation indication marker (Figure 7.1) consists of two non concentric circles.
The two centers of the non concentric circles form a straight line which intersects the vertical
line of the image. Angle ✓ of Figure 7.2 indicates the orientation di↵erence between the marker
and drone. Then the turning algorithm will send command to the drone to rotate until the
di↵erence become zero so that they will align with each other. At the same time, the marker
is also an visual servoing marker. The drone will be able to track and align with it. While the
drone is not facing to the hoop, the ground robot will lead the drone to face to the hoop. Then
26
Figure 7.1: Orientation Indication Marker
Figure 7.2: Orientation Calculation
the drone will only need to do the hoop detection and then going through.
27
(a) Ground robot with marker
(b) UAV is ready to fly through
Figure 7.3: UAV and ground robot
28
Chapter 8
Experiment Results
8.1 Experiment Setup
We use TurtleBot 2 as the ground robot platform for tracking purpose. TurtleBot is controlled
by a joystick and it has moving forward/backward and turning functions. A marker is placed
on the TurtleBot so that the UAV is able to track the TurtleBot (i.e., marker).
Figure 8.1: Experiment Setup
A marker is on the ground behind the hoop. The marker points to west. At two meters
west of it, another marker is placed on the ground pointing to south.
29
8.2 UAV Behaviors
Basically, the overall UAV behaviors can be described as a finite state machine (Figure 8.2).
First, the UAV takes o↵ from the TurtleBot and starts tracking it.
Next, the TurtleBot will navigate the UAV to the front of the hula hoop. After that, the
TurtleBot turns to face the hula hoop which indicates the orientation of the hoop for the UAV.
The UAV will align with the TurtleBot, and in doing so, the UAV is aligned with the hula hoop.
Then the UAV starts to detect the hula hoop and fly through. At the same time, the
TurtleBot will go to the end of the route to prepare. After the UAV finishes flying through the
hoop, it should be able to find a marker pointing to west. Hence, it will hover on the marker
and turn to align with it. After finish turn, it will perform next task — fly forward to find next
marker.
Similarly, as the next marker points to south, the UAV will hover on the marker and turn
to face south. Then it will fly forward with climbing up and down. After finished, it will start
hover until the TurtleBot navigates it to the next hoop and perform other tasks. Since we only
have one hula hoop, the TurtleBot will navigate the UAV to the same hoop and fly through
again with other tasks following.
8.3 Discussion
We conduct the same experiments thirteen times to make sure our system works as expected.
Through our observation, the average performance is satisfactory. The UAV is able to track
the ground robot consistently. For flying through the hoop, the UAV is able to fly through
accurately and fast. However, some drifting problems are present especially while the UAV
is turning. The drifting problem may cause the UAV to lose the ground robot and marker
temporarily. A demo video will be presented during my presentation.
30
Take o↵
Track
TurtleBot
Ready to go?
Detect and
fly through
the hoop
Track and
align with
the marker
Go to next
marker
Track and
align with
the marker
Fly forward,
climb up
and down
Track
TurtleBot
Detect and
fly through
next hoop...
no
yes
Figure 8.2: The state transition diagram for the final experiment
31
Chapter 9
Conclusion And Future Work
We have presented the design and implementation of a real-time vision-based system for au-
tonomous control and navigation. The autonomous control algorithm is accurate and stable
with high flexibility. The usage of wide-angle lens improves the vision system. The vision
algorithm is fast, robust, and computationally inexpensive.
Observation from thirteen flight experiments shows that our system works accurately and
repeatably. At present, the detection of the hoop while not aligned relies on the indication of
ground robot. In the future, we plan to improve the vision algorithm so that the UAV is able
to detect the orientation of the hoop itself. Also, we plan to improve the landing algorithm so
that the UAV is able to land on a moving target.
In the future, we plan to focus our attention on multi-UAV coordination and motion-
planning. Since complex tasks will require more than one UAV and more than one ground
robot, the coordination and motion-planning is important to prevent collision between UAVs
and to allocate the tasks e�ciently.
32
References
Amidi, O. (1996). An autonomous vision-guided helicopter. PhD thesis, Carnegie MellonUniversity.
Conway, A. R. (1995). Autonomous control of an unstable model helicopter using carrier phasegps only. PhD thesis, Stanford University.
Gurdan, D., Stumpf, J., Achtelik, M., Doth, K.-M., Hirzinger, G., & Rus, D. (2007). Energy-e�cient autonomous four-rotor flying robot controlled at 1 khz. 2007 IEEE InternationalConference on Robotics and Automation (pp. 361–366), IEEE, Roma, Italy, April, 2007.
Hjalmarsson, H., Gevers, M., Gunnarsson, S., & Lequin, O. (1998). Iterative feedback tuning:theory and applications. Control Systems, IEEE, 18 (4), August, 1998, 26–41.
Rufli, M., Scaramuzza, D., & Siegwart, R. (2008). Automatic detection of checkerboards onblurred and distorted images. Proceedings of the IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2008), Nice France, September, 2008.
Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006a). A flexible technique for accurateomnidirectional camera calibration and structure from motion. Proceedings of the FourthIEEE International Conference on Computer Vision Systems (ICVS 2006), New York,NY, January, 2006.
Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006b). A toolbox for easily calibratingomnidirectional cameras. Proceedings to IEEE International Conference on IntelligentRobots and Systems (IROS 2006), Beijing China, October, 2006.
33