research on vision systems for small unmanned vtol vehicles k. p. valavanis, m.kontitsis, r.garcia
TRANSCRIPT
![Page 1: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/1.jpg)
Research on Vision Systems for Small Unmanned VTOL
Vehicles
K. P. Valavanis, M.Kontitsis, R.Garcia
![Page 2: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/2.jpg)
A UAV Vision System for Airborne Surveillance
M.Kontitsis, K. Valavanis
Technical University of Crete
University of South Florida
N. Tsourveloudis
![Page 3: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/3.jpg)
Objectives
• Present a methodology for the design of a machine vision system for aerial surveillance by Unmanned Aerial Vehicles (UAVs)
• Identify specified thermal source• Perform these functions on board the
UAV in Real time• Flexible enough to be used in a variety
of applications
![Page 4: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/4.jpg)
Machine Vision System
IR/NIR image Noise reduction
Feature extraction
(Size, Mean intensity)
Feature vectors classification
Alarm on/off
Persistence
![Page 5: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/5.jpg)
Input Images
• IR (3μm ~ 14μm)
• 8bit grayscale
• Near IR camera (1μm ~ 3μm)
![Page 6: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/6.jpg)
Noise Reduction
• 5x5 spatial Gaussian filter
+ Smoothes noise while preserving most of the features on the image
![Page 7: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/7.jpg)
Feature Extraction
• Size of region using a region growing algorithm
• Mean intensity of region defined as
This module attempts to extract information about the regions on the image
![Page 8: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/8.jpg)
Feature Vector Classification Subsystem
Mean Intensity of Region
Size of Region
Target Identification Possibility
Fuzzy Classifier
![Page 9: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/9.jpg)
Mean Intensity Membership Functions
Grayscale values
HighMidLow
![Page 10: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/10.jpg)
Region Size Membership Functions
# of Pixels
Small Medium Large
![Page 11: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/11.jpg)
Objective ID Possibility Membership Functions
![Page 12: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/12.jpg)
Output of the Fuzzy Classifier
![Page 13: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/13.jpg)
Classification Example
![Page 14: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/14.jpg)
Classification Result
p>0.8
0.5<p<0.8
p<0.5
![Page 15: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/15.jpg)
Alarm raising
• Persistent classification of a certain region as of High Possibility raises the alarm
• The region that raised the alarm is pin-pointed by a red cross
• The alarm stays on even if the thermal source is temporarily occluded by surroundings or lost due to violent camera vibration
![Page 16: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/16.jpg)
Alarm raising
Mechanism used : Alarm Registry
Region Coordinates Variable persistancei1,j1 p1
i2,j2 p2
…. ……..
in,jn pn
If pi > Ton => Activate alarm
If pi < Toff => Deactivate alarm
![Page 17: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/17.jpg)
Complexity
• Noise Reduction O(n2) for (nxn) image
• Region Growing O(n2) for (nxn) image
• Fuzzy Logic Classifier* O(nxm)
*in its current implementation
n inputs, m rules
![Page 18: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/18.jpg)
Case Study: Forest fires
Adjusting membership functions manually
![Page 19: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/19.jpg)
Classification Example
Thermal source (fire)
objective present
![Page 20: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/20.jpg)
Classification Result
possibility>0.7 0.5< possibility <0.7 possibility <0.5
objective present
![Page 21: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/21.jpg)
Classification Example
objective absent
![Page 22: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/22.jpg)
Classification Result
possibility>0.7 0.5< possibility <0.7 possibility <0.5
objective absent
![Page 23: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/23.jpg)
Classification Result (Video)
(objective present)
![Page 24: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/24.jpg)
Classification Result (Video) (objective absent)
![Page 25: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/25.jpg)
Automatic Parameter Selection
• aij bij cij dij for i=1 and j =1,2,3 which define the form of the membership functions of Mean Intensity
aijbij cij dij
![Page 26: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/26.jpg)
Basic Elements of the Genetic Algorithm
Fitness function
0.05( ) distfitness x e
fitness(x)=1 correct deactivation of the alarm
fitness(x)=0 in any other case
correct activation of the alarm
• Chromosome => parameters x=(aij bij cij dij)
![Page 27: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/27.jpg)
Basic Elements of the Genetic Algorithm
Selection operator selects individuals for mating as many times as the ratio of their fitness to the total fitness of the population
Crossover operator crossover probability pc=0.7
Mutation operator mutation probability pm=0.001
![Page 28: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/28.jpg)
Mean Intensity M.F. as evolved by GA
![Page 29: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/29.jpg)
Result (using GA for parameter selection)
![Page 30: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/30.jpg)
Remarks
• Adjustable for a variety of applications
• Real time execution
• Correct identification rate of about 90%
• False alarms not entirely avoided (especially in the system evolved by the GA)
![Page 31: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/31.jpg)
Design, Implementation and Testing of a Vision System for
Small Unmanned VTOL Vehicles
K. P. Valavanis, M.Kontitsis, R.Garcia
![Page 32: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/32.jpg)
Aim of the work
• To explore the design alternatives in the attempt to implement a functional vision system for a small Unmanned VTOL.
• Two different approaches examined: – On board processing– On the ground processing
![Page 33: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/33.jpg)
Limitations
• Weight Limitations
• Power Supply Limitations
• Processing power issues
• Communications
![Page 34: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/34.jpg)
Ground Control Station (GCS )
UAV / VTOL
Data link (telem
etry + video)
•Sensing (camera)
•Transmitting data to GCS
•Map Building
•Target Identification
•Command Issuing
Com
man
ds
Centralized approach (processing wise)
Processing is left to the PC on the Ground Control Station
![Page 35: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/35.jpg)
Ground Control Station (GCS )
UAV / VTOL
Data link (telem
etry + video)
•Sensing (camera)
•Map Building
•Target Identification
•Transmitting data and alarm signal to GCS
•Command Issuing
Com
man
ds
De-centralized approach (processing wise)
Processing is carried out locally on the PC onboard the UAV / VTOL
![Page 36: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/36.jpg)
General trends in the area
Institution
Machine Vision techniques
used Processing unit VehicleBerkeley University [6] No details provided No details provided BEAR
Georgia Tech [18] [19]
Edge detectors, morphing,statistical pattern matching
On- board Rmax by Yamaha
Standford University [10] [12]
YUV color segmentation, signum of Laplacian of Gaussian (sLoG)
On-the-ground Hummingbird Aerospace Robotic Laboratory at Standford
MIT [21] Template matching On-the-ground Black Star by TSK
Rose Hulman IT (RHIT) [22] Template comparison On-board Bergen Twin
IT Berlin [15] No details provided On-the-ground MARVIN by SSM Technik
University of Texas [13] Edge linking matching On-the-ground XCell .60
Swiss Federal Institute of Technology (ETH) [23]
No details providedOn-board integrated in
camera Huner Technik
Carnegie Mellon University [24]
Template matching and RGB color
On-the-ground Rmax by Yamaha
USC [3] [5]Omnidirectional, optic flow
On-board Bergen Twin
Southern Polytechnic State Univesity [14]
Stereo vision, Sobel egde detector
On-the-ground Vario Robinson R22
Linkoping University, Sweden (WITAS) [25]
No details provided On-board Rmax by Yamaha
![Page 37: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/37.jpg)
Functionality and characteristics
Institution Berkeley University
Georgia Tech
Univ. of South
California
COMETS*
[26]WITAS
+[25]CNRS~
[27]
Experimental setup
Dynamic observer X X X X X X
Dynamic environment
X X
Static / man-made environment
X X X
Known landmarks X X X
Natural landmarks X
Calibrated cameras X
Capabilities 3D reconstruction / depth mapping
X X
Object identification
X X X X
Object tracking X X X
Methods used
Optic flow X X X
Motion estimation X X X X
IMU data X
Template matching X X X X
![Page 38: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/38.jpg)
Processing on the Ground
![Page 39: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/39.jpg)
Hardware configuration (on the ground processing)
![Page 40: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/40.jpg)
Vision algorithm overview
![Page 41: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/41.jpg)
Experimental Results
![Page 42: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/42.jpg)
“Mine” detection results
![Page 43: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/43.jpg)
“Mine” detection results 2
![Page 44: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/44.jpg)
“Mine” detection results 3
![Page 45: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/45.jpg)
“Mine” detection results 4
![Page 46: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/46.jpg)
Processing on board the VTOL
![Page 47: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/47.jpg)
Raptor 90 with on board vision system.
![Page 48: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/48.jpg)
On board processing
Firewire camera
Onboard PC Wireless 802.11b
Wireless 802.11b
Ground Computer
Used for processing
Used for monitoring
![Page 49: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/49.jpg)
On board system
![Page 50: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/50.jpg)
Onboard system architecture (hardware)
![Page 51: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/51.jpg)
On board system• 1.2 GHz EPIA Processor• Via Embedded motherboard• Unibrain Firewire Camera• 1 Gig 266 MHz RAM• 1 Gig Compact Flash• Compact Flash to IDE adapter• Motorola M12+ GPS Receiver• 8 Channel Servo Controller• 200 W Power Supply• 14.8 V LiPo Battery• 12 V Voltage Regulator• 802.11B Cardbus
![Page 52: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/52.jpg)
Key Hardware Components
• Mini-ATX motherboard– Low weight– Small size
• Unibrain Firewire camera– Lightweight (60g)– Built-in Firewire interface
• 1 Gig Compact Flash– Substitutes the vibration sensitive hard-drive
• Lithium Polymer (LiPo) batteries– High amperage output for it’s size
![Page 53: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/53.jpg)
Software Details
• Linux operating system (Slackware v10)
• Open source libraries libdv, libraw1394, libavc1394, libdc1394 used for Firewire access
• Vision code written in C-language for speed
![Page 54: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/54.jpg)
Minor Software Enhancements and
Experimental Results
![Page 55: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/55.jpg)
Detection of more than one objects
• Using the same vision algorithm• Employing Regions of Interest (ROIs) on the
image to separate objects– Byproduct execution speedup (x2 in average) since
only the pixels in the ROIs are processed over every frame.
• Every X (typically 510) frames the algorithm searches the whole image for new objects.
– Regions can be tracked using the pan/tilt to keep the object inside the frame while the VTOL moves
![Page 56: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/56.jpg)
Objects of interest are enclosed in a rectangle
Detection of more than one objects
![Page 57: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/57.jpg)
Detection of more than one objects
Objects of interest are enclosed in a rectangle
False alarm
![Page 58: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/58.jpg)
Detection of more than one objects
Objects of interestNot well positioned rectangle
![Page 59: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/59.jpg)
Communication Issues (for the on-board system)
![Page 60: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/60.jpg)
Communication channels on the VTOL
On board computer
IMU
802.11b/g
GPS
Pan/tilt servos
Control servos
autonomous operation
![Page 61: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/61.jpg)
Communication channels on the VTOL
On board computer
IMU
802.11b/g
GPS
Pan/tilt servos
Control servos
Critical channels
autonomous operation
![Page 62: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/62.jpg)
Data going into the PC
• Uncompressed images at 30 frames / sec (critical for object recognition/navigation)
• Inertial Measurement Unit (IMU) data 4 to 10 Hz (critical for navigation)
• GPS data (critical for navigation)
![Page 63: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/63.jpg)
Data coming out of the PC
• For monitoring purposes (not critical)
– Compressed images at 30 frames / sec– IMU data 4 to 10 Hz– GPS data
• Critical to the operation of the VTOL– Commands to servo-boards– Object identification alarms
![Page 64: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/64.jpg)
Bandwidth requirements for input channels
• 640x480 images at 30 frames / sec approx 220Mbps– Firewire link IEEE 1394 (400Mbps)
• IMU data at 10 Hz approx 6kbps– Serial RS-232
• GPS data at 4 Hz < 1kbps– Serial RS-232
![Page 65: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/65.jpg)
Bandwidth requirements for output channels
• Commands to servos < 1 kbps– Serial RS-232
• Telemetry data and compressed video at 30 fps approx 1.5 to 4 Mbps (depending on image quality)
– 802.11 b/g (11/54 Mbps)
![Page 66: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/66.jpg)
Bandwidth issues of the 802.11
• Video data flood the wireless channel
• Bandwidth decreases with range
– As a result the video displayed on the ground station at less than 30 fps.
![Page 67: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/67.jpg)
Analog video channel
• Substitutes the digital firewire (IEEE 1394)
• Delivers 30 fps regardless of range
• Independent from the onboard PC
• Frees bandwidth of the 802.11 to be used for other purposes
– Frame grabber required for digitization in order for the PC to process the images.
![Page 68: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/68.jpg)
Communication channels on the VTOL
On board computer
IMU
802.11b/g
GPS
RF Transmitter (900MHz)
Pan/tilt servos
Control servos
(alternative design)
![Page 69: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/69.jpg)
Communication channels on the VTOL
On board computer
IMU802.11b/g
GPS
Pan/tilt servos
Control servos
Critical channels
RF Transceiver (72MHz)
semi-autonomous operation
![Page 70: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/70.jpg)
EM compatibility
• The onboard digital channels exhibited no compatibility problems
• Theoretically the RF channels are very well separated in frequency but….
• The RF are still vulnerable to interference
![Page 71: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/71.jpg)
Security issues
• Both 802.11 and RF channels can be made secure using encryption
• Easier done on the 802.11
![Page 72: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/72.jpg)
Conclusions
• Real time processing rate achieved
• On board system preferred because it promotes autonomy
![Page 73: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/73.jpg)
Future work
• Motion estimation
• Structure from motion
• Expand the feature space (size,colortexture,shape etc)
• Visual Simultaneous Localization And Mapping
• Quantify relationship between weight-power-algorithmic complexity
• Optimization of vision routines
• Use of a better processor
![Page 74: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/74.jpg)
Application on real-time traffic data extraction
![Page 75: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/75.jpg)
Aim of the work
• Design a vision algorithm to run on a small VTOL capable of extracting real time traffic data from video.
• The data will be used as input to traffic simulation models
![Page 76: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/76.jpg)
Algorithm overview
Stabilization
Motion Extraction
Feature Extraction
Feature Grouping
Vehicle Tracking
IMU & GPS data
Environment Setup Selection
Traffic Statistics
Images from camera
![Page 77: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/77.jpg)
Motion estimation• Motion is extracted by differencing two consecutive frames
( - ) ( = )
![Page 78: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/78.jpg)
Feature extraction and grouping
• A morphological operator (dilation) is used on the image of differences to group together scattered pixels of an object
(dilation x2)
![Page 79: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/79.jpg)
Grouping (continued)
• The extracted regions are enclosed in Minimum Bounding Rectangles (MBRs)
![Page 80: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/80.jpg)
Selecting “counting zones”
• Regions of the image are selected as counting zones.
• Cars are counted as they enter and leave them.
• Shaded areas mark the counting zones.
• Colors are used to differentiate between them.
![Page 81: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/81.jpg)
Sample result
Relative traffic load per regionInput with overlaid region markers
![Page 82: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/82.jpg)
Output Data
• Algorithm can provide the following data:
– # of cars on a specific link at any point in time• Assuming that a link is sufficiently small to fit in the
camera’s field of view. (multiple VTOLs needed to cover a significant area)
– Average flow per link
![Page 83: Research on Vision Systems for Small Unmanned VTOL Vehicles K. P. Valavanis, M.Kontitsis, R.Garcia](https://reader038.vdocument.in/reader038/viewer/2022103100/56649e7d5503460f94b801c9/html5/thumbnails/83.jpg)
Ongoing work
• Automatic selection and placement of “counting zones”
• Create tables of data suitable for the simulation software
• New input data available (show video)