REAL TIME DUAL CAMERA SPECTRAL IMAGINGBASED ON NVIDIA TEGRA SOC TO ASSESS UAV MISSIONS

Michele Moscaritolo, Alessandro Della Villa, Giacomo BenelliSan Jose (CA) - 07 April 2016

PRESENTATIONOVERVIEW

Introduction

Requirements

Proposed solution

Development assessment

Discussion

MILITARY COMMERCIAL HOBBY

UAV (Unmanned Aerial Vehicle) known as a drone (Dynamic Remotely Operated Navigation Equipment).

An aircraft without a human pilot aboard.

DRONEAPPLICATIONS

Inspection Agriculture CreativeSurvey &mapping

Safety &Security

Energy

Infrastructure

Telecom

Mining

Infrastructure

Construction

Fire fighting

Surveillance

SAR

Farming Videography

Photography

Research

Vegetationmanagement

DRONE APPLICATIONSPOSE A CHALLENGES

Inspection

Data acquisition & Analysis

Computationalpower

Real timeonboard processing

Multi spectralsensor

Agriculture

ASSESSMENT OF PAYLOAD FOR:

Survey &mapping

Safety &Security

PRESENTATIONOVERVIEW

Introduction

Requirements

Proposed solution

Development assessment

Discussion

BASICREQUIREMENTS

Data acquisition & Analysis

Computational power

Real time on board processing

Multi spectral sensor

Light and small sensor

Reduce cost

Scalable and modular

User/Developer friendly

1

2

3

4

5

6

7

8

MULTISPECTRALSOLUTIONS

Multispectral camera ideal “tool” to capture data in different scenario but:

Expensive Cost !

Processing power hungry !

To satisfy this requirement we must find an alternative solution

PRESENTATIONOVERVIEW

Introduction

Requirements

Proposed solution

Development assessment

Discussion

ADVANCED UAVBASE ON COMPUTER VISION

IR+Visible image sensors coupled with high processing power of Jetson Tegra can be the ideal solution since Tegra can real time associate to each pixel Visible and IR Spetctrum

Nvidia Tegra Family K1, X1, ...

DAYLIGHT CAMERA

THERMALCAMERA

PROCESS UNIT

GOAL:ADVANCED UAVs

Flir DigitalSony optical zoom

Nvidia Tegra Family K1, X1, ...

SOFTWARE - SDK

DAYLIGHT CAMERA

PROCESS UNIT

THERMAL CAMERA

Library of functions real-time computer vision and image acquisition and processing

PERFORMANCECOMPARISON

0 2 4 6 8 10 12 14 16 180

3

1

200

400

600

800

1000

Performance compare for Embedded Computer Vision and Image Processing

Costs

WATTS

X1 NVIDIA 2016oOCCHI II

GFL

OPS

K1 NVIDIA oOCCHI I

SMARTPHONES

MULTISPECTRAL SOLUTION

FPG

A

PRESENTATIONOVERVIEW

Introduction

Requirements

Proposed solution

Development assessment

Discussion

ARCHITECTURE

Custom Algorithms for UAV applications

Metadata, Communication Protocol, Data Storage, Video Output

Software + SDKtegra

V4L

Tegra SoC GPU

Sensor interface

Camera SensorsVisible and Infrared (LWIR)

HDMI

USB

SOFTWAREARCHITECTURE

1

2

3

4

Receive commands from remote and provide output (HDMI or SD)

Efficiently capture from generic USB cameras

Deliver a constant frame rate for UAV fly operations

Take benefit of GPU to combine sensor data

In order to meet requirements we need a software capable to:

SOFTWAREARCHITECTURE

GPU

Pre-process

CORE 2 CORE 1 CORE 3

Visiblecapture MASTER THREAD IR

capture

Process

Outputframe

FrameSX

FrameDX

8.33 fps25 fps

MAVLINK

Pre-process

SOFTWARE ARCHITECTUREFEATURES

Capture and Master have been isolated in separate threads attached to dedicated cores

Capture and Master have fixed pulse interval in order to deliver fixed frame rate (8.33 fps or 25 fps)

To respect time processing constrain GPU is used

GPU processing pipelines can be configured setting up wanted transformations to exetend and customize functionalities

1

2

3

4

SOFTWARE ARCHITECTUREFEATURES

video replacement

TYPICAL SCENARIO

RAW DATA

IMAGE ACQUISITION

Typical scenario Based on previous requirement roof inspection

IMAGE FILTERING& PRE-PROCESSING

SEGMENTATION

MEASUREMENT & ANALYSIS

TYPICAL SCENARIO

RAW DATA

IMAGE ACQUISITION

Typical scenario Based on previous requirement Body signature detection

IMAGEPROCESSING

IMAGE REGISTRATION

MEASUREMENT & ANALYSIS

PRESENTATIONOVERVIEW

Introduction

Requirements

Proposed solution

Development assessment

Discussion

OVERLAYSOLUTION

To adopt in different applications and drone missions

To get relevant spectral information

To have real time on board processing with Nvidia Tegra

Scalar and modular platform has been developed for UAV Multi Spectral applications

It offers less spectral information with respect to hyperspectral but it is 3-4 times cheaper.

Dual head camera with overlay solution is possible:

1

2

3

4

Michele Moscaritolo, Alessandro Della Villa, Giacomo BenelliSan Jose (CA) - 07 April 2016

REAL TIME DUAL CAMERA SPECTRAL IMAGINGBASED ON NVIDIA TEGRA SOC TO ASSESS UAV MISSIONS