| Mechanical Engineering

Introduction Methodology Results Conclusion

Acoustic Source Localization with a VTOLsUAV Deployable Module

Kory Olney

Department of Mechanical EngineeringUniversity of South Florida

October 18, 2018

Kory Olney 1/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Overview1 Introduction

BackgroundObjectivesContributions

2 MethodologyMathematical ModelsHardware

Localization ModulesUAS Design

Software3 Results

ExperimentPerformance MetricsError Analysis

4 ConclusionSummaryFuture WorkAcknowledgments

Kory Olney 2/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Background I

The recent increase in availability of low cost small unmannedaircraft system (sUAS) has led to new opportunities

Sensors are becoming smaller and less expensive whileproviding more computational power

Many open source contributions in software and hardwareoriginating from the hobbyist radio control (RC) market

Kory Olney 3/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Background II - sUAS

Useful in applications where humans cannot safely go

Use cases for sUAS

Industrial inspection

Shipping and delivery

Agriculture

Cinematography

Military

And many more...

Kory Olney 4/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Background III - Source Localization

Applied throughout the acoustic and electromagneticspectrum

Beamforming

Wifi, GPS

Radio communications

Seismology

Acoustic range

Speaker localization

Critical infrastructure security

Military and law enforcement

Kory Olney 5/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Objectives

Construct a low-cost prototype sUAS with a payload capacityof at least 4 lb

Construct a microphone array that can addresses constraintsof the sUAS problem

Develop a data acquisition unit with DSP methods to performdirection of arrival (DOA) estimation in real time

Kory Olney 6/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Contributions

Developed code for 8 channel synchronous sampling

Designed and fabricated prototype direction finding module

Designed and fabricated a sUAS that can host the module

Applied DSP methods to the acoustic noise and direction ofarrival problems

Kory Olney 7/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Mathematical Models

Signal Modelx1(t) = s1(t) + n1(t)

where x1(t), s1(t), and n1(t) are real, stationary randomprocesses and s1(t) is assumed to be uncorrelated with n1(t)

E{n1(t)} = 0

Kory Olney 8/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Array Time Difference of Arrival (TDOA) Model

x1(t) = s1(t) + n1(t)

x2(t) = s1(t − D1) + n2(t)

...

xm(t) = s1(t − Dm) + nm(t)

D1 through Dm are thetime of arrival delaysbetween channels whichare to be estimated

Kory Olney 9/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Cross Correlation - Continuous

Rxy (τ) =

∫ ∞−∞

x∗(t)y(t + τ)dt

Rxy (τ) =1

T

∫ T

0x(t)y(t + τ)dt

Rx1x2(τ) = Rs1s1(τ + D) + Rn1n2(τ)

* denotes complexconjugate

T represents theobservation interval

Kory Olney 10/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Cross Correlation - Discrete

Rxy [D] =∞∑

n=−∞x∗[n]y [n + D]

Rxy (r∆t) =1

N − r

N−r∑n=1

xnyn+r

r∆t ≈ τr = 0, 1, 2, ...,m withm < N

r represents the index ofthe sequence

Kory Olney 11/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Cross Power Spectrum

Sxy (f ) =

∫ ∞−∞

Rxy (t)e−j2πftdt Fourier transform of thecross correlation

Kory Olney 12/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Generalized Cross Correlation

H1

H2

x1

x2

y1

y2

Delay

Cross

Correlation

Figure: Filter Model

Sy1y2(f ) = H1(f )H2∗(f )Sx1x2(f )

Kory Olney 13/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Generalized Cross Correlation - PHAT

ψp(f ) =1

|Sx1x2(f )|= H1(f )H∗2 (f )

Ry1y2(τ) =

∫ ∞−∞

ψp(f )Sx1x2(f )e j2πftdf

Phase transform (PHAT)

Weighting function appliedfor whitening the signals

Normalizes cross powerspectrum

Sharpens cross correlationfor better resolution intime domain

y1 and y2 are signals x1and x2 respectively, afterfiltering

Kory Olney 14/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Discrete Implementation

hj =N−1∑b=0

y1by2j+b

for j = −(N − 1),−(N − 2), ...,−1, 0, 1, ..., (M − 2), (M − 1)

Ry1y2(i) = hi−(N−1)

τy1y2 = argmaxi

Ry1y2(i)

N is the number of elements in sequence y1

M is the number of elements in sequence y2

τ is the estimate of time delay

Kory Olney 15/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Mic Position and DOA Vector

Pi =

xiyizi

K =

kxkykz

=

sinθcosφsinθsinφcosθ

where Pi is the position of the ith microphone in the arraywith respect to an arbitrarily defined reference

K is the DOA vector

Kory Olney 16/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

TDOA Between Two Channels

τi ,j =1

c[(xj − xi )sinθcosφ+ (yj − yi )sinθsinφ+ (zj − zi )cosθ]

τi ,j =1

c[Pj − Pi]K

Given by projecting the position difference vector onto theDOA

Assumes wavefront propagates as a plane

Kory Olney 17/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Reference Free Position Difference Matrix

S = [P2 − P1, ...,PN − P1,P3 − P2, ...,PN − PN−1]T ∈ RN(N−1)

2×3

Calculating the positioning difference between eachcombination of microphones in the array

N is the number of sensors in the array

Kory Olney 18/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Reference Free TDOA Vector

~τ = [τ2,1, ..., τN,1, τ3,2, ..., τN,2, ..., τN,N−1]T ∈ RN(N−1)

2×1

TDOA between all combinations of microphones in the array

Kory Olney 19/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

TDOA Equation set

τ =1

cSK

c is the estimate of the speed of sound

Leads to a linear least squares solution

K = −c(STS)−1ST τ

Kory Olney 20/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Elevation and Azimuth Estimates

φ = tan−1(ky

kx)

θ = tan−1(

√k2x + k2y

kz)

Azimuth

Elevation

Kory Olney 21/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware Overview

Localization module

Microphones

Array Design

TI ADS1278

NI myRIO

Interface Board

sUAS Design

Frame

Powertrain

Autopilot

Tx Rx

Kory Olney 22/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - Localization Module - Microphones

Figure: Adafruit Silicon MEMSMicrophone Breakout-SPW2430

Low cost, lightweight,MEMS breakout board

Amplifies signals to linelevel, with max 1V peakto peak

Flat frequency responsefrom 100Hz to 10kHz

3 wire configuration: 3.3Vsupply, ground, DC output

0.67V DC bias

Kory Olney 23/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - Localization Module - Array Design

Figure: 8 Channel MicrophoneArray Design

8 analog microphones inspherical arrangement

3D printed and carbonfiber frame

Radius of 11.25 in

Single 24 pin femaleribbon cable

Kory Olney 24/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - Localization Module - TI ADS1278

Figure: ADS1278 EVM-PDKADC

Synchronous sampling of 8analog channels

SPI output

∆Σ configuration -Oversampling

Onboard or supplieddigital clock

Up to 144 ksps/channel

Built in 1st-order analoglowpass filters

Kory Olney 25/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - Localization Module - NI myRIO

Figure: National InstrumentsmyRIO Embedded System

Reconfigurable InputOutput - RIO

Xilinx FPGA and CPU

40 MHz onboard clockthat can simulate higherrates

Built-in accelerometer

34 pin MXP connections

6V to 16V power supply

Simple functionality withLabVIEW

Kory Olney 26/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - Localization Module - Interface Board

Figure: Top of Interface Boardwith 34 and 24 pin Connectors

Interface betweenADS1278, myRIO, andmicrophones

Custom designed printedcircuit board (PCB)

Power: 3.3V, 5V, frommyRIO and 1.8V regulatoronboard

Kory Olney 27/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - sUAS Design - sUAV Frame

Figure: sUAV Hexacopter Frame

Turnigy Talon from HobbyKing

Carbon fiber frame exceptfor assembly hardware andmotor mounts

Boom radius of 12.5 in

Kory Olney 28/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - sUAS Design - Powertrain

Figure: KDE 2315XF Motor

Brushless DC motor

2050 RPM/V - 30340Max RPM at 14.8V

Motor torque 0.0035ft-lbs/A

2.66A at 14.8V

Theoretical max thrustwith 8 in propellers 3.5lb

Kory Olney 29/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - sUAS Design - Autopilot

Figure: Pixhawk 2.1

Open source hardwaredesign

Designed specifically tofunction with Ardupilot -open source software

Large user base forsupport and debugging

Redundant IMUs

Intel Edison port

Kory Olney 30/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - sUAS Design - Tx Rx

Figure: Taranis QX7 Transmitter

16 Channels

Haptic vibration feedbacksystem

Runs on OpenTX

Reciever signal strengthindicator

Real time flight datalogging to microSD

Kory Olney 31/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Hardware - sUAS Design - Tx Rx

Figure: FrSKY X8R 2.4GHzReceiver

8 channel telemetryreceiver with smart bus

Full duplex

Daisy chainable for 16channels

Kory Olney 32/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

System Overview

Source

Mic Array Interface

Board ADC

myRIO

FPGA

SPI Parsing

CPU

DMA

FIFO

DOA Vector

DOA

Algorithm

Bandstop

Filter

Match

Pattern?

GCC-

PHAT

TDOA

(F)

(T)

Figure: General Data Flow Through System

Kory Olney 33/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Field Programmable Gate Array

Figure: FPGA Top Level

State machine

SCTL at 48MHz

24MHz modulator clockto ADC

8 channels of 24 bits eachat 93.75 kHz

U32 to I32

Target Scope FIFO toDMA FIFO

Kory Olney 34/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

CPU I

Figure: RT Top LevelKory Olney 35/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

CPU II

Producer-consumer loop

Monitors single channel for impulse detection

Trigger sends array to local FIFO for DOA estimate

User defined bandstop filter and trigger threshold

Kory Olney 36/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Data Collection

Recordings of 9 different calibers at Pasco County Range

Microphones were approximately 29 feet from the source

5 shots of each caliber were recorded with only ambient noise

Shots were then recorded with the sUAS motors operating

Kory Olney 37/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Gunshot Waveform 9mm

Vol

ts

0.035

-0.04

-0.035

-0.03

-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

Time (s)0.430 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Time Domain Channel 1

Figure: 9mm Time Domain

Shifted from DC Bias

≈ 0.07V P2P

≈ 3 ms in length

Kory Olney 38/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Dual Channel 9mm

Volts

0.79

0.64

0.65

0.66

0.67

0.68

0.69

0.7

0.71

0.72

0.73

0.74

0.75

0.76

0.77

0.78

Time (s)0.350.1 0.125 0.15 0.175 0.2 0.225 0.25 0.275 0.3 0.325

Waveform Plot 2 channels

Figure: Comparing 2 CH 9mm

Channels 1 and 4

≈ 0.35 ms TDOA

Kory Olney 39/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Propeller Noise

Volts

0.79

0.66

0.67

0.68

0.69

0.7

0.71

0.72

0.73

0.74

0.75

0.76

0.77

0.78

Time (s)0.1280 0.02 0.04 0.06 0.08 0.1 0.12

Mic 5

Mic 6

Mic 7

Mic 8

Mic 1

Mic 2

Mic 3

Mic 4

Waveform Plot 1-8

Figure: All Channels of Props

Channels 1 through 8

Varying DC bias betweenchannels

≈ 0.02 V P2P

Recorded at 50 % throttle

Kory Olney 40/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Propeller Noise Filtering

Ampl

itude

1

0

0.2

0.4

0.6

0.8

Frequency (Hz) 25000 250 500 750 1000 1250 1500 1750 2000 2250

FFT Channel 1Raw FFT

Figure: Frequency Plot of Prop Noise

Ampl

itude

1

0

0.2

0.4

0.6

0.8

Frequency (Hz) 25000 250 500 750 1000 1250 1500 1750 2000 2250

Filtered FFT Channel 1 Filtered FFT

Figure: Filtered Sequence FFTKory Olney 41/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

9mm FFT

Ampl

itude

1

0

0.2

0.4

0.6

0.8

Frequency (Hz) 40000 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000 3250 3500 3750

FFT Channel 1Raw FFT

Figure: Extended Band FFT

Ampl

itude

1

0

0.2

0.4

0.6

0.8

Frequency (Hz) 6000 50 100 150 200 250 300 350 400 450 500 550

FFT Channel 1Raw FFT

Figure: Lower Band FFTKory Olney 42/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Frequency Response

Mag

nitu

de d

B5.0

-250.0

-200.0

-150.0

-100.0

-50.0

Frequency (Hz)3000.00.0 250.0 500.0 750.0 1000.0 1250.0 1500.0 1750.0 2000.0 2250.0 2500.0 2750.0

MagnitudeMagnitude response

Figure: Frequency Response of Bandstop Filter

Finite impulse response (FIR)Cutoff from 500Hz to 1.9kHzLinear phase responseMaximum 511 taps

Kory Olney 43/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

System Weight

Component Weight(g)sUAV 1520

Battery 416

Mic Array 200My PCB 45

ADC 36myRIO 235Housing 201

Module: 717Total: 2653

Kory Olney 44/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Impulse Detection Reliability

Currently cross correlating each new frame against a templatewavelet

Normalized, windowed, and filtered

A trigger for an impulse detection is set to true if the maxvalue of the cross correlation exceeds a user defined threshold

False negatives and false positives are a significant concern forthis system

Kory Olney 45/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Estimation Error

Initial tests show that the error for elevation and azimuth isapproximately +/- 7 %

Recently simulated with clapping and snapping

Future experiment soon to study the localization error

Demonstration...

Kory Olney 46/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Summary

Successfully demonstrated that a lightweight acoustic sourcelocalization module can function in real time onboard a sUAVwhile estimating both elevation and azimuth without motorsoperating

GCC-PHAT method is sufficient for TDOA estimates inmodestly noisy environments

Noise characteristics of the current sUAS significantly degradesignal quality

Higher quality mics are necessary for long range estimates

Low cost solution to serve a variety of potential engineeringchallenges

Driver for ADS1278 has potential for a variety of applications

Many improvements can be made

Kory Olney 47/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Future Work

Perform a thorough evaluation of the system performanceunder varying conditions to determine the maximum effectiverange for accurate DOA estimation

Improve the digital signal processing techniques to be morerobust and better tuned for specific applications

Acoustic classification i.e. caliber recognition

Incorporate a camera and machine vision techniques toattempt at precisely identifying the source of an acoustic event

Develop a single PCB that is smaller, lighter, and performs allof the necessary computations internally

Create a module that is designed specifically to comply with aCOTS sUAS

Kory Olney 48/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Acknowledgments

Dr. Robert H. Bishop for providing me the opportunity toundertake this research and for finding funding for theassistantship. Dr. Jonathan Gaines for providing guidancethroughout this process. Michael Guinn for helping me obtain theposition to make this all happen. SOFWERX for providing thefinancial support and lab space. Pasco County Sheriff’s Office forallowing me to use their outdoor range for testing.

I would also like to thank my family and friends that havesupported me throughout my entire graduate research experience.

Kory Olney 49/50

| Mechanical Engineering

Introduction Methodology Results Conclusion

Questions?

Thanks for your attention!

koryolney@mail.usf.edu

Kory Olney 50/50