dddams-based surveillance and crowd control via uavs and ugvs sponsor: air force office of...

DDDAMS-based Surveillance and Crowd Control via UAVs

and UGVs

Sponsor: Air Force Office of Scientific Research under FA9550-12-1-0238

Program Manager: Dr. Frederica Darema

PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2

Students: A. Khaleghi1, D. Xu1, Z. Wang1, M. Li1, and C. Vo2

1Systems and Industrial Engineering, University of Arizona2Computer Science, George Mason University

PI Contact: [email protected]; 1-520-9530

ICCS 2013, June 7, 2013

mailto:[email protected]

Computer Integrated Manufacturing & Simulation LabDepartment of Systems and Industrial Engineering, The University of Arizona, Tucson

Motivating Problem: TUCSON-1 (Border Patrol)

• 23-mile long area surrounding the Sasabe port of entry in AZ

• Major components: sensor towers, communication towers, ground sensors, UAVs (Arducopter; Parrot AR.Drone), UGVs (sonar, laser), crowds, terrain (NASA)

Eagle Vision Ev3000-D-IR Capabilities Up to:

IR Laser Illuminator 3 kmWireless Link 50 kmIR Thermal Camera Range 20 km Uncooled Thermal Camera Range 4 km

Trident Sentry Node Capabilities Life Span 30-45 daysRadio Frequency Range 300 mPersonal Detection and Tacking Range 30-50 m


Overview of Project• Problem: A highly complex, uncertain, dynamically changing border

patrol environment• Goal: To create robust, multi-scale, and effective urban surveillance

and crowd control strategies using UAV/UGVs– Various modules: detection, tracking, motion planning

• Approach: Comprehensive planning and control framework based on DDDAMS– Involving integrated simulation environment for hardware,

software, human components– Key: What data to use; when to use; how to use[1]

Surveillance and Crowd

Control

Formation control, swarm

coordination

Target Tracking

Vehicle Assignment

and Searching

Target Detection

Surveillance region

selection

Vehicle motion

planning

DDDAMS


Proposed DDDAMS Framework


Fidelity Representation via Crowd Dynamics

Next waypoints of UGV

Low density region invisible by UAV

Representation of crowd individuals

Crowd view by UAV

Crowd view by UGV

Coarser visibility cellView of dense crowd regions

Finer visibility cellView of individuals

Next waypoints of UAVs

High density region visible by UAV

High fidelity

Low fidelity

Hybrid fidelity

Control command generation for UAV/UGV motion planning


Framework of Proposed Methodology


• Detection algorithms (using open source computer vision libraries - OpenCV)

– Human detection (upright humans)• Histogram of Oriented Gradient (HOG)[5]

– Motion detection • Background subtraction algorithms[6]

• Sensors in Parrot AR. Drone 2.0– Front Camera: CMOS sensor with a 90

degrees angle lens – Ultrasound sensor

• Emission frequency: 40kHz • Range: 6 meters

UAV UGV for Crowd Detection


Crowd Tracking Module

UGV Negligible observation error

With observation error

Low resolution

UGV autoregressive model-> forecasting

UAV aggregation model -> prediction

UGV state space model -> filtering


High Resolution

UGV autoregressive model -> forecasting

UAV state space model -> filtering



• Crowd Dynamics Identification

• Crowd Motion Prediction

UAV


UAV with Low Resolution

• Ci,j(t): unit cell at the ith row and the jth column at time t.

• Visibility state: 0 (invisible), 1 (visible).• Si,j(t)=[Ci-1,j-1(t), Ci-1,j(t), Ci-1,j+1(t), Ci,j-1(t), Ci,j(t), Ci,j+1(t), Ci+1,j-

1(t), Ci+1,j(t), Ci+1,j+1(t)]T.

Crowd view by UAV at time t

C3,4(t)=0, invisibleC2,4(t)=1, visible

S3,4(t)=[0,1,1,0,0,0,0,0,0]T

C2,3(t) C2,4(t) C2,5(t)

C3,3(t) C3,4(t) C3,5(t)

C4,3(t) C4,4(t) C4,5(t)


UAV with Low Resolution – Visibility Prediction

• Let ps(i,j)=Pr(Ci,j(t)=1|Si,j(t-1)=s), s ranging from [0,0,0,0,0,0,0,0,0]T

to [1,1,1,1,1,1,1,1,1]T.• Predict Ci,j(t+1)’s visibility

Observation at time t Prediction at time t+1

?

C3,4(t+1)=0

C3,4(t+1)=1

If ps(3,4)>0.5

If ps(3,4)<0.5

ps(3,4)=Pr(C3,4(t+1)=1|S3,4(t)=s),where s=[0,1,1,0,0,0,0,0,0]T

Estimate ps(i,j)


Crowd Prediction by UAV • Bayesian estimation of ps(i,j)

Beta prior: (3,4) ~ beta( (3,4), (3,4))s s sp

(3,4, 1) ~ Binomial( (3,4, 1), (3,4))s s sY t N t p

Example: ps(3,4), s=[0,1,1,0,0,0,0,0,0]T

Prior knowledge of visibility

t

t+1

…

1,..., yt t t

Total # of observations

(3,4, 1)sY t counts

t1

t1+1

t’1

t’1+1

ty

ty+1

t’N-y

t’N-y+1

…

(3,4, 1) (3,4, 1)s sN t Y t

counts

Data:


Crowd Prediction by UAV (Cont’d)

t+1

Posterior: updated belief of cell (3, 4) being visible.

where

A A A(3,4) | (3,4, 1) ~ beta( (3,4), (3,4))s s s sp y t

A (3, 4) (3,4, 1) (3,4, 1) (3,4)s s s sN t y t

A (3, 4) (3,4) (3,4, 1)s s sy t Based on UAV’s information

Prediction at t+2

Aˆ (3,4) 0.5sp Aˆ (3,4) 0.5sp

UAV’s prior

UAV’s prior

UAV’s data

UAV’s data

A Aˆ (3, 4) [ (3,4) | (3,4, 1)]s s sp E p y t

Bayesian estimator: expected probability for cell (3, 4) to be visible.


Crowd Prediction by UAV with UGVs’ Information Aggregation

G G G(3,4) ~ beta( (3,4), (3,4))s s sp Individual dynamics captured by UGVs

G (3,4)sp

UGVs’ prediction at t+2

G1 (3,4)sp

t+1

t+1

A (3,4)sp

A1 (3,4)sp

C (3,4)sp

Combined prediction at t+2

C1 (3,4)spUAV’s prediction at t+2


Crowd Prediction by UAV with UGVs’ Information Aggregation (Cont’d)

C C C(3,4) ~ beta( (3,4), (3,4))s s sp

With aggregated information:

C G(3, 4) (3,4) (3,4) (1 (3,4))( (3,4) (3,4, 1))s s s s s sw w y t where

C G(3, 4) (3,4) (3,4) (1 (3,4))( (3,4, 1) (3,4, 1) (3,4))s s s s s s sw w N t y t

UAV’s informationUGV’s information

UAV’s informationUGV’s information

Balance weight: (3,4) [0,1]sw Close-to-one weight assigned if UGV’s prediction is accurate

Close-to-zero weight assigned if UGV’s prediction is inaccurate

UGV’s information = predicted individual dynamics by UGVs

Crowd Dynamics Identification


Crowd Dynamics Identification - Model• State space model for linear dynamic systems

: stack up of NI state vector,

: state noise

: measurement noise

( ) ( ) ( 1) ( ) ( ) ( ) ( ) ( )t t t t t t t t f ux A x B f B u w

( ) ( ) ( ) ( )t t t t y C x v( )tx T T T T

1 2( ) [( ( )) ( ( )) ... ( ( )) ]IN

t t t t x x x x

,1 ,2T

,1 ,2( ) [ ( ) ( ) ] , [1( ) ,( ) ]ii i i i It x t x v t Nt vt i x

10 0 ... 0

010 ... 0( )

.............

0 0 01

t

A

location speed

( )tw

( )tv

( )tw ( )tv Gaussian random variable

( )ty

( )tA : state transition matrix (interaction among individuals)

: stack up of NI observation vector


Crowd Dynamics Identification – Model (Cont’d)

• State space model for linear dynamic systems

: observation matrix (link state with observation)

: environmental effect; : environmental factors

: effects of UAV/UGVs; : state of UAV/UGVs

( )tC( )tfB

( )tuB

( )tf

( )tu

With UAV/UGV effect

Without UGV/UAV effect

( )tuB


( ) ( ) ( ) ( )t t t t y C x v

Functionality of


Tracking Algorithms

• ObjectiveIdentify the crowd motion dynamics and predict crowd location

• State-of-the-art methods

Algorithm Feature Advantage Limitation

Kalman Filter[1]

Linear dynamic systemsGaussian noise

Continuous state spaceExact solution

Normality and linearity

assumptions

Extended Kalman filter[2]

First-order Taylor series Linearization

For nonlinear system

Computational complexity

Unscented Kalman filter[3]

Sampling-based linearization

For nonlinear system

Uni-modal assumption


Tracking Algorithms (Cont’d)

• State-of-the-art methods

Algorithm Feature Advantage Limitation

Switching Kalman filter[] Dynamic switching

For nonlinear systems

Computationally intensive,

approximate solutions

Grid-based filter[4]

Discrete state space

No linear/normality assumption

Computationally intensive

Particle filter[5]

Monte Carlo estimation

No linear/normality assumption

Computationally intensive


Crowd Dynamics Identification for UGV

• Assumption: negligible measurement noise in state space model

• Equivalence to autoregressive model with exogenous input (ARX model[7])


( )tv

( ) ( ) ( )t t ty C x

( ) ( ) ( ) ( )[ ( ) ( 1) ( ) ( ) ( ) ( ) ( )]t t t t t t t t t t t f uy C x C A x B f B u w

( 1) ( 1) ( 1)t t t y C x 1( 1) ( 1) ( 1)t t t x C y

1

1

( ) ( )[ ( ) ( 1) ( ) ( ) ( ) ( ) ( )]

( )[ ( ) ( 1) ( 1) ( ) ( ) ( ) ( ) ( )]

[ ( ) ( ) ( 1)] ( 1) [ ( ) ( )] ( )

[ ( ) ( )] ( ) ( ) ( )

t t t t t t t t t

t t t t t t t t t

t t t t t t t

t t t t t

f u

f u

f

u

y C A x B f B u w

C A C y B f B u w

C A C y C B f

C B u C w

1 2 3( ) ( 1) ( ) ( ) ( ) ( ) ( )t t t t t t t y β y β f β u ε

ARX model


Crowd Dynamics Identification for UGV (Cont’d)

• ARX Model fitting– Data: most recent observations for each detected individual – Estimation method: ordinary least square

When certain cell becomes dense according to prediction

• Benefit: effective, lower computation cost in identification

predicted location at time t+1

Update UAV prior

observed location at time t observations

up to time t


Motion Planning Module - Algorithm• Find the optimal path, given a map, start location & destination• A grid is constructed based on physical obstacles in the environment

for vehicle movements– 0 for empty grids– 1 for occupied grids

• A* algorithm with 8- point connectivity used, considering two objectives for UAV (, ) and UGV (, )

Graph Search Algorithms Method Complexity Features

Dijkstra Exact cost from start point to any vertex n high Not fast enough

Best-First Search Estimated cost from vertex n to the end point Low Not optimal

A* Total cost from start point to end point lowGuarantees the shortest path in a reasonable time


Motion Planning Module - Objectives Objective 1: Minimize Euclidian traveling distance of UAVs and UGVs

Objective 2: Minimize energy consumption considering UAV’s above ground level (AGL) change and UGV’s downhill/uphill movement

where,

elev

Pe

nalty

Angle (degree)

UGVUAV

𝑓 1( 𝐴 )=∑

𝑡=𝑡 0

𝑇 −1

¿¿¿

𝑓 1(𝐺 )=∑

𝑡=𝑡0

𝑇− 1

¿¿¿

angle=arc tan( |𝑒𝑙𝑒𝑣 (𝑡+1 )−𝑒𝑙𝑒𝑣 (𝑡)|

[[𝑥𝐺 (𝑡+1 )−𝑥𝐺 (𝑡 ) ]2+[ 𝑦𝐺 (𝑡+1 )−𝑦𝐺 (𝑡 ) ]2 ]0.5 )

𝑓 2( 𝐴 )=∑

𝑡=𝑡 0

𝑇 −1

𝑒𝑙𝑒𝑣𝑃𝑒𝑛𝑎𝑙𝑡𝑦 (𝑎𝑛𝑔𝑙𝑒 )∗ ¿¿¿

: Cartesian coordinates of UAV at time t: Cartesian coordinates of UGV at time t: terrain elevation at time t

𝑓 2(𝐺 )=∑

𝑡=𝑡0

𝑇− 1

𝑒𝑙𝑒𝑣𝑃𝑒𝑛𝑎𝑙𝑡𝑦 (𝑎𝑛𝑔𝑙𝑒 )∗¿ ¿¿¿


Control Strategies

minimizing travel time ( )

minimizing energy consumption ( )

minimizing a linear combination of both objectives

( )

( )( ) ,k kk

k k

f x mf x

M m

• Multi-objective optimization for the evaluation function of various paths based on control strategies

• Weighting method for multi-objective motion planning problem with– Normalize each objective as

– Compute the convex combination of all objectives using weights for

1min f 2min f

inf 𝑓 𝑘(𝑥 )=𝑚𝑘 ¿ 𝑓 𝑘(𝑥)=𝑀𝑘

min𝛼1 𝑓 1+𝛼2 𝑓 2


Assembled Arduino-based UAV (Arducopter)

Specifications

• Additional payload: up to 1200g• Flying time: up to 30 minutes (with one 5000 mAh

11.1V lithium polymer battery)• Radio range: approx. 1 mile (more range with amplifier)• Air data rates: up to 250kbps• Radio Frequency: US915 Mhz, Europe433 Mhz

Flight test path on Google mapRadio Control Training Simulator(AeroSIM®) used for practice to minimize risk of damaging real hardware


Agent-based Hardware-in-the-loop Simulation

Another simulator


Finite State Automata based UAV/UGV Execution

0 1 2 3 4 5

6

Connect_IVEstablish

Connection

I T

Connect_OK_VI

O I T

O

Message syntaxxxx_IV: message from hardware interface to vehiclexxx_VI: message from vehicle to hardware interfacexxx: task performed by unmanned vehicle

Message functionsI: input message to unmanned vehicleO: output message from unmanned vehicleT: task message by unmanned vehicle

Arm_IV Arm

Arm_OK_VI

7891011

12

Update Control Commands_IV

Update Control Commands

IT

Update Control Commands_OK_VI

OIT

O

Execute Commands_IV

Execute Commands

Command Execution_OK_VI

13 14 15 16 17Disarm_IV Disarm

I T

Disarm_OK_VI

O I T

Disconnect_IV Discounnt

18

ODisconnect_OK_VI

Update Control Commands_EV

I


UGV Without observation error


Low resolution





High resolution





UAV


Experiment – (1)• Investigate the effect of

detection range on crowd coverage performance

• Setting– A crowd of 40 individuals– 1 UAV, 1 UGV; same

detection range– Fixed value for tracking

horizon and motion planning frequency

• Result Explanation– If detection range increases,

crowd coverage percentage increases and gets smoother over time Performance Improved significantly


Experiment – (2)

• Investigate the effect of number of grids on crowd coverage performance (e.g. GPS resolution)

• Setting– 1 UAV, 1 UGV; Detection

range is 15m for both UAV and UGV

– A crowd of 40 individuals;

• Result Explanation– Performance fluctuates a lot

at number of grids 20*20– Performance improves and

fluctuation diminishes over time as number of grids increases. Performance Improved significantly


Experiment – (3)

40 50 60 70 80 90 100 110 120 130 1400

5

10

15

20

25

Low Fidelity High FidelityTime (Second)

CP

U U

sage

(%)

• Investigate different fidelity levels on crowd coverage performance and computational resource usage (e.g. CPU usage)

• Setting– 2 UAVs, 2 UGVs– A crowd of 80 individuals

separated as two different groups

• Result Explanation– High simulation fidelity

improve the system performance, but consuming more computational power

Low fidelity performance reduced


UGV Without observation error


Low resolution





High resolution





UAV


Numerical Simulation Study

• Simulation setting– 50 individuals with changing dynamics (e.g. changing speed) – One UAV (monitor whole crowd) and two UGVs (track two subgroups of

individuals)– UGV tracking: AR model prediction– UAV tracking: proposed Bayesian updating approach– Crowd dynamics

Time 1 to 4 Time 5 to 6 Time 7 to 15

UGV view range

UAV view range


Crowd Tracking Outcome


Summary and Ongoing Works

• DDDAMS-based planning and control framework for surveillance and crowd control via UAVs and UGVs– A series of algorithms for crowd tracking– Motion planning algorithms– Integrated agent-based simulation platform with UAVs/UGVs

• Ongoing works– Address different control architectures (hierarchical,

heterarchical, hybrid)• Coordinators: ground controller, UAV (team leader)

– Further develop vision-based crowd detection– Enhancement of crowd motion dynamics by extended-BDI– Enhance crowd motion dynamics identification + crowd motion

prediction (e.g. multiple visibility level)– Expand system automation test


QUESTIONS

• Young-Jun Son; [email protected]• http://www.sie.arizona.edu/faculty/son/index.ht

ml• 1-520-626-9530

mailto:[email protected]

http://www.sie.arizona.edu/faculty/son/index.html

http://www.sie.arizona.edu/faculty/son/index.html


References – (1)[1] Darema, F. "Dynamic Data Driven Applications Systems: A New Paradigm for

Application Simulations and Measurements.“ International Conference on Computational Science. 2004, pp. 662–669.

[2] Hays, R., Singer, M. Simulation fidelity in training system design: Bridging the gap between reality and training. Springer-Verlag, 1989.

[3] Celik, N., Lee, S., Vasudevan, K., Son, Y., "DDDAS-based Multi-fidelity Simulation Framework for Supply Chain Systems," IIE Transactions, 2010, 42(5), 325-341.

[4] Celik, N., Son, Y.-J., “Sequential Monte Carlo-based Fidelity Selection in Dynamic-data-driven Adaptive Multi-scale Simulations,” International Journal of Production Research, 2012, 50, 843-865.

[5] Dalal N., Triggs, B., “Histograms of Oriented Gradients for Human Detection,” Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, 886-893.

[6] Sheikh, Y., Javed, O., Kanade, T., “Background Subtraction for Freely Moving Cameras,” Proceedings of IEEE 12th International Conference on Computer Vision, 2009, 1219-1225.

[7] Welch, G. and G. Bishop, An introduction to the Kalman filter. 1995.

[8] Einicke, G.A.; White, L.B. (September 1999). "Robust Extended Kalman Filtering". IEEE Trans. Signal Processing 47 (9): 2596–2599.


References – (2)[9] Julier, S.J. and J.K. Uhlmann, Unscented filtering and nonlinear estimation.

Proceedings of the IEEE, 2004. 92(3): p. 401-422.

[10] Pavlovic, V., J.M. Rehg, and J. MacCormick, Learning switching linear models of human motion. Advances in Neural Information Processing Systems, 2001: p. 981-987.

[11] Silbert, M., T. Mazzuchi, and S. Sarkani. Comparison of a grid-based filter to a Kalman filter for the state estimation of a maneuvering target. in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series. 2011.

[12] Ristic, B., S. Arulampalam, and N. Gordon, Beyond the Kalman filter: Particle filters for tracking applications. 2004: Artech House Publishers.

[13] Lütkepohl,H. New Introduction to Multiple Time Series Analysis. Berlin: Springer, 2006.

[14] Patel, A. 2003. "Amit's Thoughts on Path-finding- Introduction to A*" Accessed Jun 2, 2013, http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html.

[15] Sinnott, R. W. “Virtues of the Haversine.” Sky telescope, 1984, 68, 159.

[16] Moussaïd, M., D. Helbing, and G. Theraulaz. “How Simple Rules Determine Pedestrian Behavior and Crowd Disasters.” In Proceedings of the National Academy of Sciences 2011, 108, No. 17:6884-6888.

http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html

http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html

dddams-based surveillance and crowd control via uavs and ugvs sponsor: air force office of...

Documents

university of arizona

industrial engineering

uav crowd view

crowd control strategies

tucson detection algorithms

dddams framework slide

crowd dynamics

computer science