CoMotionComputational Methods for

Collaborative Motion

Pursuit Evasion Games for Networks of UUVs

November 2004

Mike Eklund, Jonathan Sprinkle, Shankar Sastry

Pursuit Evasion Games for Multiple UUVs

• UUVs have the potential to provide an effective defense against submersible threats to military and civilian assets

• The strategies and protocols for their operation are at least as big a challenge as the design and construction of the vehicles themselves

• This project address the strategies and coordination protocols necessary to enable this technology

• Defense against enemy subs is the number one FNC that is called for in every briefing since 2000.

Bear UUV Being DevelopedLt Tulio Celano III, USN

UUV Resistance Curve

0

10

20

30

40

50

60

0 2 4 6 8 10 12 14 16

Water Speed (knots)

To

tal

Dra

g (

lbf)

UUV Range-Speed Curve

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20

Water Speed (knots)

Ran

ge

(NM

) (B

lue)

0

5

10

15

20

25

30

35

40

45

50

Ho

urs

on

Sta

tio

n (

hr)

(P

ink)

8 ft X 10 in., 400 lbs. displacement, upto 100 ft. depth, Top speed 12 knots, effective cruising speed 5 knots, endurance at 5 knots is 45 hours, Modular design

Prior PEG Experience and Lessons Learned

• Pursuit evasion games (PEGs) have been demonstrated within our group using unmanned aerial and ground vehicle (UAVs and UGVs) in both two and three dimensions.

• Additionally, they have also been performed in both symmetrical and asymmetrical games

Previous UC Berkeley PEG Experiments

• Berkeley Aerobot Project (BEAR)– Goal: to build a coordinated, intelligent network with multiple

heterogeneous agents• 11 Rotorcraft-based unmanned aerial vehicles (UAVs) • 5 Unmanned ground vehicles (UGVs)• Shipdeck simulator (landing platform)

• Stochastic Pursuit-Evasion Games (PEG)– Self-localization– Target detection– Map building– Pursuit policy– Trajectory generation– Control / Action

Previously: PEGs with 4 UGVs and 1 UAV

• Sub-problems for Pursuit Evasion Games– Sensing

• Navigation sensors -> Self-localization• Detection of objects of interest

– Framework for communication and data flow

– Map building of environments and evaders• How to incorporate sensed data into agents’ belief states • probability distribution over the state space of the world• (I.e. possible configuration of locations of agents

and obstacles)• How to update belief states

– Strategy planning • Computation of pursuit policy• mapping from the belief state to the action space

– Control / Action

PEG Experiment with UAV/UGVsPEG with four UGVs• Global-Max pursuit policy• Simulated camera view

(radius 7.5m with 50degree conic view)• Pursuer=0.3m/s Evader=0.5m/s MAX

Evaluation of Policies for different visibility

• Global max policy performs better than greedy, since the greedy policy selects movements based only on local considerations.

• Both policies perform better with the trapezoidal view, since the camera rotates fast enough to compensate the narrow field of view.

Capture time of greedy and glo-max for the different region of visibility

of pursuers

3 Pursuers with trapezoidal or omni-directional view

Randomly moving evader

Evader’s Speed vs. Intelligence

• Having a more intelligent evader increases the capture time

• Harder to capture an intelligent evader at a higher speed

• The capture time of a fast random evader is shorter than that of a slower random evader, when the speed of evader is only slightly higher than that of pursuers.

Capture time for different speeds and levels of intelligence of the evader

3 Pursuers with trapezoidal view & global maximum policy

Max speed of pursuers: 0.3 m/s

SEC Capstone Demo: Fixed-Wing PEGs

• Capstone Demonstrations were proposed to highlight and test the technologies developed in the SEC program

• One was a fixed-wing UAV flight test– 6 participant technology developers (TDs)

• Honeywell, Northrop Grumman, U Minnesota, MIT, Stanford, and UCB/U Colorado/CalTech

– System Integrator was Boeing– OCP would be software framework– A T-33 trainer as UAV surrogate– An F-15 as wingman/opponent

• 13 month schedule May 03 – June 04• UCB Contribution: Fixed-Wing PEGs

Nonlinear Model Predictive Trajectory Control

• Explicitly addresses nonlinear systems with constraints on operation and performance

• A cost minimization problem in the presence of state and input constraints– Control resulting in the minimum cost is determined over a

model predicted horizon

• Previously demonstrated in rotary wing UAVs[1]

[1] H.J. Kim, D.H. Shim and S. Sastry, ACC, 2002

Demo: UCB PEG Development

• 20 – 60 min. games confirm NMPTC feasibility at real-time

– Evader goal: get to final waypoint or avoid evader

– Pursuer goal: ‘target’ evader

• Pursuer and evader restricted to same performance limits

• Planes on the same logical plane, but separated by 6000ft altitude at all times

• Evader and pursuer have a few scenarios

– UAV as evader – UAV can become pursuer

OCP Experiment Controller SnapshotT-33: Evader (yellow)F-15: Pursuer (blue)

Demo: Controller Coverage

• Final waypoint is the major component

• Travel to the waypoint can be interrupted by the actions of the pursuer

– Relative positions– Angle-off-tail– Distance from each other– Proximity to boundaries of the

testing area– Physical limitations of the

aircraft• Velocity, climb/turn rate

• Use a simpler model for predictive behavior under certain conditions

– Still use the Boeing-supplied (DemoSim) model for behavioral simulations

Approx. same timePursuer goes fortarget cone

Evader turns away(regardless of endpoint)

Endpoint

Target cone definition (θ=10˚,d=3 nm)Left: F15 not behind UAV, middle: F15 not pointed at

UAV, right: F15 behind AND pointed at UAV

F15

UAV

F15

UAV

F15

UAV

(b)(c)(a)

F15

UAV

F15

UAV

F15

UAV

(b)(c)(a)

UCB PEA Experiment #1 Test Plan:UAV as Evader

• UAV attempts to cross Scenario Area (SA) from East to West without being targeted by the F15

• UAV “wins” by:– Reaching the RVPT– Not being targeted for 20 minutes

• F15 “wins” by targeting the UAV

• Note: F15 performance is restricted

UCB PEA Experiment #2 Test Plan:UAV as Evader and Pursuer

• UAV attempts to cross Scenario Area (SA) from East to West without being targeted by the F15, however, UAV will attempt to target F15 if suitable conditions arise

• UAV “wins” by:– Reaching the END ZONE– Not being targeted for 20 minutes– Targeting the F15

• F15 “wins” by targeting the UAV

Note: F15 performance is restricted

Flight Test 1 (UAV as evader)

Flight Test 2 (UAV as evader/pursuer)

UUV PEGs: Multiplayer Games

• In littoral waters the pursuit evasion game consists of an enemy submarine attempting to cross a line of UUVs which are protecting an asset

• The enemy submarine has a speed advantage over the blue force UUVs

• UUVs play a role in between a sensor web and a group of pursuers

• Research aimed at determining new approaches to teaming for multi-player games. Current literature focuses exclusively on either Nash or Stackleberg solutions

Multiplayer PEG Challenges

• The research challenge includes extending the strategies to: – Large multi-player teams– Asymmetric platform characteristics– Limited communications– High level of uncertainly

UUV PEG: Approach

• The UUV PEG involves two distinct phases:– Detection phase

• Maximize chances of detection constrained by:– Area to cover

– Number of UUVs available

– Possible evader strategies

– Capabilities of UUVs and evader (sensors and noise signatures)

– Response phase• Maximize chances of catching the pursuer constrained by:

– Capabilities of UUVs and evader (speed, manueverability and communications)

– Number of UUVs available

• These two phases also depend on each other as both must succeed– How to share resources to maximize overall chance of success

– How to overlap the strategies: detectors are responders as well

Multiplayer PEGs: Proposed Solution

• A close analogy is football or other team games:– Multi player

– Initial (global) strategies well defined

– Limited (local) coordination after the snap

• What can we learn?• How can we apply this?• How far does the analogy

go?

Multiplayer PEGs

• Preseason (Off-line precomputed strategy)– Play book:

• Evaluate strategies and configurations that will maximize chance of success based on best estimate of other team’s tactics

– Practice and preseason games:• Test playbook and find problems

• Game time (On-line adaptive strategy)– Choose play based on best knowledge and experience

• Line up (in best detection configuration, not necessarily static)

– Execute the play• Active and reactive actions (respond to detected evader)• Local communication• Adapt to evolving behavior

• Learn from experience, repeat as necessary (Learning by Doing)

Pre-game: Strategies for Detection

• Maximize the chance of detecting the evader• Tradeoffs

– Movement of the pursuer:• Moving quickly covers more area, but

• This makes it easier for evader to see the pursuer and avoid the pursuer

– Sensors:• Using passive sonar reduces the range of detection

• Using active sonar reveals the pursuers location

– Number of pursuers in detector role:• Increases chance of detection

Basic principle: Defense in depth

• May not be the optimal with limited resources,– for instance if there are not

enough UUVs to ensure detection of the evader by the front line.

Options: Zone Defense

• Would this leave seams for an evader to exploit if they have superior sensors, for instance?

• Is communication necessary to make such a zone defense effective?

• Is there an alternative?

Options: Channeling the evader

• A coordinated, heterogenous detection strategy.

• For instance, some pursuers could use a very active strategy that exposed them to intentional detection by the evader, with the intension of “channeling” the evader towards other more passive pursuers.

Pursuer Strategies: Capture

• Speed disadvantage means that simply optimizing the detection probability is not sufficient

• Reachability of UUVs must be known• Communication and coordination will be necessary to

overcome speed disadvantage

Defining the Problem:Basic UUV PEG Scenarios

• Scenario 1– Single evader infiltration– Objectives

• Red: pass through game area undetected• Blue: detect red team only

• Scenario 2– Single evader attack– Objectives

• Red: get within weapons range of some objective• Blue: prevent red attack on objective

– Capabilities• Red: limited number of torpedoes available to attack target or blue team

UUVs• Red: suicide attacks only, must get within effective range

• Scenario 3– Multiple evader attack– Objectives & Capabilities

• Same as Scenario 2

UUV Characteristics in General

• Performance– Speed and acceleration– Maximum rate of turn– Maximum rate of ascent/descent (not symmetric in general)– Maximum depth

• Sensors– Effective range in passive detection mode– Effective range in active detection mode– Deployable sonar buoys

• Communications– Effective communication range (variable in general)

UUV Characteristics, cont.

• Method of attack, including:– Self detonation, effective range and perhaps effectiveness as a

function of range– Missile (i.e. torpedo) capabilities

• Counter measures, including:– Sonar buoys– Noise canisters

• Noise signature as a function of:– Speed– Acceleration– Rate of turn– Rate of ascent/descent– Sensor mode– Communication mode

• And others

The First Problem Definition

• Based on “Scenario 2”:– Single evader attack, many defenders– Objectives

• Red: get within weapons range of some objective• Blue: prevent red attack on objective

– Capabilities• Red team:

– Limited number of torpedoes available to attack target or blue team UUVs

– 3 times speed advantage over the Blue (pursuer) team

• Blue team:– Suicide attacks only, must get within effective range– 3 times manuever (turn rate ) advantage over the Red team– Limited communication range– Passive and active sensors available

The First Problem Definition, cont

• Characteristics– Speed

• 3X advantage to Red Team

– Maneuver advantage• 3X to Blue Team

– Detection a function only of • speed,• communication use • Distance

– Sonar• Active & passive available

– Communications available for each team:

• range a function of power• detectability also a function

of power

0 5 10 15 20 25 300

0.5

1

1.5Noise profile of a UUV

Det

ectio

n ra

nge

(nm

)

Evader speed (knots)

Detection: Strategy Comparison

-25 -20 -15 -10 -5 0 5 10 15 20 25-20

-15

-10

-5

0

5

10

15

20

-20 -10 0 10 20-20

-15

-10

-5

0

5

10

15

20

Detection: Monte Carlo results

• Goals: – Statistical model as a

function of configuration, spacing, etc.

– Test strategies

0 10 20 30 40 50 60 7030

40

50

60

70

80

90

100Detection probability for line abreast

Evader speed (knots)

Pro

babi

lity

of d

etec

tion

(%)

0 10 20 30 40 50 60 7030

40

50

60

70

80

90

100Detection probability for staggered configurationt

Evader speed (knots)

Pro

babi

lity

of d

etec

tion

(%)

0 5 10 15 20 25 300

0.5

1

1.5Noise profile of a UUV

Det

ectio

n ra

nge

(nm

)

Evader speed (knots)

Line abreast Staggered

(Recall: detection function)

Capture: Strategies based on reachability (Mitchell’s Level Set Toolbox)

From Airplane example Still even, turn rate reduced to 1/3

Pursuer speed reduced to 1/3 Pursuer speed reduced to 1/3, turn rate increased by 3

Combined the Strategies: Advances in Game Theory

• These PEGs fit Game Theory descriptions as: – mixed strategy– simultaneous move– multiplayer– coordinated games – games with incomplete information

• Specific tactics can be evaluated to find the equilibria and optimal strategies in certain situations, e.g.:– the best search patterns within a single zone for one UUV,

or for two adjacent zones/UUVs

• Statistical methods or Monte Carlo methods could be used to determine the changes of success for each player

Communication Between UUVs

• The issue of the short range of underwater communications with multiple players is very similar to the problem of ad hoc wireless networks of motes devices

• This experience is directly adaptation to the multiple, coordinated UUV scenarios and used to disseminate information within the team regarding: – Detection of an evader– Likely detection by the other team– Coordination instructions– Strategic commands – etc.

• This is integral into the simulation environment

Future Research

• We are building a group of UUVs at the USNA in Anapolis. These are also being shared with NUWC, Newport

• We will develop a theory of multi-player pursuit evasion games with off-line strategies (using robust optimization, the level set toolbox), on-line modifications of the strategy using Model Predictive Control, and an outerloop of learning

• Expect to have simulation results by June 05, experiments for single UUV by June 05, multiple UUV experiments by June 06