U.S. Army Research, Development and Engineering Command

COMBINE

11 Sept 2012

Ken Renard

US Army Research Lab

High Performance Network

Modeling in the US Army Mobile

Network Modeling Institute

(MNMI)

Institute Objectives

Objectives ● Develop and apply HPC software for the

analysis of MANETs in complex environments

● Develop an enabling interdisciplinary computing environment that links models throughout the Simulation, Emulation, and Experimentation cycle

● Leverage the powerful synergistic relationship between simulation, emulation, and experimentation

● Expand DoD workforce that is cross-trained in computational software and network science skills

● Deliver/support software and train the DoD HPC user community, and significantly extend it to key NCW transformation programs

Develop multi-disciplinary expertise/software that transforms the way

DoD models-simulates-emulates-tests mobile networks

Key Technical Barriers

Modeling the Full Protocol Stack. Realistic simulation and emulation of the network requires

looking at every layer of the protocol stack, from the physical, to the medium-access-control,

network, information, and application layers.

RF Propagation and Terrain Effects Dynamic networking must account for RF propagation

performance, yet there is insufficient fidelity to model large-scale mobile networks with realistic

propagation effects in difficult propagation environments (i.e. urban, foliage, mountainous terrain)

and under adverse conditions (i.e. interference, jamming).

Command and Control Traffic and Command Hierarchies: Traffic models for the network must

be accounted for and be directly related to the command and control systems and command

hierarchies.

Modeling Scope: These models must address the full capability of the LandWarNet including its

multiple tiers (terrestrial, airborne, space), its key layers (transport, services, applications, platforms),

and its interactions with the Global Information Grid.

Real-Time Operation: High-fidelity and scalable modeling must run in real-time to enable

hardware-in-the-loop and emulations that are key to minimizing the risks inherent in fielding

complex NCW technologies

4

● Actual hardware in field environment

● Traffic generated from applications

● Realistic scenarios

● HPC used to augment and stimulate

environment

Experimentation

Use theory to define:

● Objective function

● Behavioral relationships

● Parameters

● Variables

Simulation

● PC processors represent nodes

● Laboratory environment

● MANE software to model node movement

and radio access

● Actual MANET protocols

run on nodes

● Applications run on nodes

Emulation

HPC environment to couple

all three phases together

NiCE

‘SEE’ Concept

ARL Emulation Environment

Internet Radio

Recording

.wav file Compression

Packets

Node A and

Emulated Radio

Channels

Record .wav file

Decompression Incorporate delay

Packets in error and delayed

Measure Audio

Quality

Measure

Packet

Delay and

Packet

Error

Caller B

Caller C

A

C

B

Real-Time Mobile Ad-Hoc Network (MANET)

provides a platform for analysis of full applications

including comparison of waveforms, routing

algorithms, antennae, and other radio parameters in a

controllable and repeatable laboratory environment.

Human-in-the-Loop LVC experiments allowed with

real-time RF propagation calculations

Real-Time RF Propagation

Modeling

Free Space

• Simple calculation on CPU

• Does not require digital

terrain data

• Does not consider terrain

• Inaccurate if ground is not

flat

Real Time Path Loss Progress

Pre 2010

2011

2012 and Beyond

ITM (Longley-Rice)

• Efficient GPGPU

implementation (>10x faster

than single core)

• Considers terrain

• Does not consider human

made structures

TLM

• Very efficient GPGPU

computation (60x faster than

single core)

• Typically used for pico-cell

modeling

• Scale as O(n3) with spatial

discretization

Ray Tracing

• Perceived efficiency on

GPGPUs

• Capable of accurately

predicting propagation in

urban environment

• Requires 3-D model of

environment

• Computationally expensive

3D view of GPU accelerated ray-

tracing calculation.

TLM simulation models propagation

of energy through space (the grid)

ITM path-loss

calculation

integrated with

emulation server

NS-3 Performance Testing

Scenario

• Balance of realism and performance

• “Reality is complex and dynamic” vs. “High Performance can be unrealistic”

• Split network into federates (1 federate per core)

• Optimizing inter-federate latency and limiting inter-federate communication

• A “Subnet” is a collection of nodes on the same channel (802.11 or CSMA)

• Typically a team or squad that has similar mobility profile

• Single router in subnet that connects to WAN

• “Router-in-the-sky” that connects subnets via Point-to-Point links

• Ad-hoc 802.11 networks use OLSR routing

• Significant processing and traffic overhead

• Situational Awareness (SA) reported by each node to subnet router

• Random walk within bounded area

OLSR SA

ARP

802.11

Subnet Traffic Distribution 100m x 100m

75%

11%

12%

Simulator Performance

with OLSR

– Packet event rate (per wall-clock time) shows linear

scaling versus number of cores

– Promising results assuming that wireless networks can be

broken into independent federates

Simulator Performance

without OLSR

• Drop-off observed in scaling of CSMA/static routing

– Much less work done per federate

– Workload per grant time not enough to offset increasing time for federate

synchronization [MPI_AllGather()]

• Expect to see this for large enough core counts with larger workloads

(OLSR)

NS-3 Scaling Tests

• MNMI goal is to enable scaling of MANET simulations on the order of 105 nodes

while maintaining high fidelity protocol, traffic, and propagation models.

• Simple scaling tests with NS-3 were conducted to understand effects of

distributed scheduler and MPI interconnect latencies.

– Simple Point-to-Point and CSMA “campus” networks

Campus

Department Department

Net Net Net Net … …

…

…

Campus

Department Department

Net Net Net Net … …

…

…

• UDP packet transfer within and among campuses (campus = federate)

• Only 40% of hosts were communicating during simulation

• 1% of those were communicating across federate boundaries

• IPv6 with static routing

NS-3 Scaling Results

• Achieved best results limiting each compute node to a single federate

– Each compute node has 8 cores and 18G of usable memory

• Largest run:

– Each federate used 1 core and 17.5G on a compute node

– 176 federates (176 compute nodes)

– 360,448,000 simulated nodes

– 413,704.52 packet receive events per second [wall-clock]

NS-3 in Experimentation

• C4ISR-Network Modernization holds annual events to test emerging

tactical network technologies and their suitability for Army deployment

• Live (20-40) and virtual (3k-10k) entities deployed at Ft. Dix, NJ

conducting missions

• Live vehicles and dismounted soldiers have access to range facilities

• Infrastructure provided to measure network performance and connectivity

• Virtual assets constructed in OneSAF environment interact with live

assets

• Gateways connect [and optionally translate]

operational messaging between live and virtual

entities

• Brigade and Battalion TOCs with live C2 systems

NS3 in Experimentation

• Real-time Distributed scheduler developed for NS-3

– Combination of real-time (best-effort) and distributed schedulers

– MPI communication is simplified

• Timing is only synchronized on start

• Only packets are exchanged (w/ delay tolerance)

• DIS interface to other M&S tools

– Forces and ISR Modeling

ARL-

APG

Lab

C4ISR

APG

Lab

Ft Dix

M&S

Tactica

l

DIS/VMF

Bridge

XML Based Interface Definitions

Existing and

New Tools

Existing and

New Tools

Network Interdisciplinary Computing

Environment (aka “the plumbing”)

Scenario

Generator

Scenario

Conversion

Network Simulator (Open Source and/or Commercial)

Emulation

Experiment /

Testing

Visualization

and Analysis