network simulation and emulation in a testbed era tom ... · uw ee colloquium feb. 2009 1 network...

UW EE Colloquium Feb. 2009 1

Network Simulation and Emulationin a Testbed Era

Tom HendersonUniversity of Washington

The Boeing Company

UW EE ColloquiumFebruary 24, 2009

Engineering, Operations & Technology | Boeing Research & Technology

Copyright © 2009 Boeing. All rights reserved.

Networked Systems Technology

Information Assurance

Digital Communications &

Network Technology

Information Management and Transformation

Applied Mathematics

Architecture and Networked Systems

Technology

Contract Research and Development

Programs

Boeing Research & Technology organization

• Development of high assurance multi-level security technologies

– EAL7 common criteria evaluation experience

– Formal methods in support of EAL 7 evaluation

• High assurance guard solutions for edge, core, & tactical environments

• Security modeling & formal method analysis of end to end systems

• Policy-based secure computing– Systems of Systems

IA approach• Intrusion detection correlation

technologies• Secure workflow and SOA

technologies• Information operations• System security engineering• Advanced trusted system

implementations• Text analysis for information

assurance• Proprietary program support in IA

technologies• Secure RFID infrastructure• Anti-tampering technologies


• Phased array antennas• Free air laser

communications• High capacity RF

communications• Bandwidth efficient

modulation• On board wireless

communications• Network processors• AFRL RAPID• Integrated communications

systems• MANET protocols• Reliable routing• Network management for

tactical environments• Auto-configuration• Network gateways• End to end QoS• Network security• UAV network router

• Data & Text Mining• Human Language technologies• Decision support information• Information visualization• Machine translation• Knowledge representation• Data integration• Simulation visualization• 3D visualization• Model-based collaboration• Social networks analysis• Intelligent graphics & docs.• Collab. human robot systems• Autonomous systems

– Architectures– World modeling– Multi-vehicle behaviors

• Reasoning• Adaptive planning• Learning • Distributed multi-agents • Global information sharing• RFID info. mgmt. services• Adaptive intelligent info. integration• Semantic modeling, ontologies,

adaptive workflow modeling• Quality of Service info. algorithms• Wireless mobile dynamic info. mgmt.• Sensor networks• Data & information fusion• Info. assessment aids• Trends analysis/intent inferencing• Situation assessment• Resource allocation techniques• Deconfliction algorithms• Course of action analysis/generation

• Variation analysis & reduction• Reliability & time dependent data

analysis• Analysis of large datasets• Financial modeling & decisions• Survey & test design• Statistical training• Quality assurance & Lean+• Design of Experiments• Computer-aided geometric design• Numerical optimization• Design space exploration• Multidisciplinary Design

Optimization • Geometric modeling• Multivariate data fitting• Optimal control• Engineering modeling & analysis• Computational mechanics• Sensor Fabric design & operation• System dynamics & control

modeling• Data structures for complex

product families• Kalman filter applications• Operations research• Systems of Systems modeling &

analysis• Discrete/network/stochastic/

Systems of Systems optimization• Data fusion• Discrete-event simulation• Transportation, manufacturing,

business, communication & military system modeling

• UML architecting• Arch. analysis• CONOPS development• System requirements• NR-KPP compliance• SoSE• Systems engineering• NCO demo.• LVC technologies• LVC tools• DIS router• Data link translation• Gateways• Networked manufacturing• Location services• Planning / visibility tools• Functional integration• High performance computing• GRID computing• Resource management• Functional integration• CINS lab• System/network mgmt.• QoS• Mobile agents• Situation awareness• Reconfigurable computing• S/W architecture• S/W processes• Arch. evaluation (ATAM)• Performance assessment

(PAPM)• QoS

• Real-time system development• Tactical info. mgmt. with an

emphasis in C2/tactical collaboration

• Quality of Service network utilization

• Software enabled control for UAV automation

• Dynamic network management• Affordable validation &

verification of flight critical software & systems

• Collaborative anticipatory systems

• Cyberwarfare / Cybercraft• Effects-based operations• System & vehicle health

management• Diagnostics• InfoCenter & information

management• Database technology &

implementation• Distributed sensor system

technology

Homeorganization

UW EE Colloquium Feb. 2009

What is a network simulator?

• A software tool and set of models used to evaluate computer network behavior

• A discrete-event network simulator describes networks in terms of a sequence of discrete events– The simulation kernel temporally orders and

dispatches events to models– Models generate output or more events– Time advances in discrete steps


Application

The basic model

Application

Protocolstack

Node

NetDeviceNetDevice

ApplicationApplication

Protocolstack

Node

NetDevice

Channel

Packet

Simul


The basic model

// Define your topologyNode n0 = new Node;Node n1 = new Node;AddInternetStack (n0, n1);Channel c0 = new Channel;.Connect (n0, n1, c0);Application a0 = new TrafficGenerator;

// Configure thingsa0.SetDataRate (1Mb/s);a0.Start (10.0 seconds);

// Define outputsWriteTraceFile (“outfile”);

// Run the simulatorSimulator::Run();


Network simulation remains popular

• Many commercial products– e.g. OPNET, QualNet, Shunra VE, Matlab Simulink,

OMNet++, NCTUns, SSFNet, Extend

• A few free, open source projects– ns-2 is the most widely used such tool– roughly 10,000 downloads/month

• Widely used in research

Source: Search of ACM Digital Library papers citing simulation, 2001-04


Skepticism in the research community

“For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;”

Nick McKeown, VINI public review, ACM Sigcomm 2006


What has changed?

• Simulators have historically been used for experiments in place of real machines– A tradeoff of realism for scale

But now...• Cost of test hardware has plummeted

• Virtualization technologies proliferate


Emulab (University of Utah)

• Source: http://www.emulab.net


Emulab

• Original Emulab allowed experiments of roughly ~100 nodes– FreeBSD Dummynet and VLANs to interconnect the

nodes

• Emulab now offers virtualization– Experiments of up to ~2000 (virtual) nodes– Requires new approaches for experimental

configuration

• Also, some 802.11 wireless nodesSource: M. Hibler et al., “Large-scale Virtualization in the Emulab

Network Testbed,” Proc. 2008 Usenix Technical Conference.


PlanetLab: Global-scale Internet testbed

• Source: http://www.planet-lab.org

PlanetLab “slices”


PL-VINI and Trellis

• PL-VINI adds virtual networking capabilities (via network overlay) to PlanetLab– A. Bavier et al, “In VINI Veritas: Realistic and

Controlled Network Experimentation,” Proc. ACM Sigcomm, 2006

• Trellis is a kernel-based implementation of the same concept– S. Bhatia et al, “Trellis: A Platform for Building

Flexible, Fast Virtual Networks on Commodity Hardware,” ROADS Workshop, Dec. 2008


ORBIT (Rutgers WINLAB)

• Source: http://www.winlab.rutgers.edu/docs/focus/ORBIT.html

• 802.11a/b/g radios

• Bluetooth

• GNU/USRP radios

• Noise generators


Simulation “out of the loop”

• Source: NSF GENI project: http://www.geni.net/docs/GENIOvrvw092908.pdf


Skepticism in the research community

“For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;”

Nick McKeown, VINI public review, ACM Sigcomm 2006


Overheard* on e2e-interest mailing list

“...Tragedy of the Commons...”“...around 50% of the papers appeared to be...

bogus...”“Who has ever validated NS2 code?”“To be honest, I'm still not sure whether I will use a

simulation in a paper.”“...I will have a hard time accepting or advocating

the use of NS-2 or any other simulation tool”...

* September 2005 archives of the e2e-interest mailing list


Questions

• Do we need to simulate networks anymore?

• What kind of network simulator is needed?

• Why should you believe network simulation results?


Why simulate?

• Wireless– Reproducibility– Fidelity (especially, real-time constraints)– Radios may not exist or be available– Field tests in realistic conditions cost $$

• Scalability– 10,000+ nodes?– For smaller configurations, execution time

Simulation is often an appropriate tool for the job


Challenges for ns (and simulators)‏

• Align with how research is now conducted

• Improve credibility

Can a new simulator help with these problems?


What is ns-3?

An open source project building a new network simulator to replace ns-2


ns-3 project goal

Develop a preferred, open simulation environment for networking research1) a tool aligned with the simulation needs of

modern networking research2) an open-source project that encourages

community contribution, peer review, and validation of the software


Relationship to ns-2

ns-3 is a new simulator, without backward compatibility

Similarities to ns-2:• C++ software core• GNU GPLv2 licensing• ported ns-2 models: random variables, error models,

OLSR, Calendar Queue scheduler, (more planned)

Differences: • Python scripting (or C++ programs) replaces OTcl• most of the core rewritten• also based on the yans and GTNetS simulators• new animators, configuration tools, etc. are in work


Funding support

ns-3 project is supported by:• NSF CISE CRI awards (2006-10)

– PIs: Tom Henderson and Sumit Roy (University of Washington), George Riley (Georgia Institute of Technology)

– NSF CNS-0551686 (University of Washington)

• INRIA Planete and the French governmentWith additional support from:• 2008 Google Summer of Code• University of Washington• Georgia Institute of Technology

‘


Test and Evaluation Options

• When mathematical analysis alone won’t do

Increasing realism

Increasing complexity

Puresimulation

Simulationcradles

Virtual/Physicaltestbeds

Fieldexperiments

Livenetworks

Test and evaluation options

Can we develop tools to span this space?


Some ns-3 design goals

1. Realism2. Software reuse3. Emulation4. Composition


1) Attention to realism

Research should often involve a mix of simulations and testbed or live experiments

• If the simulator cannot be made to closely model a real system:– hard to compare results or validate the model– hard to reuse software between the two domains

When in doubt, do what implementations do


Attention to realism (example)

An ns-3 Node is a husk of a computer to which applications, stacks, and NICs are added

ApplicationApplication

Application

“DTN”


Realism (cont.)

• Align ns-3 to be more faithful representation of real computers– sockets API– packets are “packed” binary representations

ready for direct serialization to/from network devices

– alignment with Linux architecture– multiple interfaces handled properly


ns-3 follows a Linux architecture

classNetDevice

classIpv4Protocol

• Linux corollaries– each NetDevice is modelled

like struct net_device

– each layer-3 protocol has a struct in_devicecomponent for each NetDevice

– this interface (transmit side) is aligned with Linux dev_queue_xmit()

– on receive side, demux is modelled after Linux protocol handlers


2) software integration

Do not reimplement models and tools for which open-source implementations abound

• ns-3 conforms to standard input/output formats so that other tools can be reused.– e.g., pcap trace output, ns-2 mobility scripts

• ns-3 is adding support for running implementation code– Network Simulation Cradle integration has met with

success: Linux TCP code– ns-3 “process” API


ns-3 simulation cradle

• Port by Florian Westphal of Sam Jansen’s Ph.D. work

Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.


ns-3 simulation cradle

For ns-3:• Linux 2.6.18• Linux 2.6.26• Linux 2.6.28

Others:• FreeBSD 5• lwip 1.3• OpenBSD 3

Other simulators:• ns-2• OmNET++

Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.


ns-3 “processes” and Posix API

Support for a synchronous Posix socket API• each Posix type and function is redefined in the simulator• processes get their own private stack

– somewhat like a lightweight virtual machine

• Benefits:– makes porting real world application code much easier– makes writing applications easier because the BSD socket API is

faithfully followed

• see the “mathieu/ns-3-simu” code repository


3) emulation support

Support moving between simulation and testbeds or live systems

• A real-time scheduler, and support for two modes of emulation


ns-3 emulation modes

virtualmachine ns-3

virtualmachine

1) ns-3 interconnects real or virtual machines

realmachine

ns-3

TestbedTestbed

realmachine

ns-3

2) testbeds interconnect ns-3 stacks

real machine

Various hybrids of the above are possible


Example: ORBIT and ns-3

• Support for use of Rutgers WINLAB ORBIT radio grid


4) Composition

ns-3 is a component of an overall workflow

Define ns-3 withinterfaces to supportintegration with othertools

• Topology generators• Traffic generators• Data analysis frameworks• Configuration tools• Animators




Common Open Research Emulator

Scalable Network Emulator• Network lab “in a box”

• Efficient and scalable• Easy-to-use GUI canvas

• Kernel-level networking efficiency• Reference passing packet sending

• Runs real binary code• No need to modify applications

• Connects with real networks• Hardware-in-the-loop• Distributed - runs on multiple servers• Virtual nodes process real packets

• Fork of the IMUNES project• University of Zagreb

• Open Source• http://cs.itd.nrl.navy.mil/work/core




netgraphsystem

Tcl/Tk GUIcore_wlan

core_spanCOREAPI

ng_wlan

NIC

FreeBSDkernel

virtual images(vimages)

tunnels

userspace

Overview of CORE Components

Three key components

1. virtual machines

3. graphical user interface

2. networking subsystem




CORE features

IMUNES base:• Flexible GUI-based management of FreeBSD lightweight

virtual containers• Virtual topology can be hooked to real devicesCORE extensions:• GUI enhancements, kernel enhancements, wireless

networking, Linux support (OpenVz and NetEm), distributed emulation, and packaging.

• Software leads: Jeff Ahrenholz and Claudiu Danilov, Boeing• http://cs.itd.nrl.navy.mil/work/core




CORE and ns-3 integration

ns-3 tap-wifi-dumbbell.cc program


ns-3 and research alignment

In summary, make it easier to move from simulation to emulation to experiments

• Realism and alignment with popular interfaces• Support use of real code• Emulation capabilities• Combine ns-3 with other tools


Background on simulation credibility

[1] “Why We STILL Don’t Know How to Simulate Networks”– Mostafa Ammar, Georgia Institute of Technology, Annual

Simulation Symposium 2005

[2] “Maintaining a Critical Attitude Towards Simulation Results”– Sally Floyd, WNS2 Workshop Keynote, October 2006

[3] “MANET Simulation Studies: The Incredibles”– Kurkowski, Camp, and Colagrosso, ACM Sigmobile, MC2R,

Volume 9, Issue 4, October 2005

[4] “An Integrated Approach to Evaluating Simulation Credibility”– Muessig, Laack, and Wrobleski, U.S. Naval Air Warfare Center,

August 2001


Criteria for Credibility

• Repeatable• Unbiased• Realistic Scenarios• Statistically Sound• Model Accuracy• Results Accuracy (Validation)• Data Accuracy• Usability

from [3]

from [4]


Repeatability

A hallmark of the scientific method...

• Papers should identify simulator, version, operating system, parameters, etc.

• Better yet, make code and configuration scripts available to the community– Yet, 0 out of 84 ACM Mobihoc MANET

simulation papers (2000-2004) referenced publicly available code (from [3])


ns-3 approach

• We are adding support to systematically document how simulations were conducted– A configuration subsystem that dumps

simulation parameters to a configuration file (input and output)

• We will host code, publications, validation results, etc.


Model accuracy

• “error-free-ness” of software and models

• ns-3 goals here:– Support real code where possible– Open source models– Maintainers


Open source models

• “Given enough eyeballs, all bugs are shallow”– Eric Raymond, “The Cathedral and the Bazaar”

• ns-3 needs ways to certify models, too– capture level of community acceptance– publication lists, cross-reference– need to identify maintainers, or state the

absence of a maintainer– validation techniques and results


Example: ns-3 Wifi development

Several research groups are maturing the original INRIA model:• Karlsruhe Institute of Technology: 802.11 PHY, 802.11e

– Equalizing PHY models including capture effects, user-definable coding rates (e.g. 5.9 GHz from 802.11p), EDCA QoS extensions of 802.11e, Nakagami/Rayleigh propagation loss model

• University of Florence: 802.11n features – Frame Aggregation, Block ACK, HCF (EDCA and support for

HCCA),TXOP, HT terminal (also with protection modes), MIMO

• Russian Academy of Sciences: 802.11s– a complete model of IEEE802.11s D2.0 Draft Standard

• Deutsche Telekom Laboratories in Berlin: 802.11 PHY• Boeing: 802.11b channel models, validation• (and others...)


Pledge break

• ns-3 needs participation from the research community1) improving simulation credibility2) contributed and supported models3) maintainers

Please support your open source projects!


Summary of simulation credibility

• Learn from good and bad examples of simulation research, to produce credible simulations

• Consider open source (or publishing of models and scripts) to be integral part of your research

• Please give back to the simulators that you use


Acknowledgments

Thanks to:• the core development team and research project leads

– Raj Bhattacharjea, Gustavo Carneiro, Walid Dabbous, Craig Dowell, Joe Kopena, Mathieu Lacage (software lead), George Riley, Sumit Roy

• 2008 Google Summer of Code mentors and students• many code authors and testers• the ns-2 PIs and developers for creating ns-2 and for

supporting ns-3 activities• USC ISI for hosting project mailing listsContact: [email protected] (office M354)

network simulation and emulation in a testbed era tom ... · uw ee colloquium feb. 2009 1 network...

Documents