Network Performance Tools

Jeff Boote Internet2/R&D

June 1, 2008 NANOG 43/ Brooklyn, NY

Overview•  BWCTL

•  OWAMP

•  NDT/NPAD

BWCTL:Whatisit?Aresourcealloca=onandschedulingdaemonforarbitra=onofiperftests

ProblemStatement• Userswanttoverifyavailablebandwidthfromtheirsitetoanother.

Methodology– Verifyavailablebandwidthfromeachendpointtopointsinthemiddletodetermineproblemarea.

TypicalSolu=on• Run“iperf”orsimilartoolontwoendpointsandhostsonintermediatepaths

Typicalroadblocks• NeedsoNwareonalltestsystems• Needpermissionsonallsystemsinvolved(usuallyfullshellaccounts*)

• Needtocoordinatetes=ngwithothers*• NeedtorunsoNwareonbothsideswithspecifiedtestparameters*(*BWCTLwasdesignedtohelpwiththese)

Implementa=onApplica=ons

– bwctlddaemon– bwctlclient

OpensourcelicenseanddevelopmentBuiltuponprotocolabstrac=onlibrary

– Supportsone‐offapplica=ons– Allowsauthen=ca=on/policyhookstobeincorporated

Func=onality(bwctl)bwctlclientapplica=onmakesrequeststobothendpointsofatest

–  Communica=oncanbe“open”,“authen=cated”,or“encrypted”(encryptedreservedforfutureuse)

–  Requestsincludearequestfora=meslotaswellasafullparameteriza=onofthetest

–  Thirdpartyrequests–  Ifnoserverisavailableonthelocalhost,clienthandlestestendpoint

–  *Mostly*thesamecommandlineop=onsasiperf(someop=onslimitedornotimplemented.)

Func=onality(bwctld)bwctldoneachtesthost

– Acceptsrequestsfor“iperf”testsincluding=meslotandparametersfortest

– Respondswithatenta=vereserva=onoradeniedmessage

– Reserva=onsbyaclientmustbeconfirmedwitha“startsession”message

– Resource“Broker”– Runstests– Both“sides”oftestgetresults

BWCTLExample

BWCTLData(Dash‐Board)

BWCTLData–PathHistory

ResourceAlloca=on(bwctld)• Eachconnec=onis“classified”(authen=ca=on)• Eachclassifica=onishierarchicalandhasanassociatedsetofhierarchicallimits:

– Connec=onpolicy(allow_open_mode)– Bandwidth(allow_tcp,allow_udp,bandwidth)– Scheduling(dura=on,event_horizon,pending)

•  A=meslotissimplya=me‐dependentresourcethatneedstobeallocatedjustlikeanyotherresource.Itthereforefollowstheresourcealloca=onmodel.

BWCTL:3‐partyInterac=on

BWCTL:NoLocalServer

TesterApplica=ons•  Iperfisprimary“tester”

– Wellknown–widelyused

• Problemsintegra=ngexec’dtool– Serverini=aliza=on(portnumberalloca=on)– errorcondi=ons– Noindica=onofpar=alprogress(Howfullwasthesendbufferwhenthesessionwaskilled?)

• thrulay/nubcpareavailableinlatest‘RC’versionofbwctl

GeneralRequirements• Iperfversion2.0and2.0.2• NTP(ntpd)synchronizedclockonthelocalsystem

–  Usedforscheduling– Moreimportantthaterrorsareaccuratethantheclockitself

• Firewalls:–  Lotsofportsforcommunica=onandtes=ng

• Endhostsmustbetuned!hbp://www.psc.edu/networking/perf_tune.htmlhbp://www‐didc.lbl.gov/TCP‐tuning/buffers.html

SupportedSystems• FreeBSD4.x,5.x• Linux2.4,2.6• (MostrecentversionsofUNIXshouldwork)

Policy/SecurityConsidera=ons• DoSsource

–  ImaginealargenumberofcompromisedBWCTLDserversbeingusedtodirecttraffic

• DoStarget–  Someonemightabempttoaffectsta=s=cswebpagestoseehowmuchimpacttheycanhave

• Resourceconsump=on–  Timeslots

–  Networkbandwidth

PolicyRecommenda=ons• Restric=veforUDP• MoreliberalforTCPtests

• Moreliberals=llfor“peers”

• ProtectAESkeys!

Availability• Currentlyavailablehbp://e2epi.internet2.edu/bwctl/

Maillists:

• bwctl‐users@internet2.edu• bwctl‐announce@internet2.eduhbps://mail.internet2.edu/wws/lists/engineering

OWAMP:Whatisit?OWDorOne‐WayPING

• Acontrolprotocol• Atestprotocol• Asampleimplementa=onofboth

WhytheOWAMPprotocol?• Findproblemsinthenetwork

–  Conges=onusuallyhappensinonedirec=onfirst…–  Rou=ng(asymmetric,orjustchanges)–  SNMPpollingintervalsmaskhighqueuelevelsthatac=veprobescanshow

• Therehavebeenmanyimplementa=onstodoOne‐Waydelayovertheyears(Surveyor,Ripe…)

–  Theproblemhasbeeninteroperability.–  hbp://www.ieo.org/rfc/rfc4656.txt

OWAMPControlprotocol• Supportsauthen=ca=onandauthoriza=on• Usedtoconfiguretests

–  Endpointcontrolledportnumbers–  Extremelyconfigurablesendschedule

–  Configurablepacketsizes• Usedtostart/stoptests• Usedtoretrieveresults

–  Provisionsfordealingwithpar=alsessionresults

OWAMPTestprotocol• Packetscanbe“open”,“authen=cated”,or“encrypted”

SampleImplementa=onApplica=ons

– owampddaemon– owpingclient

OpensourcelicenseanddevelopmentBuiltuponprotocolabstrac=onlibrary

– Supportsone‐offapplica=ons– Allowsauthen=ca=on/policyhookstobeincorporated

Func=onality(owpingclient)– owpingclientrequestsOWDtestsfromanOWAMPserver

– Clientcanbesenderorreceiver– Communica=oncanbe“open”,“authen=cated”,or“encrypted”

– Supportsthesetupofmanytestsconcurrently

– Supportsthebufferingofresultsontheserverforlaterretrieval

Func=onality(owampd)owampd

– AcceptsrequestsforOWDtests

– Respondswithaccepted/denied– TestsareformallystartedwithaStartSessionsmessagefromtheclient.

– Runstests– Sessionswithpacketsreceivedattheserverarebufferedforlaterretrieval

OWPINGExample

OWAMPData(Dash‐Board)

OWAMPData(Path)

ResourceAlloca=on• Eachconnec=onis“classified”(authen=ca=on)• Eachclassifica=onisassociatedwithasetofhierarchicallimits

–  Bandwidth(bandwidth)–  Sessionbuffer(disk)–  Datareten=on(delete_on_fetch)–  Connec=onpolicy(allow_open_mode)

(no=medependentdimensiontoresourcealloca=oninowampd)

Architecture

GeneralRequirements• NTP(ntpd)synchronizedclockonthelocalsystem

– Specificconfigura=onrequirementsasspecifiedinNTPtalk…

– Strictlyspeaking,owampwillworkwithoutntp.However,yourresultswillbemeaninglessinmanycases

• gnumakeforbuildprocess

SupportedSystems• FreeBSD4.7+,5.x,6.0(64‐bit)• Linux2.4,2.6(64‐bit)• MacOSX10.4.X

• Solaris10.4.5• (MostrecentversionsofUNIXshouldwork)

RecommendedHardware• StableSystemClock

– Temperaturecontrolledenvironment

– NopowermanagementofCPU

• NostrictrequirementsforCPU,Memory,Busspeed

– Moretaskingscheduleswillrequiremorecapablehardware

Opera=onalconcernsTime:

–  NTPissuespredominatetheproblems

–  Determininganaccurate=mestamp“error”isinmanywaysmoredifficultthangesnga“verygood”=mestamp

–  Workingasan“open”serverrequiresUTC=mesource(Forpredefinedtestpeers,otherop=onsavailable)

Firewalls:–  Portfiltertrade‐off

•  Administratorslikepre‐definedportnumbers•  Vendormanufactureswouldprobablyliketo“priori=ze”testtraffic

•  Owampdallowsarangeofportstobespecifiedforthereceiver

Policy/SecurityConsidera=ons• Third‐PartyDoSsource• DoStarget• Resourceconsump=on

– Memory(primaryandsecondary)– Networkbandwidth

PolicyRecommenda=ons• Restrictoverallbandwidthtosomethingrela=velysmall

– MostOWAMPsessionsdonotrequiremuch

• Limit“open”teststoensuretheydonotinterferewithprecisionofothertests

Availability• Currentlyavailablehbp://e2epi.internet2.edu/owamp/

Maillists:

• owamp‐users@internet2.edu• owamp‐announce@internet2.eduhbps://mail.internet2.edu/wws/lists/engineering

Advancedusertools•  NDT

– Allowsuserstotestnetworkpathforalimitednumberofcommonproblems

•  NPAD– Allowsuserstotestlocalnetworkinfrastructurewhilesimula=ngalongpath

UnderlyingUserAssump=on•  Whenproblemsexist,it’sthenetworksfault!

SimpleNetworkPicture

Bob’s Host

Network Infrastructure

Carol’s Host

Sw

itch

1

Switch 2 Switch 3

NetworkInfrastructure

R1 R3

R4

R2 R7

R6 R9

R8 R5

Switch 4

NDT:Whatisit?Webbrowserinvokedadvanceduserbaseddiagnos=cs

•  Allowsuserstotestanetworkpathforalimitednumberofcommonproblems–fromtheirdesktop

•  NDTallowsusertogivethenetworkadministratoradetailedviewofexactlywhattheusershostisdoing

•  Allowstheusertobeanac=vepar=cipantinthedebuggingprocess–allowsthemtomoredirectlyseehowhostconfigura=oneffectsperformance

Abemptstoanswertheques=ons:Whatperformanceshouldauserexpect?Whatisthelimi=ngfactor?

NDTGoals•  Iden=fyrealproblemsforrealusers

– Networkinfrastructureistheproblem

– Hosttuningissuesaretheproblem

•  Maketoolsimpletouseandunderstand

•  Maketoolusefulforusersandnetworkadministrators

NDTuserinterface•  Web‐basedJAVAappletallowstes=ngfromanybrowser

•  Command‐lineclientallowstes=ngfromremoteloginshell

NDTsampleResults

FindingResultsofInterest

•  DuplexMismatch– Thisisaseriouserrorandnothingwillworkright.Reportedonmainpage,onSta's'cspage,andmismatch:onMoreDetailspage

•  PacketArrivalOrder–  InferredvaluebasedonTCPopera=on.ReportedonSta's'cspage,(withlosssta=s=cs)andorder:valueonMoreDetailspage

FindingResultsofInterest•  PacketLossRates

– CalculatedvaluebasedonTCPopera=on.ReportedonSta's'cspage,(without‐of‐ordersta=s=cs)andloss:valueonMoreDetailspage

•  PathBobleneckCapacity– MeasuredvaluebasedonTCPopera=on.Reportedonmainpage

FindingNDTServers

NPAD/pathdiag:Whatisit?•  Webbrowserinvokedadvanceduserbaseddiagnos=cs

– Allowsuserstotestalimitedpor=onofthenetworkpathlookingforproblemsthatwouldadverselyeffectlongerpaths

– Abemptstoanswertheques=ons:

•  Whatperformanceshouldauserexpect?•  Whatisthelimi=ngfactor?

NPAD/pathdiag•  AnewtoolfromresearchersatPibsburghSupercomputerCenter

•  Findsproblemsthataffectlongnetworkpaths

•  UsesWeb100‐enhancedLinuxbasedserver

•  WebbasedJavaclient

Sw

itch

1

Switch 2 Switch 3

LongPathProblem

R1 R3

R4

R2 R7

R6 R9

R8 R5

Switch 4

H1

H2

H3 X

1 msec H1 – H2 70 msec H1 – H3

LongPathProblem•  E2Eapplica=onperformanceisdependantondistancebetweenhosts

•  Fullsizeframe=meat100Mbps– Frame=1500Bytes– Time=0.12msec

–  Inflightfor1msecRTT=8packets–  Inflightfor70msecRTT=583packets

TCPConges=onAvoidance•  Cutnumberofpacketsby½•  Increaseby1perRTT

– LAN(RTT=1msec)•  Inflightchangesto4packets•  Timetoincreasebackto8is4msec

– WAN(RTT=70msec)•  Inflightchangesto292packets•  Timetoincreasebackto583is20.4seconds

NPADServermainpage

NPADSampleresults

FindingNPADservers

TrythesetoolsNetworkPerformanceToolkit

hbp://e2epi.internet2.edu/network‐performance‐toolkit.html

Knoppixdisk(OSonaCD)thathas:Iperf,thrulay,bwctl,owamp,NDT,NPAD,reverse‐traceroute/ping…

Ques=ons?