juniper vmx a2b internet - more-ip eventswitching hardware (pfe) nic ram ssd 12x1ge 2x10ge 1ge......

ImplementationJuniperNetworksvMX atA2BInternetBy ErikBais– A2BInternet

|MORE-IP2017presentation 2016Page|2

Whatisourbusiness??

oRegistrationofIPaddressesandASnumbers

o IPTransitinvariousDutchdatacenters

o Internet(Fiber)Access&DatacenterNetworkServices

o24*7MonitoringandmanagementofBGPinfrastructure.

o SpecializedconsultancyforISPrelatedtopicslikevendorselections,networkdesign&implementation.

|MORE-IP2017presentation 2016Page|3

Currently inthe following Dutchdatacenters

|MORE-IP2017presentation 2016Page|4

Shortintro

oA2BInternetisaDutchnetworkprovider.§ Providingdatacenterconnectivityandinternetaccessonfiber.

oWeimplementedtheJuniperNetworksvMX solutionrecently§ http://newsroom.juniper.net/press-releases/a2b-internet-deploys-juniper-networks-vmx-as-the-first-virtual-network-function--nyse-jnpr-11g134000-001

oButthe realquestioniswhy govirtual??…

|MORE-IP2017presentation 2016Page|5

Casestudy online

|MORE-IP2017presentation 2016Page|6

Previoussetup

oA2BInternethasalways beenan ExtremeNetworksshop…

oWenoticed that the BGPconverge took too longwith the current DFZsize

o Some updates(especially onthe AMS-IXpeering switch)took waytoo long..§ Slowupdatesofannouncing some prefixes..§ Orevensloweraccepting certain routes..§ Droppingpeers under highBGPload....(AARGGHHH!!)§ And properfiltering,madethings evenworse …

o Limitation wasinasinglethreaded BGPprocess onadual-core CPU

|MORE-IP2017presentation 2016Page|7

Let’shavealookatthevMX

|MORE-IP2017presentation 2016Page|8

|MORE-IP2017presentation 2016Page|9

vMX ProductOverview

VCPVFP

Physical NICs Management traffic

Guest VM (Linux) Guest VM (FreeBSD)

Hypervisor: KVM, ESXi

Cores Memory

Bridge / vSwitch

Physical layerPCI P

ass

thro

ugh

SR-IO

V

Virt

IO

Virtual Control Plane (VCP)• JUNOS hosted in a VM. Offers all the capabilities

available in JUNOS• Management remains the same as physical MX• SMP capable

Virtual Forwarding Plane (VFP)• Virtualized Trio software forwarding plane. Feature

parity with physical MX. Utilizes Intel DPDK libraries• Multi-threaded SMP implementation allows for

elasticity• SR-IOV capable for high throughput • Can be hosted in VM or bare-metal

Orchestration• vMX instance can be orchestrated through OpenStack

Kilo HEAT templates• Package comes with scripts to launch vMX instance

|MORE-IP2017presentation 2016Page|10

Architectural Difference with Shipping NFX250-S2

RIOT VMXT

SwitchingHardware(CrossconnectNIC) NIC RAM SSD

12x1GE 2x10GE 1GE2x10GE(internal)

...

External

X86CPU

LinuxHostOSVFP(PFE) KVMHypervisor

L2_TVPBSDJunos(JCP)

VirtualMachine

JunosDeviceManager (JDM)Container

LinuxBridge

SwitchingHardware(PFE) NIC RAM SSD

12x1GE 2x10GE 1GE

...

External

LinuxHostOS

X86CPU

VirtualControlPlane(VCP)

VirtualMachine

vMX on NFX Native NFX

SameHW

DifferentSW

ApproachvMX

DCPFE LCMD

BCMD

LCMD

KVMHypervisor

LinuxBridge

VNF#2

VNF#3

VNF#N

…vSRX2.0

VNF#1

2x10GE(internal)

|MORE-IP2017presentation 2016Page|11

OurUsedHWkitlist

oWeselectedHPasourvendorforthevMX setup.

oTheusedkitperbox:§ 1*HPProliant DL360gen92xE5-2650v4,64GB,2xPSU§ 2*HP560SFP+10GbePCIe Intelbased82599§ 2*HP240GBSSDHotplug 2.5inchSFF

o InShort..Enoughcore’s,enoughmemory,stickwiththe‘tested/recommendedNIC’s‘andsomeSSD’sforquickerbootingifneeded..

|MORE-IP2017presentation 2016Page|12

VMXlicenses…

oThevMX licensesthatweusearetheAdvanceversion..10G

oThereare1Gband5Gbversionsaswell..

oWedidn’tneedL3VPNor4Mil.routes..(yet)

|MORE-IP2017presentation 2016Page|13

Implementation

oYouneedtofollowtheJuniperimplementationguide..

oYes..RTFM!!..§ http://forums.juniper.net/t5/Day-One-Books/Day-One-vMX-Up-and-Running/ba-p/289129

oTheimplementationisquitepickyinkernelandlibraryversions.

o StartwiththerecommendedUbuntuversion..(notthelatest)

o Skipany idea ofrunningthis onVmWare ..Use Ubuntu+KVM… <period>

|MORE-IP2017presentation 2016Page|14

/home/vmx/vmxlite/config/vmx.conf

o SelecttherightimagestouseinKVMforVMX..

§ #Configurationonthehostside- managementinterface,VMimagesetc.§ HOST:§ identifier :vmx1 #Maximum6characters§ host-management-interface:em1§ routing-engine-image :"/home/vmx/vmxlite/images/junos-vmx-x86-64-16.1R3.10.qcow2"§ routing-engine-hdd :"/home/vmx/vmxlite/images/vmxhdd.img"§ forwarding-engine-image :"/home/vmx/vmxlite/images/vFPC-20161019.img"

|MORE-IP2017presentation 2016Page|15

/home/vmx/vmxlite/config/vmx.confo #vREVMparameterso CONTROL_PLANE:o vcpus :2o memory-mb :4096 #<=- 4Gbisbetterthan2Gb.2Gbworks.o console_port:2211o

o interfaces :o - type :statico ipaddr :<privateIP>o macaddr :"0A:00:DD:B0:DE:0E"

|MORE-IP2017presentation 2016Page|16

/home/vmx/vmxlite/config/vmx.conf

o #vPFEVMparameterso FORWARDING_PLANE:o memory-mb :24576o vcpus :22o console_port:2212o device-type:sriov #<=- YouwantandNEEDSR-IOV...o

o interfaces :o - type :statico ipaddr :<privateIP>o macaddr :"0A:00:DD:B0:DE:10” #<=- StaticMAC’s...Beware!!

|MORE-IP2017presentation 2016Page|17

WhatisSR-IOV?AndwhydoIwantthis?

o Single-rootinput/outputvirtualization

oSR-IOV isa networkinterface thatallowstheisolationofthe PCIExpress resourcesformanageabilityandperformancereasons.AsinglephysicalPCIExpresscanbesharedona virtualenvironment usingtheSR-IOVspecification.

|MORE-IP2017presentation 2016Page|18

SR-IOVallowsfordedicatedaccesstotheNICbytheVM

|MORE-IP2017presentation 2016Page|19

SR-IOVtakeaway…

|MORE-IP2017presentation 2016Page|20

WhatisSR-IOV?AndwhydoIwantthis?

o Single-rootinput/outputvirtualization

oSR-IOV isa networkinterface thatallowstheisolationofthe PCIExpress resourcesformanageabilityandperformancereasons.AsinglephysicalPCIExpresscanbesharedona virtualenvironment usingtheSR-IOVspecification.

ohttps://www.youtube.com/watch?v=hRHsk8Nycdg - IntelSR-IOVExplanation

|MORE-IP2017presentation 2016Page|21

Performanceo Weusea10Gfortransitand2*10Gtoourinternalnetworkpertransitbox.

o PCIExpresscando:

§ Source:Intel- http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005811.html

PCIExpressImplementation EncodedDataRate UnencodedDataRate

x1 5Gb/sec 4 Gb/sec(0.5GB/sec) x4 20 Gb/sec 16 Gb/sec(2GB/sec) x8 40 Gb/sec 32 Gb/sec(4GB/sec) x16 80 Gb/sec 64 Gb/sec (8GB/sec)

Theoretical Maximum Bus Throughput:•PCI Express* (PCIe*) Theoretical Bi-Directional Bus Throughput.

|MORE-IP2017presentation 2016Page|22

TheIntelNIC

o Specifications:

§ HostInterface§ n PCIe BaseSpecification2.0(2.5GT/s)or(5GT/s)§ n Buswidth— x1,x2,x4,x8

oOurnetworkcardshave2*10GbSFP+..Andpercard40Gbps theoreticalthroughput..

o2NIC’sperbox..Whichleavesenoughroomforline-rateperformance…

|MORE-IP2017presentation 2016Page|23

Intothenetwork…oWestartedbymigratingIPv6ontothevMX’s beforeIPv4.

o OnceIPv6wasrunningwithoutanyissues....Wescheduledav4migrationintovMX.

o Transitsmigrationwasreallysimple..§ ThevMX’s feellikeyouareworkingonanactualJuniperMX.

oMigratingallthepeersonthepeeringbox,wasabitmorework.MostlyduetolegacypeerswithMD5orspecificroute-mapsforcertainpeers.

o Afullv4BGPtableloadisdonewithin4seconds!!§ NoneedtoprogramASICS/TCAM..Allroutesareusablewhenloaded..

|MORE-IP2017presentation 2016Page|24

Currentmaxbandwidthsincethemigration:

oAMS-IX10Glink:8.2GboTransit10Gblink:6.3Gb

oCPUusageAMS-IXrouter:

|MORE-IP2017presentation 2016Page|25

Futurepath?

o Itisalwayspossibletopastetheexactsameconfig intoaJuniperMX240orbigger...

oYoucanstarthere… and your development(automation)can be ported into anyother Junos environment.

oNextversions ofvMX will supportQSFP’s and 100GbNIC’s …

oAnd if you don’t likeit oroutgrow the setup,the hw can be re-used for other tasks …

|MORE-IP2017presentation 2016Page|26

Questions?