Symbiotic Routing in Future Data Centers

Symbiotic Routingin Future Data CentersHussam Abu-Libdeh, Paolo Costa, Antony Rowstron,Greg OShea, Austin DonnellyCornell University Microsoft Research Cambridge1

Data center networkingNetwork principles evolved from Internet systemsMultiple administrative domainsHeterogeneous environment

But data centers are differentSingle administrative domainsTotal control over all operational aspects

Re-examine the network in this new setting22Performance IsolationBandwidthFault ToleranceGraceful DegradationScalabilityTCOCommodity Components. . .Modular DesignRethinking DC networksNew proposals for data center network architecturesDCell, BCube, Fat-tree, VL2, PortLand

Network interface has not changed!

Network Interface33ChallengeThe network is a black box to applicationsMust infer network propertiesLocality, congestion, failure etcLittle or no control over routing

Applications are a black box to the networkMust infer flow propertiesE.g. Traffic engineering/Hedera

In consequenceTodays data centers and proposals use a single protocolRouting trade-offs made in an application-agnostic wayE.g. Latency, throughput, etc

44CamCubeA new data center design

Nodes are commodity x86 servers with local storageContainer-based model 1,500-2,500 servers

Direct-connect 3D torus topologySix Ethernet ports / serverServers have (x,y,z) coordinatesDefines coordinate spaceSimple 1-hop APISend/receive packets to/from 1-hop neighboursNot using TCP/IP

Everything is a serviceRun on all servers

Multi-hop routing is a serviceSimple link state protocolRoute packets along shortest paths from source to destination5

(0,2,0)xyz5Development experienceBuilt many data center services on CamCube

E.g.High-throughput transport serviceDesired property: high throughputLarge-file multicast serviceDesired property: low link loadAggregation serviceDesired property: distribute computation load over serversDistributed object cache serviceDesired property: per-key caches, low path stretch

66Per-service routing protocolsHigher flexibility

Services optimize for different objectivesHigh throughput transport disjoint pathsIncreases throughput

File multicast non-disjoint pathsDecreases network load

77What is the benefit?Prototype Testbed27 servers, 3x3x3 CamCubeQuad core, 4 GB RAM, six 1Gbps Ethernet ports

Large-scale packet-level discrete event simulator8,000 servers, 20x20x20 CamCube1Gbps links

Service code runs unmodified on cluster and simulator8

Service-level benefitsHigh throughput transport service1 sender 2000 receiversSequential iteration10,000 packets/flow1500 bytes/packet

Metric: throughputShown: custom/base ratio99Service-level benefitsLarge-file multicast service8,000-server network1 multicast groupGroup size: 0% 100% of servers

Metric: # of links in multicast treeShown: custom/base ratio

1010Service-level benefitsDistributed object cache service8,000-server network8,000,000 key-object pairsEvenly distributed among servers800,000 lookups100 lookups per serverKeys picked by Zipf distribution1 primary + 8 replicas per keyReplicas unpopulated initially

Metric: path length to nearest hit1111Network impactRan all services simultaneouslyNo correlation in link usageReduction in link utilization

Take-away: custom routing reduced network load and increased service-level performance1212Symbiotic routing relationsMultiple routing protocols running concurrentlyRouting state shared with base routing protocol

ServicesUse one or more routing protocolsUse base protocol to simplify their custom protocols

Network failuresHandled by base protocolServices route for common case

13NetworkBase Routing ProtocolRouting Protocol 1Routing Protocol 2Routing Protocol 3Service AService BService C13Building a routing frameworkSimplify building custom routing protocols

Routing:Build routes from set of intermediate pointsCoordinates in the coordinate spaceServices provide forwarding function FFramework routes between intermediate pointsUse base routing serviceConsistently remap coordinate space on node failure

Queuing:Services manage packet queues per linkFair queuing between services per link14Fpacketlocal coordnext coord14Example: cache serviceDistributed key-object caching

Key-space mapped onto CamCube coordinate space

Per-key cachesEvenly distributed across coordinate spaceCache coordinates easily computable based on key1515Cache service routingRoutingSource nearest cache or primaryOn cache miss: cache primaryPopulate cache: primary cache

F function computed atSourceCachePrimary

Different packets can use different linksAccommodate network conditionsE.g. congestion16F

v

F

F

v

vvsource/queriernearest cacheprimary serverOn link failureBase protocol routes around failure

On replica server failureKey space consistently remappedby framework

F function does not changeDeveloper only targets common caseFramework handles corner casesHandling failures17

F

vsource/queriernearest cacheprimary server17Cache service F functionprotectedoverrideListF(intneighborIndex,ulongcurrentDestinationKey,Packetpacket){ListnextKeys=newList();ulongitemKey=LookupPacket.GetItemKey(packet);ulongsourceKey=LookupPacket.GetSourceKey(packet);

if(currentDestinationKey==sourceKey)//amIthesource?{//getthelistofcaches(usingKeyValueStorestaticmethod)ulong[]cachesKey=ServiceKeyValueStore.GetCaches(itemKey);

//iterateoverallcachenodesandkeeptheclosestonesintminDistance=int.MaxValue;foreach(ulongcacheKeyincachesKey){intdistance=node.nodeid.DistanceTo(LongKeyToKeyCoord(cacheKey));if(distance