Datacenter Network Design: Performance/PowerComparison of Large-Scale Network Configurations and a

Way to Avoid it!

Christina DelimitrouElectrical Engineering

DepartmentStanford University

Stanford, CA, USA, 94305cdel@stanford.edu

Milad MohammadiElectrical Engineering

DepartmentStanford University

Stanford, CA, USA, 94305milad@stanford.edu

Frank Austin NothaftElectrical Engineering

DepartmentStanford University

Stanford, CA, USA, 94305fnothaft@stanford.edu

Laura SharplessElectrical Engineering

DepartmentStanford University

Stanford, CA, USA, 94305lsharpless@stanford.edu

ABSTRACTIn this paper, we present a datacenter interconnction net-work topology that performs at the minimal cost-latency-power product point of all Dragonfly routers in our designspace. With a 1.2x cost savings and a 4x power savingsover other Dragonfly designs at a 1.2x performance hit, ourtopology and choice of routing function are effective optimalfor a modern datacenter where design choices are centeredaround capital and operational expenses.

1. INTRODUCTIONAs more and more computing is occuring in larger and

larger clusters, the traffic that flows between the nodes inthe cluster has grown, and the importance of the networksthat connect these nodes has increased drastically. As dis-cussed in [5], datacenters are prone to network collapses(TCP incast) when using a standard TCP/IP based net-working system. This TCP incast problem causes networkbandwidth within the datacenter to fall drastically, causinguser visible application-level latency to decrease. Besidesthe TCP incast problem, a TCP/IP based networking so-lution is suboptimal in datacenters due to the (generally)short lived traffic flows that run inside the datacenter. Dueto the TCP window’s slow start, short lived flows frequentlydo not achieve peak bandwidth and are difficult to model.[6]

The aforementioned problems with the TCP/IP stack indatacenters has driven research into the design of special

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 2011 ACM X-XXXXX-XX-X/XX/XX ...$10.00.

purpose interconnection networks for data centers, similarto those in a conventional supercomputer. Although thedesign of a custom interconnection network may be moreexpensive than the implementation of a normal network us-ing COTS parts, a well designed interconnection networkcan provide substantially improved performance and lowerthe power consumed by the network. In this project, we ex-plored the Dragonfly topology as explained in [7]. Althoughmost current datacenters use a fat tree topology, the Drag-onfly topology provides improved cost and increased faulttolerance due to link redundancy. This is especially signifi-cant, due to the tree saturation issues that are found in fattrees. [8]

As mentioned above, we chose a dragonfly based topologydue to it’s improved costs. As discussed in [7], the use of highradix routers in a dragonfly topology serves to decrease costby minimizing global channels, which dominate the costs ofoperating a datacenter. While high radix routers had previ-ously been frowned upon because they increased the lengthof the longest link in the datacenter, improvements in fiberoptics technology have made long, high speed links feasi-ble. [9] Besides the improvements in high speed links thathave made high radix networks possible, datacenter topol-ogy research had been constrained to TCP/IP networking.Although TCP/IP is commonly used, it has large faults.Some of the seminal datacenter networking papers acknowl-edge that one of the significant reasons to choose a fat treetopology is that it allows for preventing loops from occuringwith TCP/IP traffic. [3][11]

2. DESIGN OVERVIEW - TOPOLOGYThe state-of-the-art topology deployed in large-scale data

centers today is the fat tree [2, 3, 4]. The reason behind thisimplementation is simplicity and good load balancing prop-erties due to randomized routing. However, fat tree topolo-gies are by no means the optimal for large-scale systems. Inthis work we implement a dragonfly topology to improve thenetwork’s latency by reducing the number of hops between

1m

0.25m

0.25m

R R R R

R R R R

SHARED

SHARED

3m

Figure 1: The topology of one group. Each row iscomprised by 9 racks, and the whole group has 18racks. Each square represents one rack, and eachsquare with a dot, is a rack with a router. Routersare shared among neighboring racks, as shown inFigure 3.

19 Columns

19 R

ows

Figure 2: The floorplan of the datacenter, with eachcomponent representing 9 racks.

any two nodes, and the Total Cost of Ownership by reduc-ing the number of optical cables used to connect differentgroups.

In this section, we present the design overview of our net-work, the layout of the data center as well as the configura-tion of the topology in terms of endpoints per router, routersper group and total number of groups in the system.

For our network configuration we have chosen to imple-ment a topology that promotes energy and cost consider-ations to a first-order design constraint. We try to mini-mize the use of expensive optical cables, and maintain a lownumber of routers to reduce energy dissipation, and cost.Although this design will experience lower performance com-pared to networks that optimize strictly for throughput andlatency, we have seen that quantifying the price of efficiencyoffers useful insight in provisioning trade-offs which are im-portant challenges in current data centers. In Section 6, weperform a comparative study between different configura-tions in terms of Performance/Watt and Performance/$. Inthat study our configuration ranks among the top designswhen the figure of merit is not strictly performance, butpower and cost as well.

Based on our topology, each group is comprised of 18racks, with 9 racks in each of two rows (Figure 3. Eachrack has 32 nodes, and each router services 71 endpoints(p=71). This means that in each group there are 8 routers

Router #0 Router #1 Router #2 Router #3

Figure 3: Router sharing across different racks.Nodes of the same color are serviced by the samerouter.

(a=8), and the total number of groups is 177 (g=177). Fi-nally, the radix of the router that we use is k=100, out ofwhich 71 links are devoted to the intra-group electrical con-nections, 7 to the routers of the same group, and 22 to theinter-group optical links. Routers within a group are sharedacross racks in a way that strives to keep the distance fromany node to its router as uniform as possible, in an effort tominimize the variability of latency across nodes of a group.Figure 3 shows this design, where nodes of different colorare serviced by different routers. This scheme is symmetricevery 8 racks, and although not currently adopted in datacenters, could assist with network latency variability, whichis a significant challenge for large-scale systems.

One disadvantage of this configuration is the oversubscrip-tion of the connections to the optical cables, i.e. if more than22 endpoints simultaneously need to communicate to exter-nal groups, the optical links will become congested. De-screasing the size of the groups and as a result increasingthe number of optical links is expected to solve this prob-lem, although at the cost of a higher power consumption andTCO. All connections within a group are performed via elec-tric links, and any router can communicate with any otherrouter in the same group with a single electric wire, withoutrepeaters, based on the Manhattan distance (i.e. no diag-onal wires crossing the isles). One additional optimizationthat can be deployed for cost optimality is replacing someof the optical wires with electrical. Given the maximumlength of electric wires without the use of repeaters (5m),neighboring groups of up to two groups away can be con-nected solely using electric wires, thus significantly reducingthe dominant cost factor, which is the number of opticallinks in the system.

3. ROUTINGIn determining the routing function that we wanted to

implement, we considered Minimal (MIN), Uniform Glob-ally Adaptive Load-Balanced routing (UGAL), ProgressiveAdaptive Routing (PAR), and Piggyback (PB). With anunbalanced offered load on different parts of the networkdue to hotspot traffic and varying sizes of packets, the rout-ing function needed to be adaptive, which eliminated MIN.UGAL is a less adaptive routing scheme than PAR, so thelimited extra cost of implementing PAR made it more at-tractive.Choosing between PAR and PB was determined byease of implementation in the simulator: PAR was reason-

ably straightforward to implement while PB required sig-nificant changes to the routers. These led to PAR beingselected as our routing function.

3.1 Progressive Adaptive RoutingPAR aims to route traffic along the shortest, least con-

gested path to the destination. At every hop in the sourcegroup, the congestion along the minimal path is evaluatedagainst congestion along a non-minimal path. The non-minimal path is determined by choosing an intermediatenode (in line with Valiant’s algorithm) to route to beforerouting to the destination. When the minimal path hasgreater congestion than the non-minimal path, the packetis diverted along the non-minimal path. From this decisionpoint, the packet will route minimally to the intermediatenode and then minimally to the destination node. Thus, ateach router within the source group, the current route canbe re-evaluated to determine the best overall route for thepacket.

3.2 ImplementationAt each router, when a head flit is being routed, its source

group is compared against the current router’s group. Ifthey are equal and the packet is being routed minimally,then the packet originated from the current group and therouter can decide whether to route the packet minimally ornon-minimally.

Routing non-minimally requires an intermediate node toroute through, as per Valiant’s algorithm, so a random nodein the network is chosen as the intermediate. The numberof hops to reach the destination by routing through the in-termediate node, along with the queue length for routing tothe intermediate node at the router are compared to thosefor continuing to route minimally:

HopCountmin ∗QueueLengthmin <=

HopCountnon−min ∗QueueLengthnon−min + Threshold

(1)

Given that the apparent congestion for routing throughthe intermediate node is less than the congestion encoun-tered routing minimally, non-minimal routing begins. Oncea packet starts routing non-minimally, it cannot switch backto minimal routing, so loops can’t be created. But once apacket has an intermediate node chosen, it will be routedminimally to that node, and then routed minimally to thedestination from there.

3.2.1 Virtual ChannelsThere are four virtual channels used in our implementa-

tion of PAR to avoid deadlock. These are labeled in Figure4 for reference.

Within the source group, packets are routed minimally onVC0. When packets begin to be routed non-minimally, theyswitch to VC1. This prevents packets that are routed backalong the same path from deadlocking the network. As pack-ets are routed to the intermediate node, they use VC1, andwhen packets arrive at the intermediate node, they switchto use VC2. All packets being routed to their destinationminimally use VC2. Thus, when minimally routed packetsleave their source group, they route via VC2 on the opticalcable. When packets reach the destination group, they swapto route on VC3, since they have crossed the network date-

Src DstVC0 VC2

VC1

VC2

VC2

VC3

VC3

Source Group Destination Group

Intermediate Group

Figure 4: PAR routing using 4 virtual channels.

40 50 60 70 80 90 100123.5

124

124.5

125

125.5

126

126.5

127

127.5

Threshold Value

Ave

rage

Lat

ency

Figure 5: Average packet latency across variousthreshold values.

line. From then on, within the destination group, packetsare routed minimally on VC3.

This scheme allows packets being routed minimally andnon-minimally to be on separate virtual channels, eliminat-ing the threat of deadlock.

3.2.2 ThresholdFigure 11 shows the effect on average packet latency with

varying threshold in all routers across the system. With verylow thresholds, packet latency increases due to many packetsbeing routed non-minimally when congestion isn’t very bad.By increasing the threshold, packet latency drops until athreshold value of 85. Further increasing the threshold skewsthe decision towards minimal routing, at which point latencyplateaus (beyond 100 in Figure 11).

The latency flattens out past 1 threshold of 100 due toour topology. Since we are bandwidth constrained betweengroups, queues for optical links become congested and pack-ets are misrouted. But all of the optical link queues in thegroup are congested, so mis-routing packets increases thehop count without decreasing the latency. Packets will facesimilar congestion at the next router and won’t have bene-fited from being misrouted. Thus, as the threshold increases,biasing the decision towards minimal routing, most pack-

0 500 1000 1500 2000 25000

10

20

30

40

50

60

70

80

Simulation Time (1000 Cycles)

Gen

erat

ion

Rat

e (P

acke

ts p

er C

ycle

)

Figure 6: Number of packets generated in the net-work on a per-cycle basis.

ets are routed minimally unless congestion is very lopsidedwithin a router.

The only downside of setting the threshold very high isthat it doesn’t allow for significant backpressure. With chang-ing traffic patterns, the router will be slow to switch fromminimal to non-minimal routing, increasing the latency ofsome packets. But the provided traffic patterns are reason-ably well-behaved, so backpressure isn’t a critical issue facedby the network.

4. TRAFFIC MANAGEMENTThe main focus of the implementation coding of the sim-

ulator was to correctly handle traffic management. Thissection explains our implementation approach to generatingthe desired mix of network traffic.

4.1 Packet Injection and EjectionAt every cycle, each node attempts to send a packet to

the network. A node successfully sends its packet when ithas fewer than four request packets in flight and when nooutstanding replies are pending to be serviced. These con-dictions avoid congesting the network when older packetsin the network have not yet been serviced, hence avoidinginstability. Figure 6 confirms the stability of our networkby showing the relatively constant rate of packet genera-tion over time. Although the rate of generation fluctuatesbetween cycles, the overall amount of traffic generated isbounded, ensuring that the network doesn’t generate moretraffic than it can process. The flow diagram on Figure 7describes illustrates a simplistic version of our generationscheme.

The four traffic patterns outlined in the project disctip-tion (Uniform, HotSpot, Bad Dragon, and Uniform) are im-plementatecd using four traffic classes. The combination ofthese traffic patterns allowed us to model a realistic networktraffic pattern for our experiments. We use a flit and packetsize of 100B for all of the traffic classes, and number of pack-ets generated by each traffic pattern depend on the actualmessage size.

Larger than 4 in flight

packets?

Any outstanding

replies?

Issue a New Message Packet

Generate a New Packet

RPC with less than 500 packets issued?

No

No

No

Go

to n

ext

endp

oint

Issue a Reply Message

Yes

Yes

Is an RPC Message?

Yes

Yes

No

Figure 7: Traffic management flow control diagram(simplified) basis.

Sice RPC require 500 100B message packets to be sent, weimplemented a feedback system that enables sending 100Bpackets over 500 cycles. If an RPC message is generated bya node, that node will stop issuing new message types untilall the 500 packets are successfully injected into the network.Then, the node generates a new message with the probabil-ities outlined in the projet description. Packet destinatoinsare chosen based on the traffic pattern determined by theirmessage type. For instance, if a message type turns out tobe adverserial RPC, it will assigned as a HotSpot messageclass and a HotSpot traffic generator determines the packetdestination. Once the message type and the destination ofthe packet are determined, the packet is sent over the net-work.

4.2 HotSpotOur implemetation of the HotSpot traffic generator as-

sumes 10 fixed endpoints as the HotSpot nodes. Once amessage type is set to be RPC Adverserial, its destinationis automatically set to be one of the 10 HotSpot noeds thatare present in the network. To model the entire network, wechose to have one of our HotSpot nodes in one of the groupsthat we are simulating and leave the other nine outside ofthe cloud. This way we ensure that one of the two simulatedgroups has hotspot traffic while the other one does not.

As you will see in the next section, including HotSpotnodes in the cloud creates extra complications that requiredus to think more carefully about how the cloud latency can

be calculated.

4.3 SlicingTo slice a netowrk topology, one should take into account

the accuracy of the slice in representing the true character-istics of the network. We assumed the number of possislenodes that can be simulated in a reasonable time is about1000 (1% of the entire network size). We started off bytaking a full group (with 8 routers and 568 endpoints) and54 other groups, each with only router ndoes present forsimulation (no endpoint nodes). While we believed this ap-proach reflect a very accurate model of the network, we soondiscovered the extreme complexity of implementing such astructure in code given the time we had. As a result, wecame up with a simpler topology slice model that can rep-resent the network within our 1000 node simulation budget;we chose to model two groups and a cloud node as it is shownon Figure 9. Below is the description of how we constructedthe cloud network.

To construct the sliced topology, we need 1950 opticallinks in total, and 56 electrical links per group (112 in total)to connect the routers only. We also need a total of 1136electrical links to connect the endpoints to their respectiverouter. The algorithm below guarantees that the network isfully connected:

• Connecting Local Routers (Electrical) - Figure 8:

– If my router ID is < destination router ID: con-nect my link ID to the link ID of the destintionrouter that satisfies dest link ID = my router ID.

– If my router ID is > destination router ID: con-nect my link ID to the link ID of the destintionrouter that satisfies dest link ID = my router (ID-1).

• Connecting Global Link (Optical) - Figure 9: All op-tical links are connected to the cloud node while thelink-ID=799 on router 7 is connected to link 1599 ofrouter 15.

4.4 Cloud AbstractionIdeally, to calculate the cloud response time latency we

would measure the average traffic latency of a packet to betwice the latency of a packet going from one node on toan optical link (t node-to-optical-link). This latency valueincludes an estimate of the routing, wiring, and queueinglatecy. While this approach can give a reasonalble estimateof latency for UR traffic, we belive that it is not effectivewhen multiple traffic patterns are present in the network.This estimate becomes even worse when one of the trafficpatterns is HotSpot. Our solution for getting accurate la-tency results is to dynamically update the average latencyof packets going from any node in the slice to any opticallink (e.g. take the latency average every 20 cycles). Thislatency value takes into account the hotspot traffic latency(present in one of the two groups in the slice). Dependingon the intensity of Hotspot traffic in the network, the cloudaverage response time varies. For this project the HotSpottraffic turned out to be not hot enought to make a largeimpact on the overall latency of the packets traversing theentire data-center.

R0

R1

R2

R3

0

1

2

3

4

5

6

7

8

9

10

11

R0

R1

R2

R3

3

6

9

7

10

0

11

4

1

2

5

8

Figure 8: Intera-group router link connection exam-ple basis.

Cloud

R0 R1 R2

R3 R4 R5

R6 R7

R8 R9 R10

R11 R12 R13

R14 R15Group 0 Group 1

Figure 9: Inter-group router link connections basis.

0 20 40 60 80 100 120 140 160116

118

120

122

124

126

128

130

Simulation Time (1000 Cycles)

Ave

rage

Lat

ency

UR (50K)Bad Dragon (50K)UR (100KB WB)Hotspot

Figure 10: Trace of average latency for the four dif-ferent classes of packets.

The second issue associated with the cloud network is re-lated to deciding on how to send packets in and out of thecloud. Four different traffic scenarios are possible: send-ing packets to the slice (from the cloud), receiving packetsfrom the slice and replying to them, sending packets non-minimally from a slice node to another slice node via thecloud node, and sending packets non-minimally from thecloud to the cloud via an intermediate slice node.

Transmitting packet:

• From the cloud to the slice: In this case, packets aresent to the cloud and their reply is returned accordingto the latency protocol described above. The prob-ability of sending a packet to the slice is about 99%indicatin the need for accurate measurement of the re-ply message latencies from the cloud.

• From the slice to the cloud: As mentioned previously,the radix of the cloud is 350. This means that thetraffic manager attempts to inject up to 350 packetsfrom the cloud into network every cycle. Of all theinjected packets from the cloud, only 1% of them areactually destined to the group in the slice that theyare injected into.

• From cloud to cloud node via the slice: The remain-ing 99% of the packets injected by the cloud are non-minimally routed and their destination is the cloud.When a non-minimal packet is injected into a slice,the packet will take two optical hop and one local hopto return to the cloud.

• From slice to slice via the cloud: less than 1% of thetime packets are routed non-minimally from a slicenode to another slice node. The amount of time thecloud waits to reply to the sender is 2xt node2opt− link.The amount of time the cloud waits to before it injectsthe packet to the destination group/router is 2xt node2optlink+ t local − hop&routing. t local-hop&routing is therouting latency + local link hop latency in the net-work. This value can also be meaured the same wayt node2optlink is measured.

0 1000 2000 3000 4000 5000 60000

500

1000

1500

2000

2500

3000

3500

4000

4500

Simulation Time (1000 Cycles)

Num

ber

of P

acke

ts in

Flig

ht

UR (50K)Bad Dragon (50K)UR (100KB WB)Hotspot

Figure 11: Total number of packets in flight over thesimulation, broken down by packet class.

5. COMPARATIVE DESIGN ANALYSIS:QUANTIFYING THE PRICE OF EFFICIENCY

5.1 Comparison of Performance/Watt and Per-formance/$ between Different DC NetworkConfigurations

In Section 2 we presented the topology implemented in thecurrent design. We discussed the advantages and disadvan-tages of this decision in terms of performance and efficiencyand provided some reasoning as to why it was preferred overother configurations.The scope of this Section is to perform a comparison betweendifferent configurations (e.g. topology, routing) in terms ofPerformance/Watt and Performance/$ and to provide in-sight in the trade-off between performance, efficiency. Inorder to do this, we perform an exhaustive simulation of alldragonfly network configurations that adhere to the design,cost and power constraints of the assignment and plot theranking of each network design in terms of Latency overPower and Latency over Cost.

All simulations are run for 1 million cycles using virtual-cut through flow control, fully simulating around 1100 nodesin the Datacenter. These nodes are divided across a differ-ent number of groups, depending of the configuration thatis simulated. The configuration of the network: the (a, p, g)parameters and the routing algorithm vary across differentdesigns.Additionally evaluating alternative flow control techniques,as well as a comparison between dragonfly and the previ-ously state-of-the-art DC network, i.e. Fat Tree, is deferredto future work.

We calculate the power consumption and cost for the sys-tem for each configuration based on the provided guidelines.Figure shows how each configuration ranks in terms of Per-formance and Efficiency. The y-axis represents latency inms and the x-axis power consumption in MW.

Figure 12 shows how the different configurations rank in

(4, 71, 354)

(8, 71, 177) (18, 41, 135) (20, 40, 125)

(30, 15, 331)

(34, 34, 86)

(100, 50, 20)

(4, 71, 354) (8, 71, 177) (18, 41, 135)

(20, 40, 125)

(34, 34, 86) (34, 34, 86)

(30, 15, 331)

(45, 22.5, 98) (60, 30, 55)

(75, 38, 35)

(90, 45, 24) (94, 47, 22)

(100, 50, 20)

(4, 71, 354)

(8, 71, 177) (18, 41, 135)

(20, 40, 125)

(34, 34, 86)

(34, 34, 86)

(30, 15, 331)

45, 22.5, 98 (60, 30, 55)

(75, 38, 35)

(90, 45, 24)

(94, 47, 22)

(100, 50, 20)

50

70

90

110

130

150

170

190

210

1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3

Late

ncy

(m

s)

Power (MW)

PAR

MIN

UGAL

Figure 12: Performance over Watt comparison between different network configurations (topology and rout-ing).

terms of latency over power consumption. The optimalpoints in the plot are the ones in the bottom left cornernear the pareto-optimal curve, while the worst ones are theones in the top right one. Our configuration ranks amongthe best for Latency/Watt (highlighted region), althoughmore power hungry configurations (e.g. (a, p, g) = (30,15, 331)) achieve lower latency. Also, although designs thatuse PAR tend to have lower latency than those with eitherMIN or UGAL routing functions, it is the configuration ofthe topology than mainly affects latency here (almost anover of magnitude difference between the weights of topol-ogy and routing in latency). There are some configurationsthat achieve lower latency per Watt than our network, andthat brings up the issue of oversubscribing the optical linksin the system. If we retune our design tohave slightly fewerendpoints per node, and more routers per group we expecta shift towards the bottom side of this graph. The bestconfiguration for performance and power is (34, 34, 86) us-ing PAR. We observe that, as expected designs with manyrouters per group tend to have lower latencies, but increasedpower, while networks with few routers per group experiencelower performance.

We note here that the power shown in the graph is cal-culated and not simulated, therefore it does not account forany impact the routing decision might have on energy con-sumption, i.e. configurations that only differ in routing willhave the same power budget.

Similarly for cost, in Figure 13 we show how latency ranksfor different designs over the deployment cost of the network.Based on the provided guidelines, networks with high powerconsumption tend to have high cost as well, therefore thesimilarities between the two graphs are expected. There arehowever a few outliers, e.g. while cost increases monotoni-

cally as we increase the number of routers per group, powerconsumption experiences some unpredictability in its behav-ior.Again, our design ranks highly on the performance-cost plane(highlighted region), with few designs achieving better per-formance, TCO trade-offs by reducing the number of end-points per router, e.g. (34, 34, 86). Designs with high num-bers of routers per group achieve significantly better perfor-mance - and reliability - but are severely hurt in the costaspect.

5.2 Performance Predictive Scheme based onLinear Regression

In this Section we propose a performance predictive schemethat allows estimating the performance of a network config-uration without actually deploying it, either in simulationor in actual implementation. Our scheme is based on sim-ple linear regression and uses the data of the previous studyas the training set to predict the performance of an un-known topology. For example, if we have simulated a setof networks (with different design parameters (a, p, g) anddifferent routing algorithms, or flow-control techniques) wecan extract the feature vector from these parameters anduse the data as a training set to estimate the performanceand/or efficiency and cost of a different configuration.In our current evaluation, we collect statistics for latencyover a long simulation period for a large number of differentconfigurations, and use the configuration parameters, alongwith the routing and flow-control techniques, as the featuresthat train the linear regression. The benefit of this scheme,apart from the insight it provides for different network con-figurations, is the ability to predict the performance (in ei-ther throughput or latency) of a new configuration without

(4, 71, 354)

(8, 71, 177) (18, 41, 135) (20, 40, 125)

(34, 34, 86) (38, 40, 65)

(30, 15, 331)

(45, 22.5, 98)

(60, 30, 55) (75, 38, 35) (90, 45, 24)

(94, 47, 22)

(100, 50, 20)

(4, 71, 354)

(8, 71, 177)

(18, 41, 135)

(20, 40, 125)

(34, 34, 86)

(38, 40, 65)

(30, 15, 331)

(45, 22.5, 98)

(60, 30, 55) (75, 38, 35)

(90, 45, 24) (94, 47, 22)

(100, 50, 20)

(100, 50, 20)

(8, 71, 177) (18, 41, 135)

(20, 40, 125)

(34, 34, 86)

(38, 40, 65)

(30, 15, 331)

(45, 22.5, 98) (60, 30, 55)

(75, 38, 35)

(90, 45, 24)

(94, 47, 22)

(4, 71, 354)

50

70

90

110

130

150

170

190

210

0 1 2 3 4 5 6 7 8 9

Late

ncy

(m

s)

Cost (millions of $)

PAR

MIN

UGAL

Figure 13: Performance over TCO comparison between different network configurations (topology and rout-ing).

the overhead of simulating it, or worse implementing it.This greatly simplifies trimming down the available parame-ter space, which as shown in Section 6.1 is significantly large,while providing accurate estimations for the performance ofthe system.We have used our scheme to predict the latency of newconfigurations based on the behavior of already simulatednetworks and achieve a less than 5% deviation betweenpredicted and simulated performance in all cases. As ourtraining set we use all previously simulated configurations,except for the one currently being evaluated, in order toensure that training and testing set are decoupled. Figure14 shows the comparison between predicted and simulatedperformance (latency) for three configurations that we haveevaluated. In all cases the results from the predictive schemeresemble closely the simulated metrics, which is promisingtowards extending this method to a more complete and ro-bust framework.As part of future work, we plan to evaluate the sensitivity

of our method in the size and divergence of the training set(e.g. number of different simulated configurations, numberof samples per configuration) as well as the number and typeof features in the feature vector. As shown in Figure 5 thefeature that affects the results most is the number of end-points per optical channel, followed by the number of intra-group router connections. This result makes sense giventhat optical channels should not be kept idle, while intra-group router connections should not be oversubscribed, asthis would result in underutilizing the optical links, whichare the expensive resource in the system. Alternatively, onecan use a scheme like this to predict the optimal configura-tion for a network given SLAs, or a power and cost budget.Although getting the exact configuration for the absolute op-timum is an NP-hard problem, linear regression or more so-

0

50

100

150

200

250

(71, 7, 22) (30, 15, 331) (4,71,354)

Late

ncy

Network Configurations

Simulated Predicted

Figure 14: Performance Predictive Scheme Valida-tion: Comparison between predicted and simulatedperformance for three possible network configura-tions.

(4, 71, 354)

(8, 71, 177)

(18, 41, 135)

(20, 40, 125)

(34, 34, 86)

(38, 40, 65)

(30, 15, 331)

(45, 22.5, 98)

(60, 30, 55)

(75, 38, 35)

(90, 45, 24)

(94, 47, 22)

(100, 50, 20)

0

500

1000

1500

2000

2500

0 2 4 6 8 10 12 14

PAR

MIN

UGAL

Figure 15: Performance-power-cost producto fordifferent network configurations (Topology andRouting).

phisticated ML schemes, can offer design suggestions withinan error margin of the optimal configuration.

6. FUTURE WORKAlthough PAR performs better than MIN or UGAL, pig-

gyback routing (as introduced in [10]) would provide a fur-ther advantage over PAR. While both our hotspot trafficand our adversarial traffic pattern wound up being fairly in-nocuous, it is conceivable that other, more adversarial trafficdistributions could be used in a datacenter using this inter-connection network.

Since our goal was to optimize datacenter costs, we aimedto decrease the number of global optical links. Alternatively,we could have switched from optical links to global metallinks using repeaters. Since this would decouple networkcost from the number of global links, we would likely windup with a very different Pareto optimal frontier for our de-sign space. We chose not to perform a design using globalmetal links due to signal integrity concerns (this would haverequired the use of repeaters) which would have vastly com-plicated our process for costing our datacenter.

7. CONCLUSIONSIn this paper, we presented an architecture for a datacen-

ter interconnection network that uses a dragonfly topologyto optimize for cost. Our architecture was designed to op-timize cost through picking a topology that sits in a flatplane for cost. After constructing this cost curve, we per-formed a space exploration to determine which topologiesprovided best performance and power efficiency at a givenprice. Our design space exploration led to the realizationthat our chosen topology is nearly on the optimal frontierfor the performance per cost curve, and is one of the morepower efficient designs. As shown in Figure 15, our designoptimizes the product of latency, power, and cost over allother network configurations of the exhaustive search ex-ploration. Since performance is impacted by the balance oflatency over core links and cost is dominated by global linkcosts, the lone more cost efficient topology chose to havelarger groups with fewer endpoints per router. This topol-

ogy (41 nodes per router, 18 routers per group, 133 groups)had a better balance of traffic between local and global in-terconnects while keeping the number of global links ap-proximately equal. While this new topology has marginallybetter ( 5%) performance-per-cost, it is approximately 30%less power efficient than our original proposed topology (71nodes per router, 8 routers per group, 177 groups).

In modern datacenter installations, capital expendituresand operating costs dominate design choices. By acheivingnear-minimal total component cost in our datacenter andby achieving a very formidable performance-to-power ratio,we have designed a topology that is very practical and veryefficient. Although we do not have maximum performance(based off of the guidelines in [7] as well as our design spaceexploration), we achieve a 4x cost improvement and a 1.2xpower efficiency over the maximum performance router forour topology, and achieve the minimum cost-latency-powerproduct over all topologies in our design space.

8. REFERENCES[1] William J. Dally, Brian Towles. “Principles and

practices of interconnection networks”. MorganKaufmann, 2004.

[2] “InfiniBand in the Enterprise Data Center: Scaling10Gb/s Clustering at Wire-Speed”. White Paper,Mellanox Technologies, 2006.

[3] R.N. Mysore, A. Pamboris, N. Farrington, N. Huang, P.Miri, S. Radhakrishnan, V. Subramanya, and A.Vahdat. “PortLand: A Scalable Fault-Tolerant Layer 2Data Center Network Fabric”, In Proc. ofSIGCOMMaAZADAt’09, August 2009, Barcelona,Spain.

[4] M. Al-Fares, A. Loukissas, A. Vahdat. “A Scalable,Commodity DataCenter Network Architecture”. InProc. of SIGCOMMaAZADAt’08, August 2008, Seattle,Washington, USA.

[5] Chen, Yanpei and Griffith, Rean and Liu, Junda andKatz, Randy H. and Joseph, Anthony D.“Understanding TCP incast throughput collapse indatacenter networks”. In Proceedings of the 1st ACMworkshop on Research on enterprise networking.WREN ’09, Barcelona, Spain, 2009.

[6] Mellia, M. and Zhang, H. “TCP model for short livedflows”. Communications Letters, IEEE. February, 2002.

[7] Kim, J. and Dally, W. and Scott, S. and Abts,D.”Cost-Efficient Dragonfly Topology for Large-ScaleSystems”. IEEE Micro, January - February, 2009.

[8] Vogels, W., D. Follett, J. Hsieh, D. Lifka, and D. Stern.“Tree-Saturation Control in the AC3 Velocity ClusterInterconnect”. White Paper, 2006.

[9] Luxtera. “Fiber Will Displace Copper Sooner Than YouThink”. White Paper, 2005.

[10] Nan Jiang, John Kim, and William J. Dally. 2009.“Indirect adaptive routing on large scale

interconnection networks.” SIGARCH Comput. Archit.News 37, 3 (June 2009), 220-231.

[11] Albert Greenberg, James R. Hamilton, Navendu Jain,Srikanth Kandula, Changhoon Kim, Parantap Lahiri,David A. Maltz, Parveen Patel, and Sudipta Sengupta.2009. VL2: a scalable and flexible data center network.In Proceedings of the ACM SIGCOMM 2009conference on Data communication (SIGCOMM ’09).ACM, New York, NY, USA, 51-62.