CSE 222A: Computer Communication NetworksAlex C. Snoeren

Lecture 7:Data Center Networks

Thanks: Nick Feamster

Lecture 7 Overview● Project discussion

● Data Centers overview

● “Fat Tree” paper discussion

2CSE 222A – Lecture 7: Data Center Networks

Traditional DC TopologyInternet

Servers

Layer-2 switchAccess

Data Center

Layer-2/3 switchAggregation

Layer-3 routerCore

CSE 222A – Lecture 7: Data Center Networks 3

DC Network Requirements● Scalability

u Incremental build out?● Reliability

u Loop free forwarding● VM migration

● Reasonable management burdenu Humans in the loop?

CSE 222A – Lecture 7: Data Center Networks 4

● Over subscription of links higher up in the topology

● Tradeoff between cost and provisioning

● Single point of failure

Traditional Topologies

CSE 222A – Lecture 7: Data Center Networks 5

Capacity Bottlenecks

CR CR

AR AR AR AR

SS

SS

A AA …

SS

A AA …

. . .

SS

SS

A AA …

SS

A AA …

~ 5:1

~ 40:1

~ 200:1

CSE 222A – Lecture 7: Data Center Networks 6

Management: L2 vs. L3● Ethernet switching (layer 2)

u Cheaper switch equipmentu Fixed addresses and auto-configurationu Seamless mobility, migration, and failover

● IP routing (layer 3)u Scalability through hierarchical addressingu Efficiency through shortest-path routingu Multipath routing through equal-cost multipath

● Data centers often connect layer-2 islands by IP routers

CSE 222A – Lecture 7: Data Center Networks 7

Advantages of Layer 2● Certain monitoring apps require server with same role

to be on the same VLAN

● Using same IP on dual homed servers

● Allows organic growth of server farms

● VM migration is easier

CSE 222A – Lecture 7: Data Center Networks 8

Layer 2 Pods w/L3 Backbone

CR CR

AR AR AR AR. . .

SS

DC-Layer 3

Internet

SS

A AA …

SS

A AA …

. . .

DC-Layer 2

Key• CR = Core Router (L3)• AR = Access Router (L3)• S = Ethernet Switch (L2)• A = Rack of app. servers

~ 1,000 servers/pod == IP subnet

S S S S

SS

CSE 222A – Lecture 7: Data Center Networks 9

FAT Tree-Based Solution● An all Layer-3 solution

● Connect end-host together using a “fat-tree” topology u Infrastructure consist of cheap devices

» Each port supports same speed as endhostu All devices can transmit at line speed if packets are distributed

along existing pathsu A k-port fat tree can support k3/4 hosts

CSE 222A – Lecture 7: Data Center Networks 10

“Fat-Tree” Topology

CSE 222A – Lecture 7: Data Center Networks 11

Fat-Tree Challenges● Layer 3 will only use one of the existing equal cost

paths● Packet re-ordering occurs if layer 3 blindly takes

advantage of path diversityu E.g., ECMP

CSE 222A – Lecture 7: Data Center Networks 12

Modified Fat Tree● Enforce special addressing scheme in DC

u Allows host attached to same switch to route only through switch

u Allows inter-pod traffic to stay within podu unused.PodNumber.switchnumber.Endhost

● Use two level look-ups to distribute traffic and maintain packet ordering.

CSE 222A – Lecture 7: Data Center Networks 13

Two-Level Lookups● First level is prefix lookup

u Used to route down the topology to endhost● Second level is a suffix lookup

u Used to route up towards coreu Diffuses and spreads out trafficu Maintains packet ordering by using the same ports for the

same endhost

CSE 222A – Lecture 7: Data Center Networks 14

Diffusion Optimizations● Flow classification

u Eliminates local congestionu Assign to traffic to ports on a per-flow basis instead of a per-

host basis

● Flow schedulingu Eliminates global congestionu Prevent long lived flows from sharing the same linksu Assign long lived flows to different links

CSE 222A – Lecture 7: Data Center Networks 15

Discussion● VM mobility

● Data center build out

● Internet connectivity

● Address space allocationu Doing flow placement anyway?

CSE 222A – Lecture 7: Data Center Networks 16

● Flat addressing and self-learning enablesplug-and-play networking

● Permanent and location independent addresses also simplifyu Host mobilityu Access-control policiesu Network troubleshooting

The Beauty of Ethernet

17CSE 222A – Lecture 7: Data Center Networks

● Flooding-based deliveryu Frames to unknown destinations are flooded

● Broadcasting for basic serviceu Bootstrapping relies on broadcasting

u Vulnerable to resource exhaustion attacks

● Inefficient forwarding pathsu Loops are fatal due to broadcast storms; uses STPu Forwarding along a single tree leads to

inefficiency and lower utilization

Ethernet Scaling Challenges

18CSE 222A – Lecture 7: Data Center Networks

Enterprise networks comprised of Ethernet-based IP subnets interconnected by routers

R

R

R

R

Ethernet Bridging- Flat addressing- Self-learning- Flooding- Forwarding along a tree

IP Routing (e.g., OSPF)- Hierarchical addressing- Subnet configuration- Host configuration- Forwarding along shortest paths

R

Broadcast Domain(LAN or VLAN)

Traditional L2/L3 Model

19CSE 222A – Lecture 7: Data Center Networks

For Next Class…

● Read and review PortLand paper

● Work on project proposalsu Details available on the course Web page

20CSE 222A – Lecture 7: Data Center Networks