data center architecture -...

Ankit SinglaData center architecture

2

A server rack

A rack of servers

A top-of-rack switch

3

Lots of racks

How to network the racks?

”

“Big switch” approach

4

…

“Big switch” approach

5

…

Jupiter Rising: A Decade of Clos Topologies and

Centralized Control in Google’s Datacenter Network

Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat

Google, Inc.j it i @ l

ACM SIGCOMM, 2015

512 ports x 1G

40 servers x 1G

4 ports x 1G

610:1 “oversubscription”

“Big switch” approach

7

$$$$$$$$$$ $$$$$$$$$$

…

$$$ $$$

Alternative: tree network

8

9

Connect many cheap, identical switches?

Goals: high capacity, low latency

If you know your application …

10

… design for it

11

NERSC

But, other apps may not work well …

12

13

What’s so hard about this?6-port switches

4-port switches

�

���

�����

�����

�����

������

������

� � � �� �� �� �� �� ��

������������������������

������ �� ��������

14

What’s so hard about this?

15

Clos networks

Use small, cheap elements to build large networks!!

Fat-tree

17

Fat-tree network

Pod 0

10.0.2.1

10.0.1.1

Pod 1 Pod 3Pod 210.2.0.2 10.2.0.3

10.2.0.1

10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2

10.2.2.1

10.0.1.2

ACM SIGCOMM, 2008

A Scalable, Commodity Data Center Network Architecture

Mohammad Al-Fares Alexander Loukissas Amin Vahdat

Servers

Core

18

Fat-tree network

19

Jupiter Rising: A Decade of Clos Topologies and

Centralized Control in Google’s Datacenter Network

Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat

Google, Inc.j it i @ l

ACM SIGCOMM, 2015

[Image: Robert Harker]

[Image: Robert Harker]

22

Jupiter Rising: A Decade of Clos Topologies and

Centralized Control in Google’s Datacenter Network

Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat

Google, Inc.j it i @ l

ACM SIGCOMM, 2015

23

Cabling[Robert Harker

24

DC architecture

How to network the racks?

But can we do better?

25

Other topologies besides fat-trees?

6-port switches

4-port switches

�

���

�����

����������������������

� � � �� �� �� �� �� ��������������������������

������ �� ��������

Surprise: picking randomly from this space works better than fat-trees!

Deterministic expander constructions can also achieve these gains

But do we need full bandwidth everywhere at all times?

25% is the high mark for overall utilization (?)  ”

  2012.

  Facebook

27

hotspots

Skew in network traffic

29

How does this observation help?

“Unfortunately, given current data center network architectures, the only way to provision required bandwidth bandwidth between dynamically changing sets of nodes is to build a non-blocking switch fabric at the scale of an entire data center”

.

Set up network connections on the fly?

Servers

ToR

switch

Aggregate

switch

Core

switch

Electrical

Network

Optical

Network

Reconfigurable

optical paths

Array of Micromirrors

Diffracted beam Towards destinationReceived beam

Input beam LasersDMDs

Photodetectors

Mirror assembly Reflected beam

TX

RX

Rack%1%Rack%N%Rack%r%

Steerable%%FSOs%

Ceiling%mirror%

ToR%switch%

FireFly%Controller%

Traffic%Pa=erns%

Rule%change%

FSO%reconf%

Can statically wired networks compete with this?

31

DC architecture @ SIGCOMM 2017RotorNet: A Scalable, Low-complexity,

Optical Datacenter Network

William M. Mellette, Rob McGuinness, Arjun Roy, Alex Forencich,

George Papen, Alex C.Snoeren, and George Porter

University of California, San Diego

A

T

a

l

e

o

f

T

w

o

T

o

p

o

l

o

g

i

e

s

:

E

x

p

l

o

r

i

n

g

C

o

n

v

e

r

t

i

b

l

e

D

a

t

a

C

e

n

t

e

r

N

e

t

w

o

r

k

A

r

c

h

i

t

e

c

t

u

r

e

s

w

i

t

h

F

l

a

t

-

t

r

e

e

Yiting Xia, Xiaoye Steven Sun, Simbarashe Dzinamarira,

Dingming Wu, Xin Sunny Huang, T. S. Eugene Ng

Rice University

Beyond fat-trees without antennae, mirrors, and disco-balls

Simon KassingETH Zürich

simon.kassing@inf.ethz.ch

Asaf ValadarskyHebrew University of Jerusalem

asaf.valadarsky@mail.huji.ac.il

Gal ShahafHebrew University of Jerusalem

gal.shahaf@mail.huji.ac.il

Michael SchapiraHebrew University of Jerusalem

schapiram@huji.ac.il

Ankit SinglaETH Zürich

ankit.singla@inf.ethz.ch