![Page 1: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/1.jpg)
Ankit SinglaData center architecture
![Page 2: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/2.jpg)
2
A server rack
A rack of servers
A top-of-rack switch
![Page 3: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/3.jpg)
3
Lots of racks
How to network the racks?
”
![Page 4: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/4.jpg)
“Big switch” approach
4
…
![Page 5: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/5.jpg)
“Big switch” approach
5
…
![Page 6: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/6.jpg)
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
512 ports x 1G
40 servers x 1G
4 ports x 1G
610:1 “oversubscription”
![Page 7: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/7.jpg)
“Big switch” approach
7
$$$$$$$$$$ $$$$$$$$$$
…
![Page 8: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/8.jpg)
$$$ $$$
Alternative: tree network
8
![Page 9: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/9.jpg)
9
Connect many cheap, identical switches?
Goals: high capacity, low latency
![Page 10: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/10.jpg)
If you know your application …
10
![Page 11: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/11.jpg)
… design for it
11
NERSC
![Page 12: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/12.jpg)
But, other apps may not work well …
12
![Page 13: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/13.jpg)
13
What’s so hard about this?6-port switches
4-port switches
�
���
�����
�����
�����
������
������
� � � �� �� �� �� �� ��
������������������������
������ �� ��������
![Page 14: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/14.jpg)
14
What’s so hard about this?
![Page 15: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/15.jpg)
15
Clos networks
Use small, cheap elements to build large networks!!
![Page 16: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/16.jpg)
Fat-tree
![Page 17: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/17.jpg)
17
Fat-tree network
Pod 0
10.0.2.1
10.0.1.1
Pod 1 Pod 3Pod 210.2.0.2 10.2.0.3
10.2.0.1
10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2
10.2.2.1
10.0.1.2
ACM SIGCOMM, 2008
A Scalable, Commodity Data Center Network Architecture
Mohammad Al-Fares Alexander Loukissas Amin Vahdat
Servers
Core
![Page 18: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/18.jpg)
18
Fat-tree network
![Page 19: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/19.jpg)
19
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
![Page 20: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/20.jpg)
[Image: Robert Harker]
![Page 21: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/21.jpg)
[Image: Robert Harker]
![Page 22: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/22.jpg)
22
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
![Page 23: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/23.jpg)
23
Cabling[Robert Harker
![Page 24: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/24.jpg)
24
DC architecture
How to network the racks?
But can we do better?
![Page 25: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/25.jpg)
25
Other topologies besides fat-trees?
6-port switches
4-port switches
�
���
�����
����������������������
� � � �� �� �� �� �� ��������������������������
������ �� ��������
Surprise: picking randomly from this space works better than fat-trees!
Deterministic expander constructions can also achieve these gains
![Page 26: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/26.jpg)
But do we need full bandwidth everywhere at all times?
![Page 27: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/27.jpg)
25% is the high mark for overall utilization (?) ”
2012.
27
![Page 28: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/28.jpg)
hotspots
Skew in network traffic
![Page 29: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/29.jpg)
29
How does this observation help?
“Unfortunately, given current data center network architectures, the only way to provision required bandwidth bandwidth between dynamically changing sets of nodes is to build a non-blocking switch fabric at the scale of an entire data center”
.
![Page 30: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/30.jpg)
Set up network connections on the fly?
Servers
ToR
switch
Aggregate
switch
Core
switch
Electrical
Network
Optical
Network
Reconfigurable
optical paths
Array of Micromirrors
Diffracted beam Towards destinationReceived beam
Input beam LasersDMDs
Photodetectors
Mirror assembly Reflected beam
TX
RX
Rack%1%Rack%N%Rack%r%
Steerable%%FSOs%
Ceiling%mirror%
ToR%switch%
FireFly%Controller%
Traffic%Pa=erns%
Rule%change%
FSO%reconf%
Can statically wired networks compete with this?
![Page 31: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/31.jpg)
31
DC architecture @ SIGCOMM 2017RotorNet: A Scalable, Low-complexity,
Optical Datacenter Network
William M. Mellette, Rob McGuinness, Arjun Roy, Alex Forencich,
George Papen, Alex C.Snoeren, and George Porter
University of California, San Diego
A
T
a
l
e
o
f
T
w
o
T
o
p
o
l
o
g
i
e
s
:
E
x
p
l
o
r
i
n
g
C
o
n
v
e
r
t
i
b
l
e
D
a
t
a
C
e
n
t
e
r
N
e
t
w
o
r
k
A
r
c
h
i
t
e
c
t
u
r
e
s
w
i
t
h
F
l
a
t
-
t
r
e
e
Yiting Xia, Xiaoye Steven Sun, Simbarashe Dzinamarira,
Dingming Wu, Xin Sunny Huang, T. S. Eugene Ng
Rice University
Beyond fat-trees without antennae, mirrors, and disco-balls
Simon KassingETH Zürich
Asaf ValadarskyHebrew University of Jerusalem
Gal ShahafHebrew University of Jerusalem
Michael SchapiraHebrew University of Jerusalem
Ankit SinglaETH Zürich
![Page 32: Data center architecture - SIGCOMMconferences.sigcomm.org/sigcomm/2017/files/program/topic-preview-2-2.pdf · Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead,](https://reader033.vdocument.in/reader033/viewer/2022041612/5e38743f4c571f549a5776eb/html5/thumbnails/32.jpg)