60 ghz flyways: adding multi-gbps wireless links to data centers daniel halperin srikanth kandula,...
Post on 19-Dec-2015
225 views
TRANSCRIPT
![Page 1: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/1.jpg)
60 GHz Flyways:Adding multi-Gbps wireless
links to data centers
Daniel HalperinSrikanth Kandula, Jitu PadhyeVictor Bahl, David Wetherall
![Page 2: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/2.jpg)
Presented by Daniel Halperin @SIGCOMM2011
Perform well in average case with
job placement
Bottlenecks in core can be workload
“hotspots”
Today’s data center networksare oversubscribed in the core
2
…
…
![Page 3: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/3.jpg)
Presented by Daniel Halperin @SIGCOMM2011
No core hotspotsNo job placement
Costly switchesComplex wiring
Eliminating oversubscriptionis expensive
3
…
…
![Page 4: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/4.jpg)
Presented by Daniel Halperin @SIGCOMM2011 4
Our goal: Flyways
To enable a network withan oversubscribed core to act like
a non-oversubscribed networkby dynamically injecting
high-bandwidth links.
![Page 5: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/5.jpg)
Presented by Daniel Halperin @SIGCOMM2011
Low cost
Perform well in most cases withjob placement
Dynamically inject links where needed
Our approach: Wireless Flyways
5
…
Oversubscribed Core
![Page 6: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/6.jpg)
Presented by Daniel Halperin @SIGCOMM2011 6
System overview
DCScheduler
JobsData
placement
FlywayController
Demands
![Page 7: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/7.jpg)
Presented by Daniel Halperin @SIGCOMM2011 7
Outline of the rest of this talk
• 60 GHz wireless technology
• Wireless flyways system design
• Evaluation on real data center workloads
![Page 8: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/8.jpg)
Presented by Daniel Halperin @SIGCOMM2011
60 GHz WIRELESS
![Page 9: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/9.jpg)
Presented by Daniel Halperin @SIGCOMM2011 9
60 GHz primer
• 7 GHz of unlicensed spectrum @60 GHz• Forthcoming IEEE 802.11ad: 3 channels,
bitrates to 6.76 Gbps• Challenge:– 60 GHz link has 55 dB (312,000x) worse SNR
than 2.4 GHz link– Directionality is a fundamental mechanism
to enable 60 GHz communication
2.4 GHz
5 GHz
60 GHz
![Page 10: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/10.jpg)
Presented by Daniel Halperin @SIGCOMM2011 10
Directionality is crucial
Goo
dput
(Gbp
s)
123
0
4
0 5 10 15 20Distance (m)
![Page 11: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/11.jpg)
Presented by Daniel Halperin @SIGCOMM2011 11
60 GHz directional technologyPhased Array
Compact (1 in2)Electronic steering (µs)
Physical Antenna
Fixed position
![Page 12: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/12.jpg)
Presented by Daniel Halperin @SIGCOMM2011 12
60 GHz for Flyways
60 GHz links• Multi-Gbps• Directional• Steerable
Flyways must be• Reliable• Densely
deployed
![Page 13: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/13.jpg)
Presented by Daniel Halperin @SIGCOMM2011 13
Directional 60 GHz linksare not robust to blockage
Beam Interrupted
SNR
(dB)
Time (s)
30
15
0
![Page 14: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/14.jpg)
Presented by Daniel Halperin @SIGCOMM2011 14
A 60 GHz link in a data center
![Page 15: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/15.jpg)
Presented by Daniel Halperin @SIGCOMM2011 15
Directional 60 GHz linksare stable in a data center
24h in Data Center
Time (h)
SNR
(dB)
40
20
0
Beam Interrupted
SNR
(dB)
Time (s)
30
15
0
![Page 16: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/16.jpg)
Presented by Daniel Halperin @SIGCOMM2011 16
Measurement-based 802.11ad simulator
• Simulator to evaluate many concurrent flyways– Channel model from indoor/DC RF measurements– Measured 60 GHz antenna patterns– Also compared to 8-element 2.4 GHz “Phocus” array
• Implementation in ns-3– 802.11ad physical layer and protocol– TCP and UDP packet simulations– Dozens of concurrent multi-Gigabit links
![Page 17: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/17.jpg)
Presented by Daniel Halperin @SIGCOMM2011 17
Flyways can be densely deployed
• 160 racks, based onreal DC topology
• Draw random links until no more can be added
• Ensure all links meet rate threshold
• 12-30 links per channel, depending on rate
# Co
ncur
rent
link
s pe
r cha
nnel
Omni Wide Phocus Narrow
![Page 18: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/18.jpg)
Presented by Daniel Halperin @SIGCOMM2011 18
Measurement summary
• 60 GHz offers multi-Gbps, directional, steerable wireless links with IEEE 802.11ad
• Measurements and simulations show– Links are reliable in data centers– With directionality, links can be densely deployed
• Many additional measurements in paper
![Page 19: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/19.jpg)
Presented by Daniel Halperin @SIGCOMM2011
WIRELESS FLYWAYS SYSTEM DESIGN
![Page 20: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/20.jpg)
Presented by Daniel Halperin @SIGCOMM2011 20
System overview
DCScheduler
JobsData
placement
FlywayController
Demands
![Page 21: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/21.jpg)
Presented by Daniel Halperin @SIGCOMM2011 21
Flyway controller architecture
TrafficDemands
Wireless links& Rates
Compute optimal Flyways set
Configure flyways
![Page 22: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/22.jpg)
Presented by Daniel Halperin @SIGCOMM2011 22
Flyway controller architecture
TrafficDemands
Wireless links& Rates
Iteratively choose best flyway
Update
Configure flyways
More?
Y
N
![Page 23: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/23.jpg)
Presented by Daniel Halperin @SIGCOMM2011 23
How to setup links,predict bitrates,
and manage interference
How to select flyways that will improve performance
Flyway controller architecture
Wireless links& Rates
Iteratively choose best flyway
![Page 24: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/24.jpg)
Presented by Daniel Halperin @SIGCOMM2011 24
Leverage the wired backboneto sidestep issues of coordination
Coordinating devices
✘
![Page 25: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/25.jpg)
Presented by Daniel Halperin @SIGCOMM2011 25
Orienting antennas
Traditional algorithms search, e.g. sector sweep
Data center topology is known and stable
![Page 26: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/26.jpg)
Presented by Daniel Halperin @SIGCOMM2011 26
Predicting bitrate
This is hard inmulti-path environments
Directionality alleviatesmulti-path: SNR lookup table
[DIRC, SIGCOMM’09]
Use SINR for interference
![Page 27: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/27.jpg)
Presented by Daniel Halperin @SIGCOMM2011 27
High-efficiency MAC
Offload small reverse TCPpackets to wired network:
+25% wireless goodput
![Page 28: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/28.jpg)
Presented by Daniel Halperin @SIGCOMM2011 28
How to setup links,predict bitrates,
and manage interference
How to select flyways that will improve performance
Flyway controller architecture
Iteratively choose best flyway
Wireless links& Rates
![Page 29: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/29.jpg)
Presented by Daniel Halperin @SIGCOMM2011 29
Selecting flyways: Simple exampleBase 10 Gbps network:• 15 seconds
S1 D S3S2
40Gb
60Gb
50Gb
![Page 30: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/30.jpg)
Presented by Daniel Halperin @SIGCOMM2011 30
S2
“Straggler”: Flyway at largest hotspot
Base 10 Gbps network:• 15 secondsStraggler:• 12.2 seconds
S1 D S3
3 Gbps
40Gb
60Gb
50Gb
![Page 31: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/31.jpg)
Presented by Daniel Halperin @SIGCOMM2011 31
S2
“Transit”: Forward traffic on flyway
Base 10 Gbps network:• 15 secondsStraggler:• 12.2 secondsTransit:• 11.7 seconds
S1 D
3 Gbps
S3
40Gb
60Gb
50Gb
![Page 32: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/32.jpg)
Presented by Daniel Halperin @SIGCOMM2011 32
S2 S3
40Gb
60Gb
50Gb
“Greedy”: Choose faster flywaysBase 10 Gbps network:• 15 secondsStraggler:• 12.2 secondsTransit:• 11.7 secondsGreedy:• 9.4 seconds
S1 D
6 Gbps
![Page 33: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/33.jpg)
Presented by Daniel Halperin @SIGCOMM2011 33
Flyway controller architecture
TrafficDemands
Wireless links& Rates
Iteratively choose best flyway
Update
Configure flyways
More?
Y
N
![Page 34: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/34.jpg)
Presented by Daniel Halperin @SIGCOMM2011
EVALUATION
![Page 35: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/35.jpg)
Presented by Daniel Halperin @SIGCOMM2011 35
Evaluation using real DC workloads
• We studied four live data centers– Mix of applications (Cosmos, IndexSrv, 2xHPC)– Pre-production and production servers
• 76 hours of traces, 114 TB of traffic– Measured application demand
![Page 36: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/36.jpg)
Presented by Daniel Halperin @SIGCOMM2011 36
Source Rack
Des
tinati
on R
ack
Demand
Traffic matrix exampleHypothetical demandmatrix needs full-bisection
![Page 37: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/37.jpg)
Presented by Daniel Halperin @SIGCOMM2011 37
Real traces have localized hotspots
Very few hotspots!
Source Rack
Des
tinati
on R
ack
Affectonly a
few racks
![Page 38: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/38.jpg)
Presented by Daniel Halperin @SIGCOMM2011 38
Evaluation setup
• Evaluated 60 GHz flyways improvementson real demand matrices in an ns-3 topology based on real DC layout
• Metric: Completion time of Demands (CTD)– Relative to non-oversubscribed network– CTD of 1 ➠ same as non-oversubscribed– CTD of 2 ➠ same as 1:2 oversubscribed
![Page 39: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/39.jpg)
Presented by Daniel Halperin @SIGCOMM2011 39
1 flyway device / node
CDFover
DemandMatrices
CTD
CTD < 1.5 for >80% of traces
Smallest possible CTD for
50% of traces
1 1.5 2
![Page 40: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/40.jpg)
Presented by Daniel Halperin @SIGCOMM2011 40
Incremental benefit of strategies
CTD1.5 2
CDFover
DemandMatrices
1
![Page 41: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/41.jpg)
Presented by Daniel Halperin @SIGCOMM2011 41
1-3 devices / node
CTD1.5 2
CDFover
DemandMatrices
1
![Page 42: 60 GHz Flyways: Adding multi-Gbps wireless links to data centers Daniel Halperin Srikanth Kandula, Jitu Padhye Victor Bahl, David Wetherall](https://reader036.vdocument.in/reader036/viewer/2022062421/56649d2c5503460f94a02510/html5/thumbnails/42.jpg)
Presented by Daniel Halperin @SIGCOMM2011
Conclusions
• 60 GHz flyways can substantially improve performance in oversubscribed DC
• Traffic indirection crucial for practical benefit in real workloads
• Novel techniques leverage wired backbone to dramatically simplify and speed hybrid system
Read more: http://r.halper.in/paper/flyways_sigcomm11