datacenter basics fred chong 290n green computing
TRANSCRIPT
![Page 1: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/1.jpg)
Datacenter Basics
Fred Chong290N Green Computing
![Page 2: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/2.jpg)
Storage Hierarchy
Figure : Storage hierarchy of a Warehouse-scale computer
![Page 3: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/3.jpg)
Performance Variations
Figure : Latency, bandwidth and capacity of a Warehouse-scale computer
![Page 4: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/4.jpg)
Server ComparisonHP Integrity Superdome – Itanium2
HP ProLiant ML350 G5
Processor 64 sockets, 128 cores (dual-threaded), 1.6GHz Itanium2, 12MB last-level cache
1 socket, quad-core, 2.66GHz X5355 CPU, 8MB last-level cache
Memory 2048GB 24GBDisk storage 320,974GB, 7056 drives 3961GB, 105 drivesTPC-C price/performance
$2.93/tpmC $0.73/tpmC
price/performance (server HW only)
$1.28/transactions per minute $0.10/transactions per minute
price/performance (server HW only) (no discounts)
$2.39/transactions per minute $0.12/transactions per minute
![Page 5: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/5.jpg)
![Page 6: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/6.jpg)
Cost proportional to Power
• Cost proportional to power delivered• Typically $10-20/W• Power delivery
– 60-400kV transmission lines– 10-20kV medium voltage– 110-480V low voltage
![Page 7: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/7.jpg)
UPS
• Uninterruptible Power Supply• Batteries or flywheel• AC-DC-AC conversion• Conditions the power feed
– Removes spikes or sags– Removes harmonic distortions
• Housed in a separate UPS room• Sizes range from hundreds of kW to 2MW
![Page 8: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/8.jpg)
PDUs
• Power Distribution Units• Breaker panels
– Input 200-480V– Output many 110V or 220V– 75-225kW in 20-30A circuits (max 6 kW)
• Redundancy from two independent power sources
![Page 9: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/9.jpg)
Paralleling
• Multiple generators or UPSs – Feed a shared bus
• N+1 (one failure)• N+2 (one maintenance, one failure)• 2N (redundant pairs)
![Page 10: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/10.jpg)
Cooling
![Page 11: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/11.jpg)
Cooling Steps
• 12-14 C coolant• 16-20 C air at CRAC (Computer Room AC)• 18-22 C at server intake• Then back to chiller
![Page 12: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/12.jpg)
“Free Cooling”
• Pre-cool coolant before chiller• Water-based cooling towers use evaporation
– Works in moderate climate – freezes if too cold• Glycol-based radiator outside the building
– Works in cold climates
![Page 13: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/13.jpg)
Cooling is Critical
• Datacenter would fail in minutes without cooling– Cooling backed up by generators and UPSs
• Adds > 40% critical electrical load
![Page 14: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/14.jpg)
Airflow
• 100 cfm (cubic feet per minute) per server• 10 servers would require 1000 cfm from
perforated tiles• Typically no more than 150-200W / sq ft
power density• Recirculation from one server’s hot air into the
intake of a neighbor– Some avoid with overhead ducts
![Page 15: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/15.jpg)
Variations
• In-rack cooling – Water cooled coils next on the server– Cost of plumbing– Damage from leaks (earthquake zones!)
• Container-based datacenters– Shipping container 8’ x 8.5’ x 40’– Similar to in-rack cooling but for the whole
container– Higher power densities
![Page 16: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/16.jpg)
Power Efficiency
• PUE – power usage efficiency– Datacenter power infrastructure
![Page 17: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/17.jpg)
Poor PUEs
• 85% of datacenters PUE > 3• Only 5% PUE = 2.0• Chillers take 30-50% overhead• CRAC 10-30% overhead• UPS 7-12% overhead (AC-DC-AC)• Humidifiers, PDUs, lighting • EPA “achievable” PUE of 1.4 by 2011
![Page 18: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/18.jpg)
![Page 19: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/19.jpg)
Improvements
• Evaporative cooling• Efficient air movement• Eliminate power conversion losses• Google PUE = 1.21• Several companies PUE = 1.3
![Page 20: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/20.jpg)
A more comprehensive metric
)()()(
11
cbaComponentsElectronictoEnergyTotal
nComputatio
SPUEPUEEnergyTotal
nComputatioEfficiency
• (b) SPUE – server power usage efficiency• (c) computation energy efficiency
![Page 21: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/21.jpg)
SPUE
• Power delivered to components directly involved in computation:– Motherboad, disks, CPUs, DRAM, I/O cards
• Losses due to power supplies, fans, voltage regulators
• SPUE of 1.6-1.8 common– Power supplies less than 80% efficient– Voltage regulators less than 70% efficient
• EPA feasible SPUE < 1.2 in 2011
![Page 22: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/22.jpg)
TPUE
• Total PUE = TPUE = PUE * SPUE• Average of 3.2 today (2.2 Watts wasted for
every Watt in computation)• PUE 1.2 and SPUE 1.2 would give 2X benefit• TPUE of 1.25 probably the limit of what is
economically feasible
![Page 23: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/23.jpg)
Computing Efficiency
• Area of greatest potential• Hardest to measure• SPECpower• Joulesort• Storage Network Industry Association
![Page 24: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/24.jpg)
SPECPower Example
![Page 25: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/25.jpg)
Server Load
![Page 26: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/26.jpg)
Load vs Efficiency
0
20
40
60
80
100
120
140
160
180
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Load level (% of peak)
(%)
Pwr50 Pwr10 Pwr10sub Eff50 Eff10 Eff10sub
![Page 27: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/27.jpg)
Human Dynamic Range
Ax
ch
op
pin
g
Ru
nn
ing
Ju
mp
ing
ro
pe
Cy
clin
g (
rac
e)
La
mb
ad
a
Te
nn
is
So
cc
er
Wa
lkin
g
La
un
dry
Ba
llro
om
Co
ok
ing
Bo
wlin
g
Ty
pin
g
Pla
yin
g c
ard
s
Kn
itti
ng
Sle
ep
0
500
1000
1500
2000
Wat
ts
![Page 28: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/28.jpg)
Component Efficiency
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Idle 7 14 21 29 36 43 50 57 64 71 79 86 93 100
Compute load (%)
Po
we
r (%
of
pe
ak
)
CPU DRAM Disk Other
![Page 29: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/29.jpg)
CPU Voltage Scaling
0
10
20
30
40
50
60
70
80
90
100
Idle 7 14 21 29 36 43 50 57 64 71 79 86 93 100
Compute load (%)
Po
we
r (%
of
pe
ak
)
1 GHz
1.8 GHz
2.4 GHz
DVS savings (%)
![Page 30: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/30.jpg)
Disks
• As much as 70% power to keep drives spinning• 1000X penalty to spin up and access• Multiple head, low RPM drives [Gurumurthi]
![Page 31: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/31.jpg)
Server Power Supplies
![Page 32: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/32.jpg)
Power Provisioning
• $10-22 per deployed IT Watt• Given 10 year depreciation cycle
– $1-2.20 per Watt per year• Assume $0.07 per kilowatt-hr and PUE 2.0
– 8766 hours in a year– (8766 / 1000) * $0.07 * 2.0 = $1.22724
• Up to 2X cost in provisioning• eg. 50% full datacenter = 2X provisioning cost
![Page 33: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/33.jpg)
Time at Power Level
80 servers800 servers8000 servers
![Page 34: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/34.jpg)
Oversubscription Opportunity
• 7% for racks (80)• 22% for PDUs (800)• 28% for clusters (8000)
– Could have hosted almost 40% more machines
![Page 35: Datacenter Basics Fred Chong 290N Green Computing](https://reader034.vdocument.in/reader034/viewer/2022052321/55178f8b5503463e368b55d3/html5/thumbnails/35.jpg)
Underdeployment
• New facilities plan for growth• Also discretization of capacity
– Eg 2.5kW circuit may have four 520W servers• 17% underutilized, but can’t have one more