a100 capacity planning slides
TRANSCRIPT
Data Center PlanningData Center Planning
SpaceSpace PowerPower CoolingCooling Data Connections – CapacityData Connections – Capacity
Redundancy}} Density
Typical Space UsageTypical Space Usage
60%Support
40%Equipment
CYH A100 Space Usage
Total = 5,402 Sq. Ft
Servers & Storage 21%
Command/Console 11%Printers 10%
Networking 4%
Aisles 28%
Parts & Supplies 8%Power Distribution 6%
Growth 5%
HVAC 5%
Office 5%
Columns -3%Doors/Ramps -1%
Non-IT Space 58%
IT Space 42%
Our Space Usage (A100)Our Space Usage (A100)
How can we make more space?How can we make more space?
Curtail current usageCurtail current usage Shrink bulky systems, for example:Shrink bulky systems, for example:
Converting Unix timesharing machines to blades makes room for 40 Converting Unix timesharing machines to blades makes room for 40 new 2U serversnew 2U servers
““SAN-itizing” storage makes room for 75 new 2U serversSAN-itizing” storage makes room for 75 new 2U servers Relocate non-critical and development equipment to A82 or B7Relocate non-critical and development equipment to A82 or B7 Relocate UPS, HVAC equipmentRelocate UPS, HVAC equipment
Create more spaceCreate more space Add more racksAdd more racks Utilize idle, underused space in A100 (ACIS area, tape library)Utilize idle, underused space in A100 (ACIS area, tape library) Rearrange existing functionsRearrange existing functions
PrintingPrinting Repair/StagingRepair/Staging ColocationColocation
Build out B level -- Probably cost-prohibitive at ~$1,000/sq ft.Build out B level -- Probably cost-prohibitive at ~$1,000/sq ft.
PowerPower
Watts (single phase)Watts (single phase)W = Volts x AmpsW = Volts x Amps
Apparent Power (3 phase)Apparent Power (3 phase)kVA = V x Avg Amps x √3kVA = V x Avg Amps x √3
Real Power (3 phase)Real Power (3 phase)kW = kVA x Power factor ( < 1)kW = kVA x Power factor ( < 1)
PowerDensity
Watts/Sq FtOf ServerFloor Space
Other trends in computer technologyOther trends in computer technology
Disks – migration to 2.5” form factorDisks – migration to 2.5” form factor Greater manufacturer focus on heatGreater manufacturer focus on heat
Multi-core processor chipsMulti-core processor chips On-chip power managementOn-chip power management More exotic heat sinks, airflowMore exotic heat sinks, airflow Water coolingWater cooling
Denser blade servers (16 blades/2U today)Denser blade servers (16 blades/2U today) More “composite hosts” (clusters/grids)More “composite hosts” (clusters/grids)
Blade and Disk Equipment Densities
Products currently shipping
0
5
10
15
20
25
30
kW p
er R
ack
Blade Servers
Disk Arrays
Dell PowerEdge 1855
HP Proliant BL
IBM Bladecenter H20
EqualLogic PS / NetApp FAS 200
AC&NC JetStor III SATA
CYH Data Center Demand Densities CYH Data Center Demand Densities (A100+A84)(A100+A84)
100100 108 kW108 kW 3,0003,000 575575
RacksRacks PowerPower Sq FtSq Ft DevicesDevices
Per rackPer rack -- 1.08kW1.08kW 3030 5.755.75
Per kWPer kW 0.930.93 -- 27.827.8 5.35.3
Per sq ftPer sq ft 0.030.03 40W40W -- 0.190.19
Per devicePer device 0.170.17 190W190W 5.225.22 --
Device = Anything that consumes rack space and power(Computer, disk, switch, UPS)
CMU Campus Server Room SurveyFeb 2005
0
50
100
150
200
250
300
350
400
450
Wat
ts/S
q Ft
CYH A100
Beowulf clusterPDL/Cylab (2005)
Wean 3611
ECE HH-211B
Chemistry MI-502
PDL/Cylab (2011)
Qatar (Day 1)
Qatar (Maximum)
Power Density
CYH Emergency Electrical Power BudgetCYH Emergency Electrical Power Budget
Maximum LoadsMaximum Loads kVAkVA
3 UPS units (50 3 UPS units (50 kVA ea)kVA ea) 150150
4 (of 6) Edpacs4 (of 6) Edpacs 3737
Life safetyLife safety 33
HumidifiersHumidifiers 3737
LightingLighting 33
ECE UPSECE UPS 6060
ECE EdpacECE Edpac 6060
100 Ton rooftop 100 Ton rooftop chillerchiller 130130
Chilled water pumpChilled water pump 5252
Booster pumpBooster pump 1414
WH Hot water WH Hot water pumpspumps 2020
WH Air compressorWH Air compressor 1010
WH Life safetyWH Life safety 1010
586 kva586 kva
Estimated normal load500 – 550 kVA
Generator capacity625 kVA (500 kW)
A100/A84 Power Usage Projection
0.0
20.0
40.0
60.0
80.0
100.0
120.0
140.0
Po
wer
(kV
A)
Capacity – How much power?Capacity – How much power?
Redundancy – How much?Redundancy – How much?
DesignDesign LoadLoad PDU/UPSPDU/UPS
NN 11 11
N+1N+1 11 1.5*1.5*
2N2N 11 22
* Multiplier depends on topology
A100 Power DistributionA100 Power Distribution
Row 90Powered By PDUs 3 & 4
Row 80Powered By PDUs 3 & 4
Row 70Powered By PDUs 3 & 4
Row 60Powered by PDUs 2 & 4
Row 50Powered by PDUs 2 & 4
Row 40Powered by PDUs 2 & 4
Row 30Powered by PDUs 2 & 3
Row 20Powered by PDUs 2 & 3
Row 10Powered by PDUs 2 & 3
PDU 4
PDU 3
PDU 2
A-100 Power Distribution
How can we add more power capacity?How can we add more power capacity?
No upgrade to building transformerNo upgrade to building transformer Emergency (Generator + UPS)Emergency (Generator + UPS)
Increase UPS, PDU capacitiesIncrease UPS, PDU capacities Increase generator capacityIncrease generator capacity Increase panel sizes, feeder capacityIncrease panel sizes, feeder capacity StrategiesStrategies
Separate computer, HVAC loads for power qualitySeparate computer, HVAC loads for power quality Begin removing non-critical equipment from emergency Begin removing non-critical equipment from emergency
power to manage emergency capacitypower to manage emergency capacity Install pathway for future growthInstall pathway for future growth
Replacing distribution infrastructure is costly and Replacing distribution infrastructure is costly and disruptivedisruptive
Emergency CoolingEmergency Cooling
Actual LoadsActual Loads ServersServers DisksDisks Networking equipmentNetworking equipment Water pumpsWater pumps HVAC (chillers, Edpacs)HVAC (chillers, Edpacs) Humidifiers Humidifiers Emergency lightingEmergency lighting Life safety equipmentLife safety equipment SunlightSunlight Power distribution (PDU, UPS)Power distribution (PDU, UPS) Equipment inefficienciesEquipment inefficiencies PeoplePeople
Supply DependenciesSupply Dependencies Chiller capacityChiller capacity Generator capacityGenerator capacity Pipe diameterPipe diameter Chilled water temperatureChilled water temperature Circulating pump gpmCirculating pump gpm Chilled water pressureChilled water pressure Fan cfmFan cfm
Power In (kW) = Heat Out (BTUs)1 kW = 3,412 BTUs
1 kW requires 160 cfm for 20° ∆T1 Ton requires 3 gpm of chilled water
How can we add more cooling capacity?How can we add more cooling capacity?
Emergency chiller and generator must be upgraded Emergency chiller and generator must be upgraded together as critical load increasestogether as critical load increases
Options for upgrading HVAC capacityOptions for upgrading HVAC capacity As needed, per rack, using Liebert’s Xtreme Density As needed, per rack, using Liebert’s Xtreme Density
supplemental cooling products [supplemental cooling products [moremore]] As needed, per pair of rows, using APC’s NetworkAIR IR As needed, per pair of rows, using APC’s NetworkAIR IR
products [products [moremore]] Entire room capacity may be upgraded by adding additional Entire room capacity may be upgraded by adding additional
CRAC units, up to 60 T eachCRAC units, up to 60 T each
All options supplement existing HVACAll options supplement existing HVAC All options can be added incrementally, by rack(s)All options can be added incrementally, by rack(s) One high-density project does not “break” entire roomOne high-density project does not “break” entire room
Data ConnectionsData Connections
How many systems?How many systems? Types of network connectionsTypes of network connections
PrimaryPrimary RedundantRedundant StorageStorage Cluster heartbeatCluster heartbeat ManagementManagement ConsoleConsole Other?Other?
A100 Equipment Installations by Year
0
20
40
60
80
100
120
140
160
180
2002 2003 2004 2005
A100/A84 Server Installations FY 2004-05
0
20
40
60
80
100
120
Jul-0
4
Aug-0
4
Sep-0
4
Oct-04
Nov-04
Dec-04
Jan-
05
Feb-0
5
Mar
-05
Apr-0
5
May
-05
Jun-
05
Jul-0
5
Cumulative
By Month
Data Center ProjectionsData Center Projections(A100 + A84)(A100 + A84)
NowNow 5 Years5 Years 10 Years10 Years
DevicesDevices 575575
Data Data connections connections per systemper system
1.11.1
Power/CoolingPower/Cooling
(kW)(kW) 108108
RacksRacks 100100
Space (sq ft)Space (sq ft) 3,0003,000