best practices for data center energy efficiency seminar hosted … tcc... · 2020. 1. 3. ·...
TRANSCRIPT
![Page 1: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/1.jpg)
1 | Lawrence Berkeley National Laboratory eere.energy.gov
Organized by:
Dale Sartor, P.E.Lawrence Berkeley National Laboratory
Best Practices for Data Center Energy Efficiency Seminar
(Version: 02/23/15)
Hosted by the Technology Convergence Conference
Santa ClaraFebruary 25, 2015
![Page 2: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/2.jpg)
2 | Lawrence Berkeley National Laboratory eere.energy.gov
Download Presentation
This Presentation is Available for download at:
http://datacenterworkshop.lbl.gov/
![Page 3: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/3.jpg)
3 | Lawrence Berkeley National Laboratory eere.energy.gov
• Introduction, performance metrics and benchmarking• Dale Sartor, LBNL
• IT equipment and software efficiency• Dale Sartor, LBNL
• Break• Data center environmental conditions
• Magnus Hurlin, ANSIS
• Airflow management• Brian Donathan, Teladata
• Break• Cooling systems
• Dale Sartor, LBNL
• Electrical systems• Mukesh Khattar, EPRI
• Break• DCIM and integrated controls (Use IT to save energy)
• Panel• Resources and workshop summary
Agenda
![Page 4: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/4.jpg)
4 | Lawrence Berkeley National Laboratory eere.energy.gov
Provide background on data center efficiency
Raise awareness of efficiency opportunities
Develop common understanding between IT and
Facility staff
Review of data center efficiency resources
Group interaction for common issues and solutions
Seminar Learning Objectives
![Page 5: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/5.jpg)
5 | Lawrence Berkeley National Laboratory eere.energy.gov
Challenging Conventional Wisdom: Game Changers
Conventional Approach
• Data centers need to be cool and controlled to
tight humidity ranges
• Data centers need raised floors for cold air
distribution
• Data centers require highly redundant building
infrastructure
Need Holistic Approach
• IT and Facilities Partnership
![Page 6: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/6.jpg)
6 | Lawrence Berkeley National Laboratory eere.energy.gov
Introduction, Performance Metrics and Benchmarking
![Page 7: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/7.jpg)
7 | Lawrence Berkeley National Laboratory eere.energy.gov
Data Center Energy
• Data centers are energy intensive facilities
– 10 to 100 times more energy intensive than an office
– Server racks now designed for more than 25+ kW
– Surging demand for data storage
– 2% of US Electricity consumption
– Projected to double in next 5 years
– Power and cooling constraints in existing facilities
![Page 8: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/8.jpg)
8 | Lawrence Berkeley National Laboratory eere.energy.gov
Global Data Center Electricity Use
8 Source: Koomey 2011.
![Page 9: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/9.jpg)
9 | Lawrence Berkeley National Laboratory eere.energy.gov
How Much is 152B kWh?
Source for country data in 2005: International Energy Agency, World Energy
Balances (2007 edition)
Turkey
Sweden
Iran
World Data Centers
Mexico
South Africa
Italy
Final Electricity Consumption (Billion kWh)
0 50 100 150 200 250 300
![Page 10: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/10.jpg)
10 | Lawrence Berkeley National Laboratory eere.energy.gov
Energy efficiency programs have helped keep per capita electricity consumption in California flat over the past 30 years
-
2,000
4,000
6,000
8,000
10,000
12,000
14,000
1960 1965 1970 1975 1980 1985 1990 1995 2000
KW
h
US California Western Europe
Aggressive Programs Make a
Difference
![Page 11: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/11.jpg)
11 | Lawrence Berkeley National Laboratory eere.energy.gov
EPA Report to Congress 2008
Projected Data Center Energy Use
Current Estimated Energy Use
![Page 12: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/12.jpg)
12 | Lawrence Berkeley National Laboratory eere.energy.gov
The Rising Cost of Ownership
• Demand for computing is growing faster than
efficiency
• Cost of electricity and supporting infrastructure
now surpassing capital cost of IT equipment
• Perverse incentives -- IT and facilities costs
separate
Source: The Uptime Institute, 2007
![Page 13: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/13.jpg)
13 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL operates large systems along with legacy systems
Lawrence Berkeley National Laboratory
We also research energy efficiency opportunity and work on various deployment programs
![Page 14: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/14.jpg)
14 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL Feels the Pain!
![Page 15: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/15.jpg)
15 | Lawrence Berkeley National Laboratory eere.energy.gov
0
5
10
15
20
25
30
35
40
Me
ga
Wa
tts
2001 2003 2005 2007 2009 2011 2013 2015 2017
NERSC Computer Systems Power
(Does not include cooling power)
(OSF: 4MW max)N8
N7
N6
N5b
N5a
NGF
Bassi
Jacquard
N3E
N3
PDSF
HPSS
Misc
LBNL Super Computer Systems Power
![Page 16: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/16.jpg)
16 | Lawrence Berkeley National Laboratory eere.energy.gov
Data Center Energy Efficiency = 15% (or less)
100 Units
Source
Energy
Typical Data Center Energy End Use
Server Load
/Computing
Operations
Cooling
Equipment
Power Conversions
& Distribution
33 Units
Delivered
35 Units
Power Generation
Energy Efficiency = Useful computation / Total Source Energy
![Page 17: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/17.jpg)
17 | Lawrence Berkeley National Laboratory eere.energy.gov
Server Load/
Computing
Operations
Cooling
Equipment
Power Conversion & Distribution
On-site Generation
• High voltage distribution
• High efficiency UPS
• Efficient redundancy strategies
• Use of DC power
• Server innovation
• Virtualization
• High efficiency
power supplies
• Load management
• Better air management
• Move to liquid cooling
• Optimized chilled-water plants
• Use of free cooling
• Heat recovery
• On-site generation
Including fuel cells and
renewable sources
• CHP applications
(Waste heat for cooling)
Energy Efficiency Opportunities
![Page 18: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/18.jpg)
18 | Lawrence Berkeley National Laboratory eere.energy.gov
Electricity Use in Data Centers
Courtesy of Michael Patterson, Intel Corporation
![Page 19: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/19.jpg)
19 | Lawrence Berkeley National Laboratory eere.energy.gov
• 20-40% savings typical
• Aggressive strategies can yield
50+% savings
• Extend life and capacity of
infrastructures
Potential Benefits of Data Center Energy Efficiency
![Page 20: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/20.jpg)
20 | Lawrence Berkeley National Laboratory eere.energy.gov
Benchmarking for Energy Performance Improvement:
• Energy benchmarking
can allow performance
tracking and comparison
to peers
• LBNL conducted studies
of over 30 data centers:
– Wide variation in
performance
– Identified best practices
• Can’t manage what isn’t
measured
![Page 21: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/21.jpg)
21 | Lawrence Berkeley National Laboratory eere.energy.gov
Data Center
Server Load
51%
Data Center
CRAC Units
25%
Cooling Tower
Plant
4%
Electrical Room
Cooling
4%
Office Space
Conditioning
1%
Lighting
2%
Other
13%
Computer
Loads
67%
HVAC - Air
Movement
7%
Lighting
2%
HVAC -
Chiller and
Pumps
24%
The relative percentages of the energy doing computing varies considerably.
Energy Performance varies
Your Mileage Will Vary
![Page 22: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/22.jpg)
22 | Lawrence Berkeley National Laboratory eere.energy.gov
Benchmarks Obtained by LBNL
Average PUE = 1.83
High Level Metric: Power Utilization Effectiveness (PUE) = Total Power/IT Power
![Page 23: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/23.jpg)
23 | Lawrence Berkeley National Laboratory eere.energy.gov
PUE
EPA Energy Star Average 1.91
Intel Jones Farm, Hillsboro 1.41
T-Systems & Intel DC2020 Test Lab, Munich 1.24
Google 1.16
Leibniz Supercomputing Centre (LRZ) 1.15
National Center for Atmospheric Research (NCAR) 1.10
Yahoo, Lockport 1.08
Facebook, Prineville 1.07
National Renewable Energy Laboratory (NREL) 1.06*
Slide Courtesy Mike Patterson, Intel
PUEs: Reported & Calculated
![Page 24: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/24.jpg)
24 | Lawrence Berkeley National Laboratory eere.energy.gov
Energy Metrics and Benchmarking
• Key Metrics:
– PUE and partial PUEs (e.g. HVAC, Electrical distribution)
– Energy Reuse (ERF)
– Utilization
• The future: Computational Metrics (e.g. peak flops
per Watt; transactions/Watt)
![Page 25: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/25.jpg)
25 | Lawrence Berkeley National Laboratory eere.energy.gov
Other Data Center Metrics:
• Watts per square foot, Watts per rack
• Power distribution: UPS efficiency, IT power supply
efficiency
• HVAC
– Fan watts/cfm
– Pump watts/gpm
– Chiller plant (or chiller or overall HVAC) kW/ton
• Air Management
– Rack cooling index (fraction of IT within recommended temperature
range)
– Return temperature index (RAT-SAT)/ITΔT
• Lighting watts/square foot
![Page 26: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/26.jpg)
26 | Lawrence Berkeley National Laboratory eere.energy.gov
Metrics & Benchmarking
Airflow Efficiency
Cooling System Efficiency
Power Usage Effectiveness
![Page 27: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/27.jpg)
27 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 28: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/28.jpg)
28 | Lawrence Berkeley National Laboratory eere.energy.gov
IT Equipment and Software Efficiency
![Page 29: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/29.jpg)
29 | Lawrence Berkeley National Laboratory eere.energy.gov
The value of one watt saved at the IT equipment
IT Server Performance -Saving a Watt…
![Page 30: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/30.jpg)
30 | Lawrence Berkeley National Laboratory eere.energy.gov30 Source: Intel Corporate Technology Group
Core Integer Performance Over Time*Power reduction Over Time*
Pentium® Pro Processor
Pentium® -II ProcessorPentium® -IIi Processor
Pentium® 4 Processor
Pentium® 4 Processor EE
Pentium® -D Processor
Core™ 2 Duo Processor X6800
Core™ 2 Duo Extreme QX6700
i486DX2
i486
386
Pentium® Processor
1986 20081988 1990 1992 1994 1996 1998 2000 2002 2004 2006
1
10
100
1000
10000
Single Core Moore’s Law
1970 1980 1990 2000 2005 2010
1.E-07
1.E-06
1.E-05
1.E-04
1.E-03
1.E-02
1.E-01
1.E+00
• Every year Moore’s Law is followed, smaller, more energy-efficient transistors result
• Miniaturization provides 1 million times reduction in energy/transistor size over 30+
years.
• Benefits: Smaller, faster transistors => faster AND more energy-efficient chips.
Moore’s Law
Source: Intel Corp.
![Page 31: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/31.jpg)
31 | Lawrence Berkeley National Laboratory eere.energy.gov
Computing Efficiency Increases 100x Every Decade
Source: Koomey et al. 2011
![Page 32: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/32.jpg)
32 | Lawrence Berkeley National Laboratory eere.energy.gov
Data collected recently at a Fortune 100 company; courtesy of John Kuzma and William Carter, Intel
4%
3%
93%
60%
5%
35%32%
4%
64%
Age Distribution of Servers
2007 & Earlier
2008, 2009
2010 - Current
Energy Consumptionof Servers
Performance Capability of Servers
IT Equipment Age and Performance
Old Servers consume 60% of Energy, but deliver only 4% of Performance Capability.
![Page 33: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/33.jpg)
33 | Lawrence Berkeley National Laboratory eere.energy.gov
Idle servers consume as much as 50-60% of power @ full load as shown in SpecPower Benchmarks.
No Load
60% of full Load
IT Energy Use Patterns: Servers
Perform IT System Energy
Assessments
![Page 34: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/34.jpg)
34 | Lawrence Berkeley National Laboratory eere.energy.gov
PHYSICALLY RETIRE AN INEFFICIENT OR UNUSED SYSTEM
• Uptime Institute reported 15-30% of servers are on
but not being used
• Decommissioning goals include:
– Regularly inventory and monitor
– Consolidate/retire poorly utilized hardware
Decommission Unused Servers
![Page 35: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/35.jpg)
35 | Lawrence Berkeley National Laboratory eere.energy.gov
Virtualize and Consolidate Servers and Storage
• Run many “virtual”
machines on a single
“physical” machine
• Consolidate underutilized
physical machines,
increasing utilization
• Energy saved by shutting
down underutilized
machines
![Page 36: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/36.jpg)
36 | Lawrence Berkeley National Laboratory eere.energy.gov
Virtualization: Workload provisioningServer Consolidation
10:1 in many cases
HWHW
HW
VMM
Disaster Recovery
HW
VMM
HW
VMM
• Upholding high-levels of business continuity
• One Standby for many production servers
…OS
App
OS
App
OS
App …OS
App
HW
VMMHW
VMM
Balancing utilization with head room
Dynamic Load Balancing
OS
App1
OS
App2
OS
App3
OS
App4
CPU Usage
30%
CPU Usage
90%
CPU Usage CPU Usage
Enables rapid deployment,reducing number of idle, staged servers
R&D Production
HW
VMM
OS
App
Virtualize and Consolidate Servers and Storage
![Page 37: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/37.jpg)
37 | Lawrence Berkeley National Laboratory eere.energy.gov
Cloud Computing
• Dynamically scalable resources over the internet
• Can be internal or external
• Can balance different application peak loads
• Typically achieves higher utilization rates
Vertualized cloud computing can provide…
![Page 38: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/38.jpg)
38 | Lawrence Berkeley National Laboratory eere.energy.gov
• Power roughly linear to storage modules • Storage redundancy significantly increases energy• Consider lower energy hierarchal storage• Storage De-duplication - Eliminate unnecessary copies
Storage Systems and Energy
![Page 39: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/39.jpg)
39 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL/EPRI measured power supply efficiency
Typical operation
Use Efficient Power Supplies
![Page 40: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/40.jpg)
40 | Lawrence Berkeley National Laboratory eere.energy.gov
• Most efficient in the mid-range of performance curves• Right-size for load• Power supply redundancy puts operation lower on the curve• Use Energy Star or Climate Savers power supplies
Source: The Green Grid
Power Supply Units
Use Efficient Power Supplies
![Page 41: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/41.jpg)
41 | Lawrence Berkeley National Laboratory eere.energy.gov
Level of
CertificationEfficiency at Rated Load
115V Internal Non-
Redundant230V Internal Redundant
20% 50% 100% 20% 50% 100%
80 PLUS 80% 80% 80% n/a n/a n/a
80 PLUS Bronze 82% 85% 82% 81% 85% 81%
80 PLUS Silver 85% 88% 85% 85% 89% 85%
80 PLUS Gold 87% 90% 87% 88% 92% 88%
80 PLUS Platinum n/a n/a n/a 90% 94% 91%
80 PLUS Certification Levels
Use Efficient Power Supplies
![Page 42: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/42.jpg)
42 | Lawrence Berkeley National Laboratory eere.energy.gov
Power supply savings add up…
Annual Savings:Standard vs. High Eff Power Supply
$-
$1,000
$2,000
$3,000
$4,000
$5,000
$6,000
$7,000
$8,000
200 300 400 500
Watts per server
An
nu
al S
av
ing
s p
er
Ra
ck
Mechanical
UPS
Power supply
Use Efficient Power Supplies
![Page 43: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/43.jpg)
43 | Lawrence Berkeley National Laboratory eere.energy.gov
IT System Efficiency
Summary…
Enable power management capabilities!
Use EnergyStar® Servers
Storage Devices
Servers
Take superfluous data offline
Use thin provisioning technology
De-duplicate
Reconsider Redundancy
Use 80 PLUS or Climate Savers products
Power Supplies
ConsolidationUse virtualization
Consider cloud services
![Page 44: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/44.jpg)
44 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 45: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/45.jpg)
45 | Lawrence Berkeley National Laboratory eere.energy.gov
Environmental Conditions
Magnus Herrlin, Principal, ANCIS Incorporated
![Page 46: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/46.jpg)
46 | Lawrence Berkeley National Laboratory eere.energy.gov
• IT Load (up to 50,000+W per rack)
• Outdoor climate
• Air management (enables savings)
• Environmental conditions
(room temperature and humidity).
Main HVAC Energy Drivers
![Page 47: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/47.jpg)
47 | Lawrence Berkeley National Laboratory eere.energy.gov
Air IntakeTemperature
Air-cooled electronic
equipment depends
exclusively on the intake
air temperature for
effective cooling.
![Page 48: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/48.jpg)
48 | Lawrence Berkeley National Laboratory eere.energy.gov
Max Allowable
Rack Intake
Temperature
Max Recommended
Over-Temp
Recommended
Range
Under-Temp
Min Recommended
Min Allowable
Allowable
Range
Key Nomenclature
The recommended range is a statement of reliability. For extended periods of time, the IT manufacturers recommend that data centers maintain their environment within these boundaries.
The allowable range is a statement of functionality. These are the boundaries where IT manufacturers test their equipment to verify that the equipment will function.
Higher risk for
catastrophic failures;
higher IT fan energy
Higher risk for lower
reliability and longevity
Higher HVAC energy
![Page 49: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/49.jpg)
49 | Lawrence Berkeley National Laboratory eere.energy.gov
Energy vs. IT-Reliability
• The purpose of the allowable and recommended ranges is to give guidance
to data center operators on maintaining high reliability but yet operate their
data centers in an energy efficient manner.
• Ultimately, energy vs. IT-reliability in data centers is an optimization problem.
![Page 50: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/50.jpg)
50 | Lawrence Berkeley National Laboratory eere.energy.gov50
NEBS (telecom)
NEBS is the de-facto standard for
telecom equipment and facilities
Facility
Operation
Equipment
Design
![Page 51: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/51.jpg)
51 | Lawrence Berkeley National Laboratory eere.energy.gov51
Default recommended
range = 64.4 - 80.6F
Provides guidance for
operating above the default
upper limit
Default allowable range =
59.0 – 89.6F (Class A1)
Six classes with allowable
ranges up to 113.0F
ASHRAE Thermal Guidelines
![Page 52: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/52.jpg)
52 | Lawrence Berkeley National Laboratory eere.energy.gov
ASHRAE “Envelopes”
Dry Bulb Temperature
(C)
32 (F) 50 68 86 104 122 140
Allowable Envelopes A1-A4
18C 27C
15C
32C
40C
5C
![Page 53: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/53.jpg)
53 | Lawrence Berkeley National Laboratory eere.energy.gov
1. Data center temperature tracks with the outdoor temperature
2. A minimum of 15C – 20C can be maintained (heat from equipment)
3. A maximum below the maximum of the environmental class (mechanical cooling).
20C – 7 x 24 x 365 operation
< 1.0 > 1.0
ASHRAE Failure Rates(not incl. Delta-T, RH, particles)
![Page 54: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/54.jpg)
54 | Lawrence Berkeley National Laboratory eere.energy.gov
“For the majority of U.S. and European cities, the air-
side and water-side economizer projections show failure
rates that are very comparable to a traditional data
center run at a steady state temperature of 20ºC (68ºF).”
ASHRAE
ASHRAE Failure Rates(not incl. Delta-T, RH, particles)
![Page 55: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/55.jpg)
55 | Lawrence Berkeley National Laboratory eere.energy.gov
Liquid
Cooling
Class
Main
Cooling
Equipment
Supplemental
Cooling
Equipment
Facility
Supply-Water
Max Temperature
W1Chiller/
Cooling Tower
Water-Side
Economizer
17C
(63F)
W3 Cooling Tower Chiller32C
(90F)
W4Water-Side
EconomizerN/A
45C
(113F)
ASHRAE Liquid-CooledIT-Equipment Classes
These Maximum Temperatures are requirements to be met by the liquid-cooled IT equipment.
![Page 56: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/56.jpg)
56 | Lawrence Berkeley National Laboratory eere.energy.gov
• ASHRAE and NEBS provide guidance on temperature and humidity. Legacy gear may require more benign conditions
• For each update, ASHRAE provides more flexibility and more aggressive guidance and is now closing in on NEBS
• Many IT manufacturers design for harsher conditions than ASHRAE “default” Class A1. There is already A3 and A4 rated equipment
• Economizer projections show failure rates that are very comparable to traditional data centers with a constant temperature of 68ºF
• A cold data center = efficiency opportunity! However, address inadequate air management before increasing the temperature
• Energy vs. IT-reliability in data centers is an optimization problem.
Summary
![Page 57: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/57.jpg)
57 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 58: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/58.jpg)
58 | Lawrence Berkeley National Laboratory eere.energy.gov
Airflow Management
Brian Donathan, Teladata
![Page 59: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/59.jpg)
59 | Lawrence Berkeley National Laboratory eere.energy.gov
Air Management: The Early Days at LBNL
Fans were used to redirect air
High flow tiles reduced air pressure
It was cold but hot spots were everywhere
![Page 60: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/60.jpg)
60 | Lawrence Berkeley National Laboratory eere.energy.gov
Typically, more air circulated than required
Air mixing and short circuiting leads to: Low supply temperature
Low Delta T
Use hot and cold aisles Improve isolation of hot
and cold aisles Reduce fan energy
Improve air-conditioning efficiency
Increase cooling capacity
Hot aisle / cold aisle configuration decreases mixing of intake & exhaust air, promoting efficiency.
Air Management
![Page 61: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/61.jpg)
61 | Lawrence Berkeley National Laboratory eere.energy.gov
Benefits of Hot- and Cold-aisles
Improves equipment intake air conditions by
separating cold from hot airflow.
Preparation:Arranging racks
with alternating hot and cold aisles.
Supply cold air to front of facing servers.
Hot exhaust air exits into rear aisles.
Graphics courtesy of DLB Associates
![Page 62: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/62.jpg)
62 | Lawrence Berkeley National Laboratory eere.energy.gov
Separating Cold from Hot Airflow…
• Supply cold air as close to rack inlet as possible.
• Reduce mixing with ambient air and hot rack exhaust
• Flow air from the cold front aisle to the rear hot aisle
Equip.
Rack
No Air
Mixing
No Air
Mixing
Cold Front
Aisle
Hot Rear
Aisle
![Page 63: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/63.jpg)
63 | Lawrence Berkeley National Laboratory eere.energy.gov
Reduce Bypass and Recirculation
Bypass Air / Short-Circuiting… Recirculation…
Wastes cooling capacity. Increases inlet temperature to servers.
Leakage
![Page 64: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/64.jpg)
64 | Lawrence Berkeley National Laboratory eere.energy.gov
Bypass Air
Equip.
Rack
By-Pass
Air
By-Pass
Air
Some common causes:• Too much supply airflow
• Misplaced perforated tiles
• Leaky cable penetrations
• Too high tile exit velocity
Inlet Outlet
Inlet Outlet
![Page 65: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/65.jpg)
65 | Lawrence Berkeley National Laboratory eere.energy.gov
Recirculation Air
Equip.
Rack
Recirculation
Air
Some common causes:• Too little supply airflow
• Lack of blanking panels
• Gaps between racks
• Short equipment rows
Inlet Outlet
Inlet Outlet
![Page 66: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/66.jpg)
66 | Lawrence Berkeley National Laboratory eere.energy.gov
Maintain Raised-Floor Seals
Maintain sealing of all potential leaks in the
raised floor plenum.
Unsealed cable penetration Sealed cable penetration
![Page 67: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/67.jpg)
67 | Lawrence Berkeley National Laboratory eere.energy.gov
Manage Blanking Panels
Any opening will degrade the separation of hot and cold air
maintain server blanking and side panels.
Equip.
Rack
Air Recirculation
Inlet Outlet
Inlet Outlet
top of rack
middle of rack
SynapSense™
SynapSense™
One 12” blanking panel addedTemperature dropped ~20°
![Page 68: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/68.jpg)
68 | Lawrence Berkeley National Laboratory eere.energy.gov
Congested Floor & Ceiling Cavities
Empty Floor & Ceiling Cavities
Consider The Impact That Congestion Has On The Airflow Patterns
Reduce Airflow Restrictions & Congestion
![Page 69: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/69.jpg)
69 | Lawrence Berkeley National Laboratory eere.energy.gov
Reduce Cable Congestion
• Cable congestion sharply reduces airflow and
degrades airflow distribution.
• No cable trays should be placed below perforated tiles.
Generally, it is obvious when there is too much “stuff.”
![Page 70: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/70.jpg)
70 | Lawrence Berkeley National Laboratory eere.energy.gov
Resolve Airflow Balancing
BALANCING is required to optimize airflow.
Rebalance with new IT or HVAC equipment
Locate perforated floor tiles only in cold aisles
Under-floor pressure map with wireless sensors
![Page 71: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/71.jpg)
71 | Lawrence Berkeley National Laboratory eere.energy.gov
Results: Tune Floor Tiles
• Too many permeable floor tiles
• if airflow is optimized
– under-floor pressure up
– rack-top temperatures down
– data center capacity increases
• Measurement and visualization
assisted tuning process
under-floor pressures
rack-top temperatures
SynapSense™
SynapSense™
![Page 72: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/72.jpg)
72 | Lawrence Berkeley National Laboratory eere.energy.gov
Optimally Locate CRAC/CRAHs
Air-Handling Units
Locate CRAC/CRAH units at ends of Hot Aisles
![Page 73: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/73.jpg)
73 | Lawrence Berkeley National Laboratory eere.energy.gov
Elevation at a cold aisle looking at racks
Typical Temperature Profile with Under-floor Supply
Too hot Too hot
Just right
Too cold
There are numerous references in ASHRAE. See for example V. Sorell et al; “Comparison of Overhead and
Underfloor Air Delivery Systems in a Data Center Environment Using CFD Modeling”; ASHRAE Symposium Paper
DE-05-11-5; 2005
Hot air comes around top and sides of servers
Cold air escapes through ends of aisles
![Page 74: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/74.jpg)
74 | Lawrence Berkeley National Laboratory eere.energy.gov
Next step: Air Distribution Return-Air Plenum
![Page 75: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/75.jpg)
75 | Lawrence Berkeley National Laboratory eere.energy.gov
Return Air Plenum
• Overhead
plenum
converted to
hot-air return
• Return
registers
placed over
hot aisle
• CRAC intakes
extended to
overhead
Before
After
![Page 76: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/76.jpg)
76 | Lawrence Berkeley National Laboratory eere.energy.gov
Return-Air Plenum Connections
Return air duct on top of CRAC unit connects to the return air plenum.
![Page 77: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/77.jpg)
77 | Lawrence Berkeley National Laboratory eere.energy.gov
Isolate Hot Return
Duct on top of each rack connects to the return air plenum.
![Page 78: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/78.jpg)
78 | Lawrence Berkeley National Laboratory eere.energy.gov
Cabinet/row containment
![Page 79: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/79.jpg)
79 | Lawrence Berkeley National Laboratory eere.energy.gov
Other Isolation Options
Physical barriers enhance separate hot and cold airflow.
Barriers placement must comply with fire codes.
Curtains, doors, or lids have been used successfully.
Open Semi-enclosed Enclosed
cold aisle cold aisle
Doors Lid
![Page 80: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/80.jpg)
80 | Lawrence Berkeley National Laboratory eere.energy.gov
Adding Air Curtains for Hot/Cold Isolation
CoolShield or Trax SealTech
![Page 81: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/81.jpg)
81 | Lawrence Berkeley National Laboratory eere.energy.gov
Isolate Cold and Hot Aisles
95-105ºF vs. 60-70ºF (35-41C vs. 16-21C)
70-80ºF vs. 45-55ºF (21-27C vs. 7-13C)
![Page 82: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/82.jpg)
82 | Lawrence Berkeley National Laboratory eere.energy.gov
Cold Aisle Airflow Containment Example
LBNL Cold Aisle Containment study achieved fan energy savings of ~ 75%
![Page 83: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/83.jpg)
83 | Lawrence Berkeley National Laboratory eere.energy.gov
Fan Energy Savings
Isolation can significantly reduce air bypass and hence flow
Fan speed can be reduced and fan power is proportional to the cube of the flow.
Fan energy savings of 70-80% is possible with variable air volume (VAV) fans in CRAH/CRAC units (or central AHUs)
Without Enclosure With Enclosure Without Enclosure
![Page 84: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/84.jpg)
84 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL Air Management Demonstration
Cold Aisle NW - PGE12813
40
45
50
55
60
65
70
75
80
85
90
6/13/2006 12:00 6/14/2006 0:00 6/14/2006 12:00 6/15/2006 0:00 6/15/2006 12:00 6/16/2006 0:00 6/16/2006 12:00
Time
Te
mp
era
ture
(d
eg
F)
Low
Med
High
Baseline Alternate 1
Setup
Setup
Alternate 2
ASHRAE
Recommended Range
Ranges during
demonstration
Better airflow management permits warmer supply temperatures!
![Page 85: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/85.jpg)
85 | Lawrence Berkeley National Laboratory eere.energy.gov
Hot and Cold Aisle Containment
Subzero Cold Aisle Containment
APC Hot Aisle Containment
(with in-row cooling)
Ceilume Heat Shrink Tiles
![Page 86: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/86.jpg)
86 | Lawrence Berkeley National Laboratory eere.energy.gov
Isolating Hot and Cold AislesSummary
• Energy intensive IT equipment needs good isolation of
“cold” inlet and “hot” discharge.
• Supply airflow can be reduced if no bypass occurs.
• Overall temperature can be raised if air is delivered
without mixing.
• Cooling systems and economizers use less energy with
warmer return air temperatures.
• Cooling capacity increases with warmer air
temperatures.
![Page 87: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/87.jpg)
87 | Lawrence Berkeley National Laboratory eere.energy.gov
Efficient Alternatives to Under-floor Air Distribution
Localized air cooling systems with hot and cold isolation can supplement or replace under-floor systems (raised floor not required!)
Examples include:
Row-based cooling units
Rack-mounted heat exchangers
Both options “Pre-engineer” hot and cold isolation
![Page 88: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/88.jpg)
88 | Lawrence Berkeley National Laboratory eere.energy.gov
Example – Local Row-Based Cooling Units
![Page 89: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/89.jpg)
89 | Lawrence Berkeley National Laboratory eere.energy.gov
In-row cooling system
End cap
Hot aisle lid
© APC reprinted with permission
Rack
In row fan-coil
With hot aisle containment, the general data center is neutral (75-80F)
![Page 90: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/90.jpg)
90 | Lawrence Berkeley National Laboratory eere.energy.gov
Air Distribution – Rack-Mounted Heat Exchangers
![Page 91: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/91.jpg)
91 | Lawrence Berkeley National Laboratory eere.energy.gov
Review:Airflow Management Basics
Air management techniques:
– Seal air leaks in floor (e.g. cable penetrations)
– Prevent recirculation with blanking panels in racks
– Manage floor tiles (e.g. no perforated tiles in hot aisle)
– Improve isolation of hot and cold air (e.g. return air plenum, curtains, or complete isolation)
Impact of good isolation:
– Supply airflow reduced
• Fan savings up to 75%+
– Overall temperature can be raised
• Cooling systems efficiency improves
• Greater opportunity for economizer (“free” cooling)
– Cooling capacity increases
![Page 92: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/92.jpg)
92 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 93: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/93.jpg)
93 | Lawrence Berkeley National Laboratory eere.energy.gov
Cooling systems
![Page 94: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/94.jpg)
94 | Lawrence Berkeley National Laboratory eere.energy.gov
Air Management and Cooling Efficiency
Improved efficiencies
Increased cooling capacity
More hours for air-side and water-side free cooling
Lower humidification/dehumidification energy
Reduced fan energy
Linking good air management and an optimized
cooling system:
![Page 95: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/95.jpg)
95 | Lawrence Berkeley National Laboratory eere.energy.gov
HVAC Systems Overview
Adapted from ASHRAE
![Page 96: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/96.jpg)
96 | Lawrence Berkeley National Laboratory eere.energy.gov
• CRAC units
– Fan, direct expansion (DX) coil, and
refrigerant compressor.
• CRAH units
– Fan and chilled water coil
– Typically in larger facilities with a chiller plant
• Both often equipped with humidifiers and reheat for dehumidification
• Often independently controlled
– Tight ranges and poor calibration lead to fighting
Computer Room Air Conditioners (CRACs) and Air Handlers (CRAHs)
![Page 97: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/97.jpg)
97 | Lawrence Berkeley National Laboratory eere.energy.gov
Water
Air-Cooled DX
Evaporatively-Cooled DX
Water-Cooled DX
Dry-Cooler DX
DX (or AC) units reject heat outside…
![Page 98: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/98.jpg)
98 | Lawrence Berkeley National Laboratory eere.energy.gov
Water
Air-Cooled Chiller
Evap-Cooled Chiller
Water-Cooled Chiller
Cooling Tower
98
Computer Room Air Handling (CRAH) units using Chilled-Water
CRAH
![Page 99: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/99.jpg)
99 | Lawrence Berkeley National Laboratory eere.energy.gov
• Have a plant (vs. distributed cooling)
• Use “warm” water cooling (multi-loop)
• Size cooling towers for “free” cooling
• Integrate controls and monitor efficiency of all primary components
• Thermal storage
• Utilize variable speed drives on:
– Fans
– Pumps
– Towers
– Chillers
Optimize the Chiller Plant
Primary CHW
Pump10%
Secondary CHW Pump
4%
CW Pump13%
Cooling Tower 5%
Chiller
68%
![Page 100: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/100.jpg)
100 | Lawrence Berkeley National Laboratory eere.energy.gov
Select Efficient Chillers
ChillerCompressor kW / ton
25% 50% 75% 100%
400 Ton Air Cooled 0.69 0.77 0.96 1.25
1200 Ton Water Cooled w/o VFD 0.51 0.41 0.45 0.55
1200 Ton Water Cooled with a VFD 0.34 0.30 0.43 0.57
![Page 101: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/101.jpg)
101 | Lawrence Berkeley National Laboratory eere.energy.gov
Increase Temperature of Chiller Plant
Tons
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
200 300 400 500 600 700 800 900 1000
Eff
icie
ncy (
kW
/to
n)
1,000 Ton Chiller operating at 60 F CHWS
Temp and 70 F CWS Temp
1,000 Ton Chiller operating at 42 F
CHWS Temp and 70 F CWS Temp
Data provided by York International Corporation.
![Page 102: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/102.jpg)
102 | Lawrence Berkeley National Laboratory eere.energy.gov
Moving (Back) to Liquid Cooling
~ 190,000 cubic foot blimp
=
Water
Air
Volumetric heat capacity comparison
[1.5 m3]
[5380 m3]
As heat densities rise, liquid solutions become more attractive:
![Page 103: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/103.jpg)
103 | Lawrence Berkeley National Laboratory eere.energy.gov
Why Liquid Cooling?
• Heat removal efficiency increases as liquid gets closer to the heat source
• Liquids can provide cooling with higher temperature coolant– Improved cooling efficiency
– Increased economizer hours
– Greater potential use of waste heat
• Reduced transport energy:
![Page 104: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/104.jpg)
104 | Lawrence Berkeley National Laboratory eere.energy.gov
In-Row Liquid Cooling
Graphics courtesy of Rittal
![Page 105: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/105.jpg)
105 | Lawrence Berkeley National Laboratory eere.energy.gov
In Rack Liquid Cooling
Racks with integral coils and full containment
CoolingCoil
![Page 106: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/106.jpg)
106 | Lawrence Berkeley National Laboratory eere.energy.gov
106
Rear Door Heat Exchanger
• Passive technology: relies on server fans for airflow
• Can use chilled or higher temperature water for cooling
Photo courtesy of Vette
![Page 107: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/107.jpg)
107 | Lawrence Berkeley National Laboratory eere.energy.gov
Rear Door (open)
Rear-Door Liquid Cooling
Inside rack RDHx, open 90°
Liquid Cooling Connections
Rear Doors (closed)
![Page 108: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/108.jpg)
108 | Lawrence Berkeley National Laboratory eere.energy.gov
Direct touch cooling
Conducting heat to a cold plate containing refrigerant
Clustered Systems design
![Page 109: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/109.jpg)
109 | Lawrence Berkeley National Laboratory eere.energy.gov
Standoff
Gap filler
CPU or other
high dissipation
device
VRM
Chips &
Inductors
Server LidDRAM
Main
Board
Simple heat
conductor
Thermal
interface
material
Gap Filler
Heat conductor Compliant Pad
on server lid
Schematic
• Server fans are removed
• Heat risers connect to top plate which has a micro channel heat exchanger
109
![Page 110: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/110.jpg)
110 | Lawrence Berkeley National Laboratory eere.energy.gov
On Board Cooling
![Page 111: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/111.jpg)
111 | Lawrence Berkeley National Laboratory eere.energy.gov
Maui High Performance Computing Center
• Increased air intake temperature
• Turned off 3 CRAHs
• Saved almost $300,000 annually
Then:• Installed dry coolers for water cooled system
• Will save additional $200,000 annually
• 6 times more compute
![Page 112: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/112.jpg)
112 | Lawrence Berkeley National Laboratory eere.energy.gov
IBM Riptide
Maui HPC Center Warm Water Cooling
• 91% water cooled, 9% air cooled
Water inside
![Page 113: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/113.jpg)
113 | Lawrence Berkeley National Laboratory eere.energy.gov
• Cooling water temperature as high as 44° C
Dry Coolers, 10kW each
compared to 100kW Chillers
Water Piping Behind the servers
MHPCC Water Cooling, continued
![Page 114: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/114.jpg)
114 | Lawrence Berkeley National Laboratory eere.energy.gov
Server
CondenserVapor
Boiling
Liquid
Tank
Pump Dry TowerFacility
Wall
Ambient as
high as 55°C
Facility Water
Cooling Power = Pump + FanIT Power
Liquid immersion cooling demonstration
• chillers
• cooling towers
• water use
• raised floors
• computer room air conditioners
• earplugs!
No longer requires:
![Page 115: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/115.jpg)
115 | Lawrence Berkeley National Laboratory eere.energy.gov
Computer in glass tank
3M Corp.
4 server system
Phase change of dielectric fluid removes heat efficiently
![Page 116: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/116.jpg)
116 | Lawrence Berkeley National Laboratory eere.energy.gov
“Chill-off 2” Evaluation of Liquid Cooling Solutions
![Page 117: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/117.jpg)
117 | Lawrence Berkeley National Laboratory eere.energy.gov
Use Free Cooling:
Cooling without Compressors:
• Outside-Air Economizers
• Water-side Economizers
Let’s get rid of chillers in data centers
![Page 118: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/118.jpg)
118 | Lawrence Berkeley National Laboratory eere.energy.gov
Outside-Air Economizers
Advantages• Lower energy use
• Added reliability (backup for
cooling system)
Potential Issues• Space.
• Dust
– Not a concern with Merv 13
filters
• Gaseous contaminants
– Not widespread
– Impacts normally cooled data
centers as well
• Shutdown or bypass if smoke
is outside data center.
http://cooling.thegreengrid.org/namerica/WEB_APP/calc_index.html
![Page 119: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/119.jpg)
119 | Lawrence Berkeley National Laboratory eere.energy.gov
Green Grid Tool
![Page 120: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/120.jpg)
120 | Lawrence Berkeley National Laboratory eere.energy.gov
UC’s Computational Research
and Theory (CRT) Facility
![Page 121: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/121.jpg)
121 | Lawrence Berkeley National Laboratory eere.energy.gov
Free Cooling – Outside Air Based
Annual Psychrometric Chart of Oakland, CA(relative humidity lines are stepped by 10%,
wetbulb lines by 10 degrees F)
0
0.001
0.002
0.003
0.004
0.005
0.006
0.007
0.008
0.009
0.01
0.011
0.012
0.013
0.014
0.015
0.016
0.017
0.018
0.019
0.02
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120
Drybulb Temp (F)
W (
lb/l
b)
WB = 30
WB = 40
WB = 50
WB = 60
WB = 70
1. Blue = recommended supply
2. Green can become blue mixing
return and outdoor air
3. Most of the conditions below
and right of blue can be
satisfied w/ evaporative cooling
4. Hot and humid hours will enter
the “allowable” range or require
compressor air conditioning
![Page 122: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/122.jpg)
122 | Lawrence Berkeley National Laboratory eere.energy.gov
• Air-Side Economizer (93% of hours)
• Direct Evaporative Cooling for Humidification/ pre-cooling
• Low Pressure-Drop Design (1.5” total static peak)
Hours of Operation
Mode 1 100% Economiser 2207 hrs
Mode 2 OA + RA 5957 hrs
Mode 3 Humidification 45 hrs
Mode 4 Humid + CH cooling 38 hrs
Mode 5 CH only 513 hrs
total 8760 hrs
System Design Approach:
![Page 123: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/123.jpg)
123 | Lawrence Berkeley National Laboratory eere.energy.gov
• Tower side economizer
• Four pipe system
• Waste heat reuse
• Headers, valves and caps for
modularity and flexibility
Water Cooling:
Predicted CRT Performance:
• Annual PUE = 1.1
![Page 124: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/124.jpg)
124 | Lawrence Berkeley National Laboratory eere.energy.gov
Advantages
• Cost effective in cool and
dry climates
• Often easier retrofit
• Added reliability (backup in
the event of chiller failure).
• No contamination questions
Water-Side Economizers
![Page 125: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/125.jpg)
125 | Lawrence Berkeley National Laboratory eere.energy.gov
Make sure that the
heat exchanger is in
series (not parallel
with chillers on CHW
side)
You can use either a
control valve or pump
44F60F
Twb 41F
46F
49F
44F
44F
<60F
Integrated Water-Side Economizer
[7C]
[7C]
[16C]
[9C]
[7C] [16C]
[8C]
[5C]
![Page 126: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/126.jpg)
126 | Lawrence Berkeley National Laboratory eere.energy.gov
Oakla
nd
Oakla
nd
Oakla
nd
Oakla
nd
Oakla
nd
Sacra
mento
Sacra
mento
Sacra
mento
Sacra
mento
Sacra
mento
Los
Angele
s
Los
Angele
s
Los
Angele
s
Los
Angele
s
Los
Angele
s
0
1000
2000
3000
4000
5000
6000
7000
8000
45 55 60 65 104
Fre
e C
ooling P
ote
nti
al (H
ours
per
Year)
Chilled Water Design Temperature (Degrees F)
Free Cooling Opportunity
Water-Side Economizer
Rear-Door Cooling
On-Board Cooling
Potential for Tower Cooling
![Page 127: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/127.jpg)
127 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL Example: Rear Door Cooling
• Used instead of
adding CRAC units
• Rear door water
cooling with tower-
only (or central
chiller plant in
series).
– Both options
significantly more
efficient than existing
direct expansion (DX)
CRAC units.
![Page 128: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/128.jpg)
128 | Lawrence Berkeley National Laboratory eere.energy.gov
• Eliminate inadvertent dehumidification
– Computer load is sensible only
• Use ASHRAE allowable humidity ranges
– Maintain inlet conditions between 41.9° F dew-point and 59° F
dew-point and 60% RH or manufacturer’s requirements (many
manufacturers allow even wider humidity range).
– Use dew-point control, NOT %RH.
• Defeat equipment fighting
– Coordinate controls
• Disconnect and only control humidity of makeup air or one
CRAC/CRAH unit
• Entirely disconnect (many have!)
Improve Humidity Control:
![Page 129: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/129.jpg)
129 | Lawrence Berkeley National Laboratory eere.energy.gov
– Some contaminants (hydroscopic salts) with high humidity can deposit and bridge across circuits
– Operating with high humidity (>60%) in an environment with high concentrations of particulates could be a problem.
• Normal building filtration is effective in removing particulates
– Operating with high humidity (>60%) in areas with gaseous contamination could cause problems. More study is needed in this area, however few locations have such conditions.
High Humidity Limit Issues
![Page 130: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/130.jpg)
130 | Lawrence Berkeley National Laboratory eere.energy.gov
– Industry practices
• Telecom has no lower limit (personnel grounding expected)
• Electrostatic Discharge Association removed humidity control as a primary ESD control measure in ESD/ANSI S20.20
• IT equipment is qualified to withstand ESD and it is grounded
• Many centers eliminate humidification with no adverse effects
– Recommended procedures
• Personnel grounding
• Cable and floor grounding
Electrostatic discharge
Low Humidity Limit Issues
![Page 131: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/131.jpg)
131 | Lawrence Berkeley National Laboratory eere.energy.gov
Typical Server Specification
Class 1 & 2
Recommended Range
Server Performance Specifications Generally Exceed
ASHRAE Ranges
Chart courtesy of ASHRAE
Not to worry…
![Page 132: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/132.jpg)
132 | Lawrence Berkeley National Laboratory eere.energy.gov
Temp RH Tdp Temp RH Tdp Mode
AC 005 84.0 27.5 47.0 76 32.0 44.1 Cooling
AC 006 81.8 28.5 46.1 55 51.0 37.2 Cooling & Dehumidification
AC 007 72.8 38.5 46.1 70 47.0 48.9 Cooling
AC 008 80.0 31.5 47.2 74 43.0 50.2 Cooling & Humidification
AC 010 77.5 32.8 46.1 68 45.0 45.9 Cooling
AC 011 78.9 31.4 46.1 70 43.0 46.6 Cooling & Humidification
Min 72.8 27.5 46.1 55.0 32.0 37.2
Max 84.0 38.5 47.2 76.0 51.0 50.2
Avg 79.2 31.7 46.4 68.8 43.5 45.5
Visalia Probe CRAC Unit Panel
Cost of Unnecessary Humidity Control
Humidity down ~2%
CRAC power down 28%
![Page 133: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/133.jpg)
133 | Lawrence Berkeley National Laboratory eere.energy.gov
Cooling Takeaways…
• Use efficient equipment and a central plant (e.g. chiller/CRAHs) vs. CRAC units
• Use centralized controls on CRAC/CRAH units– Prevent simultaneous humidifying and dehumidifying
– Optimize sequencing and staging
• Move to liquid cooling (room, row, rack, chip)
• Consider VSDs on fans, pumps, chillers, and towers
• Use air- or water-side economizers where possible.
• Expand humidity range and improve humidity control (or disconnect).
![Page 134: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/134.jpg)
134 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 135: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/135.jpg)
135 | Lawrence Berkeley National Laboratory eere.energy.gov
Electrical Systems
![Page 136: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/136.jpg)
136 | Lawrence Berkeley National Laboratory eere.energy.gov
Electrical system end use –Orange bars
Courtesy of Michael Patterson, Intel Corporation
![Page 137: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/137.jpg)
137 | Lawrence Berkeley National Laboratory eere.energy.gov
Electrical Distribution
• Every power conversion (AC-DC, DC-AC, AC-AC)
loses some energy and creates heat
• Efficiency decreases when systems are lightly loaded
• Distributing higher voltage is more efficient and can
save capital cost (conductor size is smaller)
• Power supply, uninterruptible power supply (UPS),
transformer, and PDU efficiency varies – carefully
select
• Lowering distribution losses also lowers cooling loads
![Page 138: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/138.jpg)
138 | Lawrence Berkeley National Laboratory eere.energy.gov
Electrical Systems –Points of Energy Inefficiency
![Page 139: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/139.jpg)
139 | Lawrence Berkeley National Laboratory eere.energy.gov
UPS, Transformer, & PDU Efficiency
Factory Measurements of UPS Efficiency
70%
75%
80%
85%
90%
95%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Percent of Rated Active Power Load
Eff
icie
ncy
Flywheel UPS
Double-Conversion UPS
Delta-Conversion UPS
(tested using linear loads)
• Efficiencies vary
with system design,
equipment, and
load
• Redundancies
impact efficiency
![Page 140: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/140.jpg)
140 | Lawrence Berkeley National Laboratory eere.energy.gov
Redundant Operation
Measured UPS efficiency
![Page 141: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/141.jpg)
141 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL/EPRI Measured Power
Supply Efficiency
Typical operation
![Page 142: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/142.jpg)
142 | Lawrence Berkeley National Laboratory eere.energy.gov142
An Electric Power Research Institute case study
illustrated the savings
The 80 Plus program drives
efficiency improvement
![Page 143: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/143.jpg)
143 | Lawrence Berkeley National Laboratory eere.energy.gov143
Dynamic power software turns off redundant power supplies when not needed
Upgraded power supplies
HPRack1 Blade Enclosure 1 Dynamic Power and 80-Plus PSU Test
4
4.5
5
5.5
6
6.5
7
2/21/11 0:00 2/26/11 0:00 3/3/11 0:00 3/8/11 0:00 3/13/11 0:00 3/18/11 0:00 3/23/11 0:00 3/28/11 0:00 4/2/11 0:00 Date/Time
kW
BLENC01 Original PSU BLENC01 80-Plus PSU
Dynamic Power Enabled
DynamicPower Disabled
DynamicPower Disabled
Enabled Dynamic Power
6% Savings due to 80 plus
25% Savings with Dynamic Software
![Page 144: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/144.jpg)
144 | Lawrence Berkeley National Laboratory eere.energy.gov
Redundancy
• Understand what redundancy costs and what it gets you – is it worth it?
• Does everything need the same level?
• Different strategies have different energy penalties (e.g. 2N vs. N+1)
• It’s possible to more fully load UPS systems and achieve desired redundancy
• Redundancy in electrical distribution puts you down the efficiency curve
• Redundancy in the network vs. data center
Redundancy
![Page 145: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/145.jpg)
145 | Lawrence Berkeley National Laboratory eere.energy.gov
Inverter
In Out
Bypass
Battery/ChargerRectifier
Inverter
In Out
Bypass
Battery/ChargerRectifier
Internal Drive
External Drive
I/O
Memory Controller
m Processor
SDRAM
Graphics Controller
DC/DCAC/DC
DC/DC
AC/DC Multi output
Power Supply
Voltage Regulator Modules
5V
12V
3.3V
12V 1.5/2.5V
1.1V-1.85V
3.3V
3.3V
12V
PWM/PFCSwitcher
Unregulated DCTo Multi Output Regulated DC
Voltages
Internal Drive
External Drive
I/O
Memory Controller
m Processor
SDRAM
Graphics Controller
DC/DCAC/DC
DC/DC
AC/DC Multi output
Power Supply
Voltage Regulator Modules
5V
12V
3.3V
12V 1.5/2.5V
1.1V-1.85V
3.3V
3.3V
12V
PWM/PFCSwitcher
Unregulated DCTo Multi Output Regulated DC
Voltages
From Utility Power to the Chip –
Multiple Electrical Power Conversions
Power Distribution Unit (PDU)
ServerUninterruptible Power Supply (UPS)
AC DC AC DC
![Page 146: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/146.jpg)
146 | Lawrence Berkeley National Laboratory eere.energy.gov
AC/
DC
480 VAC
Bulk
Power
SupplyDC UPS
or
Rectifier
DC/
DCVRM
VRM
VRM
VRM
VRM
VRM
12 V
Loads
using
Legacy
Voltages
Loads
using
Silicon
Voltages
12 V
5 V
3.3 V
1.2 V
1.8 V
0.8 VServer
PSU
380 VDC
• Eliminate several conversions• Also use for lighting, and variable speed drives• Use with on-site generation including renewable energy sources
•
380V. DC power distribution
Back up -batteries or flywheel
Emerging Technology:
DC Distribution
![Page 147: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/147.jpg)
147 | Lawrence Berkeley National Laboratory eere.energy.gov
Standby generators typically use more energy than they will ever generate
Other options:
•Right-sizing of stand-by generation
•Consider redundancy options
Several load sourcesHeatersBattery chargersTransfer switchesFuel management systems
Reduce or eliminate heating, batteries, and chargers – check with manufacturer for temperature and control
Standby generation loss
![Page 148: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/148.jpg)
148 | Lawrence Berkeley National Laboratory eere.energy.gov
Standby generator heater
![Page 149: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/149.jpg)
149 | Lawrence Berkeley National Laboratory eere.energy.gov
Data center lighting
• Lights are on and nobody’s home– Switch off lights in unused/unoccupied areas or rooms
(UPS, Battery, S/Gear, etc)
– Lighting controls such as occupancy sensors are well proven
• Small relative benefit but easy to accomplish –
also saves HVAC energy
• Use energy efficient lighting
• Lights should be located over the aisles
![Page 150: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/150.jpg)
150 | Lawrence Berkeley National Laboratory eere.energy.gov
Motors and Drives
• Since most cooling system equipment operates continuously, premium efficiency motors should be specified everywhere
• Variable speed drives should be used – Chillers
– Pumps
– Air handler fans
– Cooling tower fans
![Page 151: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/151.jpg)
151 | Lawrence Berkeley National Laboratory eere.energy.gov
Key Electrical Takeaways
• Choose highly efficient components and configurations
• Reduce power conversion (AC-DC, DC-AC, AC-AC, DC-
DC)
• Consider the minimum redundancy required as
efficiency decreases when systems are lightly loaded
• Use higher voltage
![Page 152: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/152.jpg)
152 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 153: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/153.jpg)
153 | Lawrence Berkeley National Laboratory eere.energy.gov
DCIM and Integrated Controls(Using IT to Manage IT)
A Panel Discussion
![Page 154: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/154.jpg)
154 | Lawrence Berkeley National Laboratory eere.energy.gov
DCIM Panel
Bruce Myatt, Critical Facilities Roundtable (Moderator)
Dale Sartor, LBNL
Craig Compiano, Modius
![Page 155: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/155.jpg)
155 | Lawrence Berkeley National Laboratory eere.energy.gov
Use IT to Manage IT Energy
• Most operators lack “visibility” into their data center environment.
• An operator can’t manage what they don’t measure.
• Goals:
– Provide the same level of monitoring and visualization of the
physical space that exists for monitoring the IT environment.
– Measure and track performance metrics.
– Spot problems before they result in high energy cost or down
time.
Using IT to Save Energy in IT:
![Page 156: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/156.jpg)
156 | Lawrence Berkeley National Laboratory eere.energy.gov
The Importance of Visualization
• IT Systems & network administrators have tools for visualization.
• Useful for debugging, benchmarking, capacity planning, forensics.
• Data center facility managers have had comparatively poor visualization tools.
![Page 157: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/157.jpg)
157 | Lawrence Berkeley National Laboratory eere.energy.gov
LBNL installed 800+ point
sensor network.
Measures:
• Temperature
• Humidity
• Pressure (under floor)
• Electrical power
Presents real-time
feedback and historic
tracking
Optimize based on
empirical data, not intuition.
LBNL Wireless Sensor Installation
Image: SynapSense
![Page 158: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/158.jpg)
158 | Lawrence Berkeley National Laboratory eere.energy.gov
Real-time Temperature Visualization by Level
![Page 159: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/159.jpg)
159 | Lawrence Berkeley National Laboratory eere.energy.gov
Displayed Under-floor Pressure Map…
CRAC CRAC CRAC CRAC CRAC
![Page 160: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/160.jpg)
160 | Lawrence Berkeley National Laboratory eere.energy.gov
Removed guesswork by
monitoring and using
visualization tool.
Provided Real-time Feedback During Floor-tile Tuning
Under-Floor Pressure
Rack-Top Temperatures
![Page 161: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/161.jpg)
161 | Lawrence Berkeley National Laboratory eere.energy.gov
• Enhanced knowledge of
data center redundancy.
• Turned off unnecessary
CRAC units to save
energy.
Determined Relative CRAC Cooling Energy Impact
Under-Floor Pressure
Rack-Top Temperatures
![Page 162: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/162.jpg)
162 | Lawrence Berkeley National Laboratory eere.energy.gov
Feedback Continues to Help:Note impact of IT cart!
Real-time feedback identified
cold aisle air flow obstruction!
![Page 163: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/163.jpg)
163 | Lawrence Berkeley National Laboratory eere.energy.gov
Real-time PUE Display
![Page 164: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/164.jpg)
164 | Lawrence Berkeley National Laboratory eere.energy.gov
PUE Calculation Diagram
![Page 165: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/165.jpg)
165 | Lawrence Berkeley National Laboratory eere.energy.gov
Franchise Tax Board (FTB) Case Study
Description:
• 10,000 Sq Ft
• 12 CRAH cooling units
• 135 kW load
Challenges:
• Over-provisioned
• History of fighting
• Manual shutoff not successful
Solution:
• Intelligent supervisory control software with inlet air sensing
![Page 166: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/166.jpg)
166 | Lawrence Berkeley National Laboratory eere.energy.gov
FTB Wireless Sensor Network
• WSN included 50 wireless temperature
sensors (Dust Networks radios)
• Intelligent control software
![Page 167: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/167.jpg)
167 | Lawrence Berkeley National Laboratory eere.energy.gov
WSN Smart Software: learns about curtains
CRAH 3 influence at start CRAH 3 influence after curtains
CRAH-03
![Page 168: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/168.jpg)
168 | Lawrence Berkeley National Laboratory eere.energy.gov
WSN Provided Effect on Cold-aisle Temperatures:
50
55
60
65
70
75
80
85
90
A B C D E F
Time Interval
deg
F
flo
or
tile
ch
an
ge
s
VF
Ds
co
ntr
ol so
ftw
are
ho
t a
isle
iso
latio
n
rack b
lan
ks
lower limit of ASHRAE recommended range
upper limit of ASHRAE recommended range
![Page 169: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/169.jpg)
169 | Lawrence Berkeley National Laboratory eere.energy.gov
WSN Software = Dramatic Energy Reduction…
0
10
20
30
40
50
60
1/31 2/5 2/10 2/15
date/time
Main
bre
aker,
kW
BEFORE
AFTER
DASH™ software started
![Page 170: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/170.jpg)
170 | Lawrence Berkeley National Laboratory eere.energy.gov
Cost-Benefit Analysis:
• DASH cost-benefit (sensors and software)• Cost: $56,824• Savings: $30,564• Payback: 1.9 years
• Total project cost-benefit• Cost: $134,057• Savings: $42,772• Payback: 3.1 years
![Page 171: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/171.jpg)
171 | Lawrence Berkeley National Laboratory eere.energy.gov
An Emerging Technology…
Control data center air conditioning using the
built-in IT server-equipment temperature sensors
• Typically, data center cooling uses return air temperature as the primary control-variable
• ASHRAE and IT manufacturers agree IT equipment inlet air temperature is the key parameter
• Optimum control difficult
• IT equipment has multiple sensors used to protect itself by adjusting internal fans, clock speeds, etc.
• One such sensor is typically located at the air inlet to the IT equipment - monitoring intake conditions
• Information from these sensors is available on the IT network
![Page 172: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/172.jpg)
172 | Lawrence Berkeley National Laboratory eere.energy.gov
Intel Demonstration
![Page 173: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/173.jpg)
173 | Lawrence Berkeley National Laboratory eere.energy.gov
Intel Demonstration
• Demonstration showed:
– Servers can provide temperature data to facilities
control system
– Given server inlet temperature, facility controls
improved temperature control and efficiency
– Effective communications and control accomplished
without significant interruption or reconfiguration of
systems
![Page 174: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/174.jpg)
174 | Lawrence Berkeley National Laboratory eere.energy.gov
Dashboards
Dashboards can display multiple systems’ information for
monitoring and maintaining data center performance
![Page 175: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/175.jpg)
175 | Lawrence Berkeley National Laboratory eere.energy.gov
Why Dashboards?
• Provide IT and HVAC system performance at a glance
• Identify operational problems
• Baseline energy use and benchmark performance
• View effects of changes
• Share information and
inform integrated
decisions
![Page 176: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/176.jpg)
176 | Lawrence Berkeley National Laboratory eere.energy.gov
Another Dashboard Example…
![Page 177: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/177.jpg)
177 | Lawrence Berkeley National Laboratory eere.energy.gov
Dashboard Examples
Aggregates Data Across Facilities
![Page 178: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/178.jpg)
178 | Lawrence Berkeley National Laboratory eere.energy.gov
Use IT to Manage IT: Summary
• Evaluate monitoring systems to enhance
operations and controls
• Install dashboards to manage and sustain
energy efficiency.
![Page 179: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/179.jpg)
179 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 180: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/180.jpg)
180 | Lawrence Berkeley National Laboratory eere.energy.gov
Resources
![Page 181: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/181.jpg)
181 | Lawrence Berkeley National Laboratory eere.energy.gov
Federal Energy Management Program
• Workshops
• Federal case studies
• Federal policy guidance
• Information exchange & outreach
• Qualified specialists
• Technical assistance
EPA
• Metrics
• Server performancerating & ENERGY STAR label
• Data center benchmarking
DOE Better Buildings
• Tool suite & metrics for baselining
• Training
• Qualified specialists
• Showcase Case studies
• Recognition of high energy savers
Industry
• Tools
• Metrics
• Training
• Best practice information
• Best-in-Class guidelines
• IT work productivity standard
GSA
• Workshops
• Quick Start Efficiency Guide
• Technical Assistance
Resources
![Page 182: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/182.jpg)
182 | Lawrence Berkeley National Laboratory eere.energy.gov
Resources from DOE Federal Energy Management Program (FEMP)
Best Practices Guide
Benchmarking Guide
Data Center Programming Guide
Technology Case Study Bulletins
Procurement Specifications
Report Templates
Process Manuals
Quick-Start Guide
![Page 183: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/183.jpg)
183 | Lawrence Berkeley National Laboratory eere.energy.gov
DOE’s FEMP data center program provides tools and resources to help owners and operators:
• DC Pro Software and Assessment Tool Suite– Tools to define baseline energy use and identify energy-saving
opportunities
• Information products– Manuals, case studies, and other resources
• End-user awareness training
• Data Center Energy Practitioner (DCEP) certificate program – Qualification of professionals to evaluate energy efficiency opportunities
DOE Federal Energy Management Program (FEMP)
![Page 184: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/184.jpg)
184 | Lawrence Berkeley National Laboratory eere.energy.gov
High-Level On-Line Profiling and Tracking Tool
• Overall efficiency (Power Usage Effectiveness [PUE])
• End-use breakout
• Potential areas for energy efficiency improvement
• Overall energy use reduction potential
IT-Equipment
• Servers
• Storage & networking
• Software
Electrical Systems
• UPS
• PDU
• Transformers
• Lighting
• Standby gen.
Cooling
• Air handlers/ conditioners
• Chillers, pumps, fans
• Free cooling
Air Management
• Hot/cold separation
• Environmental conditions
• RCI and RTI
In-Depth Assessment Tools Savings
DOE DC Pro Tool Suite
![Page 185: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/185.jpg)
185 | Lawrence Berkeley National Laboratory eere.energy.gov
U.S. DOE certificate process for energy practitioners
qualified to assess energy consumption and energy
efficiency opportunities in Data Centers.
Key objective:
• Raise the standard of energy assessors
• Greater repeatability/credibility of recommendations.
Target groups include:
• Data Center personnel (in-house experts)
• Consulting professionals (for-fee consultants).
Data Center Energy
Practitioner (DCEP) Program
![Page 186: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/186.jpg)
186 | Lawrence Berkeley National Laboratory eere.energy.gov
Training & Certificate Disciplines, Levels, and Tracks
Cooling
Systems
Air
ManagementElectrical
Systems
IT
Equipment
HVAC
(available)
Level 2 “Specialist” (2-day):
Pre-qualifications,
Training/Exam on Select
Disciplines
+ Assessment Process
+ System Assessment Tool
Level 1 “Generalist” (1-day):
Pre-qualifications,
Training/Exam on All
Disciplines
+ Assessment Process
+ DC Pro Profiling Tool
IT-Equipment, Air-Management, Cooling Systems,
and Electrical Systems
(TBD)
Data Center Energy Practitioner
(DCEP) Program
(TBD)
There is also a “Training Track”: Training only (no pre-qualifications and no exam)
![Page 187: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/187.jpg)
187 | Lawrence Berkeley National Laboratory eere.energy.gov
DCEP training is delivered by two training organizations that were selected through a competitive process. The DCEP Program Manager is also providing training.
The training organizations:
• license training and exam content from U.S. DOE
• provide training/exams
• issue certificates.
Access up-to-date program information and complete training schedule at U.S. DOE Center of Expertise for Energy Efficiency in Data Centers:
http://datacenters.lbl.gov/dcep
DCEP Training Organizations
![Page 188: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/188.jpg)
188 | Lawrence Berkeley National Laboratory eere.energy.gov
Energy Star
A voluntary public-private
partnership program
• Buildings
• Products
![Page 189: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/189.jpg)
189 | Lawrence Berkeley National Laboratory eere.energy.gov
Energy Star Data Center Activities
•ENERGY STAR Datacenter Rating Tool
– Build on existing ENERGY STAR platform with similar
methodology (1-100 scale)
– Usable for both stand-alone and data centers housed
within another buildings
– Assess performance at building level to explain how a
building performs, not why it performs a certain way
– ENERGY STAR label to data centers with a rating of 75+
– Rating based on data center infrastructure efficiency
• Ideal metric would be measure of useful
work/energy use.
• Industry still discussing how to define useful work.
•Energy STAR specification for servers, UPSs, storage, and
networking equipment
![Page 190: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/190.jpg)
190 | Lawrence Berkeley National Laboratory eere.energy.gov
http://www1.eere.energy.gov/femp/program/data_center.html
http://datacenters.lbl.gov/
http://www.energystar.gov/index.cfm?c=prod_development.
server_efficiency
https://www4.eere.energy.gov/challenge/partners/data-centers
Resources
![Page 191: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/191.jpg)
191 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 192: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/192.jpg)
192 | Lawrence Berkeley National Laboratory eere.energy.gov
Workshop SummaryBest Practices
![Page 193: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/193.jpg)
193 | Lawrence Berkeley National Laboratory eere.energy.gov
1. Measure and Benchmark Energy Use
2. Identify IT Opportunities
3. Manage Airflow
4. Optimize Environmental Conditions
5. Evaluate Cooling Options
6. Improve Electrical Efficiency
7. Use IT to Control IT
8. Implement Energy Efficiency Measures
Data Center Best Practices Summary
![Page 194: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/194.jpg)
194 | Lawrence Berkeley National Laboratory eere.energy.gov
1. Measure and Benchmark Energy Use
Use metrics to measure efficiency
Benchmark performance
Establish continual improvement goals
Data Center Best Practices
![Page 195: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/195.jpg)
195 | Lawrence Berkeley National Laboratory eere.energy.gov
2. Identify IT Opportunities
Specify efficient servers
(incl. power supplies)
Virtualize
Refresh IT equipment
Turn off unused equipment.
Data Center Best Practices
![Page 196: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/196.jpg)
196 | Lawrence Berkeley National Laboratory eere.energy.gov
4. Manage Airflow Implement hot and cold aisles
Fix leaks
Manage floor tiles
Isolate hot and cold airstreams.
Data Center Best Practices
![Page 197: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/197.jpg)
197 | Lawrence Berkeley National Laboratory eere.energy.gov
5. Optimize Environmental Conditions
Follow ASHRAE guidelines or manufacturer
specifications
Operate to maximum ASHRAE recommended
range.
Anticipate servers will occasionally operate in
allowable range.
Minimize or eliminate humidity control
Data Center Best Practices
![Page 198: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/198.jpg)
198 | Lawrence Berkeley National Laboratory eere.energy.gov
6. Evaluate Cooling Options
Use centralized cooling system
Maximize central cooling plant efficiency
Provide liquid-based heat removal
Compressorless cooling
Data Center Best Practices
![Page 199: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/199.jpg)
199 | Lawrence Berkeley National Laboratory eere.energy.gov
7. Improve Electrical Efficiency
Select efficient UPS systems and
topography
Examine redundancy levels
Increase voltage distribution and reduce
conversions
Data Center Best Practices
![Page 200: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/200.jpg)
200 | Lawrence Berkeley National Laboratory eere.energy.gov
3. Use IT to Control IT Energy
Evaluate monitoring systems to enhance
real-time management and efficiency.
Use visualization tools (e.g. thermal
maps).
Install dashboards to manage and
sustain energy efficiency.
Data Center Best Practices
![Page 201: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/201.jpg)
201 | Lawrence Berkeley National Laboratory eere.energy.gov
Most importantly…
Get IT and Facilities People
Talking and working
together as a team!!!
Data Center Best Practices
![Page 202: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/202.jpg)
202 | Lawrence Berkeley National Laboratory eere.energy.gov
![Page 203: Best Practices for Data Center Energy Efficiency Seminar Hosted … TCC... · 2020. 1. 3. · Seminar Learning Objectives. 5 ... EPA Report to Congress 2008 Projected Data Center](https://reader035.vdocument.in/reader035/viewer/2022070114/6080fb743d420d1d53717d70/html5/thumbnails/203.jpg)
203 | Lawrence Berkeley National Laboratory eere.energy.gov
Dale Sartor, P.E.Lawrence Berkeley National LaboratoryApplications TeamMS 90-3111University of CaliforniaBerkeley, CA 94720
[email protected](510) 486-5988http://Ateam.LBL.gov
Contact Information: