Data Center Retrofit Project: 90% Energy Use Reduction on a Budget

David Gallaher, National Snow and Ice Data Center (NSIDC) Otto Van Geet, National Renewable Energy Laboratory (NREL) Rick Osbaugh, RMH Group

Rocky Mountain ASHRAE 2013

Presented by Affiliations and Sponsorships Design by

So why are we here… • Data centers can waste 90 percent or more of the

electricity they pull off the grid. • Worldwide, there are now more than 3,000,000

data centers that use about 30 billion watts of electricity; ≈30 nuclear power plants.

• “They don’t get a bonus for saving on the electric bill. They get a bonus for having the data center available 99.999 percent of the time.”

• “It’s just not sustainable….They’re going to hit a brick wall.” – Mark Bramfitt, utility and IT consultant

Source: New York Times 9/22/12

The National Snow and Ice Data Center…

Provides tools for

data access

Researches the cryosphere and data science

Educates the public about the

cryosphere Supports data users

Manages and distributes scientific data

Supports local and traditional

knowledge

So what is this about? • NSIDC is a center under the CIRES Institute at the

University of Colorado, Boulder • The NSIDC is a NASA-essential services facility

and a NSF Arctic archive • The NSIDC takes polar orbiting satellite data and

processes, archives and distributes it worldwide • The center contains petabyte-scale disk arrays • The center uses a virtualized server environment

and bare metal servers, totaling ~100 servers • The 1600 sq ft data center consumed ½ the

power of a 77,000 sq ft research lab (as of 2009)

Agenda • Project Impetus • Cooling Phase • Design Specifics • Construction • Solar Phase • Construction • Results • Summary

Photo Courtesy Andy Mahoney, NSIDC

Objectives • How retrofits can

produce performance on par with new construction

• How inexpensive solutions can deliver low energy use

• Importance of clear goal setting by project management

• Benefits of open, collaborative design environment

Photo Courtesy Robert Massom, NSIDC

Photo Courtesy Ted Scambos and Rob Bauer, NSIDC

PROJECT IMPETUS

Original Plan: CRAC UP at NSIDC • 2008 original project was to replace 20-ton

CRAC (“the rusting hulk”) with 30 ton unit • Reliability issues (cottonwood, over pressure) • Cost estimate of $160,000 installed • Crane and elevator follies

NSIDC Research Lab #2 at UC

NSIDC 1600 sq ft datacenter consumed ½ of the total power for 77,000 sq ft building.

Could We Do Better? • Irony of NSIDC research contributing to the

problem we are studying. • The data center used 96kW (51kW load for data

center equipment, 45kW for cooling). Are there other options?

• Rather than cool the air in a closed system, couldn’t we just dump it outside?

• Assumed initial budget of $200,000 • Could it be built with off-the-shelf components? • What was available and economically feasible?

Air Economizer Hours

Could we use an air economizer? What about contamination? Source: http://www.thegreengrid.org/~/media/Files/FreeCoolingTool/free_cooling_air_N_Amer_lowResv3.ashx?lang=en

NSF and NASA Funded Project • Installation of a replacement cooling system featuring air

economizer and indirect evaporative cooling systems • Consolidation of IT facilities in a single-server room • Virtualization of servers and arrangement of systems in a

hot/cold-aisle rack configuration • Solar panels to generate electrical power • A solar battery array, connected to the solar system, to

substitute for the lack of a backup generator • Upgrading internet connectivity to 1gb to better support

the needs of NASA data archive users • The systems will reduce the computer center’s carbon

footprint and support NSF’s cyberinfrastructure goals

NSF/NASA Green Data Center Proposal:

NSIDC Green Data Center Old Configuration New Configuration

Our Solution was Not Conventional • In science, your client will not fund you if your

proposal does not push the edge • In engineering, your client will not accept your

proposal if it has not been done somewhere else • We were well aware that this was a new idea • We needed to educate the mechanical engineers on

our tolerance to risk (we had a backup 30-ton CRAC unit)

• We needed to educate the mechanical engineers what was possible with this new cooling technology

COOLING PHASE

Engineer’s Original $375,000 Proposal

• Not off-the-shelf • Maintenance and

parts issues • Roof unit 12’ high

and 30’ long • Requires visual

screen and railing • Subject to freezing • Would require the

CRAC 1000 hrs/year • Way over budget

Red Rocks Data Center: a $27,000 Solution

• Housed in old satellite tracking facility

• School of Mines (initial design)

• Used unfiltered outside air

• Backed up by conventional A/C

• Not suitable for NSIDC, but interesting ideas

Perseverance for Performance: A Three-Year Odyssey

OCT 2008

JAN 2009

MAR 2009

MAY 2009

JUN 2009

SEP 2009

JAN 2010

AUG 2009

NREL consults with NSIDC and RMH

$375,000 design proposed

Tour $27,000 solution

NASA agrees to $200,000 budget

NSIDC wins internal competition to submit to NSF

NSF solicitation for facility upgrades NSIDC

desire for off-the-shelf system

Investigate CRAC replacement options

Search for Out-of-the-Box Design • We needed someone

to work within our specs and budget. We needed a miracle!

RMH Conceptual Design

Perseverance for Performance: A Three Year Odyssey

OCT 2008

JAN 2009

MAR 2009

MAY 2009

JUN 2009

SEP 2009

JAN 2010

AUG 2009

FEB 2010

NREL consults with NSIDC and RMH

$375,000 design proposed

Tour $27,000 solution

NASA agrees to $200,000 budget

NSIDC wins internal competition to submit to NSF

NSF solicitation for facility upgrades NSIDC

desire for off-the-shelf system

Investigate CRAC replacement options

RMH proposes design with 95% reduction in cooling energy

COOLING DESIGN SPECIFICS

Cooling System Design Considerations • Consolidated, reconfigured data center • Favorable low-humidity climate • Improvements in the design of

alternative cooling systems • New 2008 ASHRAE standards • IT equipment that can tolerate higher

temperatures • Third floor (directly below the roof) • Eliminate compressors, if possible • Exceed reliability standards • Move cooling units inside • Decouple AHU and cooling system

2008 ASRAE Revised Standard

Psychrometric Bin Analysis Zone1: DX Cooling Only

Zone2: Coolerado + DX (H80)

Zone3: Coolerado Only

Zone4: Evap. Cooler Only

Zone5: Evap. Cooler + Economizer

Zone6: Economizer Only

Zone7: 100% Outside Air

Psychrometric Bin Analysis: Boulder, CO

Design Conditions (0.4%): 91.2 db, 60.6 wb

How the Indirect Evaporative Units Work

1 Fresh air drawn in

2 Air is filtered

3 Heat and mass exchanger

4 Half of air saturated with water and returned to atmosphere carrying heat energy

5 Conditioned air to facility

Technology Description • Innovative indirect

evaporative cooling • 22-stage cooling with

wet-bulb effectiveness from 90% to 125%

• No moisture added to the product airstream

• Fan is the only major energy input

0

50

100

150

200

0

50

100

150

200

250

300

350

10

15

20

25

30

35

Y (mm)

y u b e pe atu es (deg C) d ect S de

Tem

pera

ture

o C

2040

6080

100120

140160

180200

0

50

100

150

200

250

300

350

10

15

20

25

30

Y (mm)

Dry Bulb Temperatures (deg C) - Direct Side

Tem

pera

ture

o C

Multiple passes of indirect

System Advantages • Cost-effective components

– Single air handling unit is simple, inexpensive – Fan, filters and dampers sized for low pressure drop, low fan

energy use • The units are expandable and modular • Precise control of temperature and humidity • No compressors used in design = greater reliability • Energy savings guaranteed; no equipment to run except

fans; 24/7 operation; lower maintenance costs • Connected HP for system is so low that it can be run from

UPS (fan is always run from UPS). • Low energy use > 90% less than existing system

Old System: Summer Operation

Two 25 HP Compressors

Four ¾ HP Condenser Fans

Refrigeration Pipe

Total system = 45 kW

Old System: Winter Operation

Two 25 HP Compressors

Four ¾ HP Condenser Fans

Refrigeration Pipe

Total system = 46 kW

New System: Summer Operation 1 Economizer Air Supply

2 Product Supply Air

3 Aisle Supply Air

4 Hot Aisle Return Air

5 Relief Air

6 Return Damper Air

7 Outside Air

8 Working Air Exhaust

Total system = 6 kW

Cooling Units Product Air Side with Working Air Humidification Dampers

Making Use of a Waste Product • Outside air humidity from

the air-economizer can drop below 10%

• The units produce cool moist air as a waste product

• Installing dampers allows the units to function as humidifiers

• During winter months, the units function only as humidifiers

• During summer months, they both chill & humidify

COOLING SYSTEM CONSTRUCTION January – September 2011

Machine Room Before Construction

4 Duct Holes

Air Handler Unit Lifted to Roof

8 am, Wed.,

March 30th 2011

96 hours

Six Trades at Once: Mechanical, Electrical, Plumbing, Sprinkler,

Sheet Metal, Controls

Ready for re-start, 1 am Saturday morning

Back Online – Saturday April 2, 3 pm

Reconfigured Data Center Servers in Hot Aisles

Installed Cooling Units with Input

Air Duct

Perseverance for Performance: A Three-Year Odyssey

OCT 2008

JAN 2009

MAR 2009

MAY 2009

JUN 2009

SEP 2009

JAN 2010

AUG 2009

FEB 2010

JUN 2010

SEP 2010

NREL consults with NSIDC and RMH

$375,000 design proposed

Tour $27,000 solution

NASA agrees to $200,000 budget

NSIDC wins internal competition to submit to NSF

NSF solicitation for facility upgrades NSIDC

desire for off-the-shelf system

Investigate CRAC replacement options

NSF awards NSIDC facility upgrade grant $525,000

RMH proposes design with 95% reduction in cooling energy

University issues overcome, construction RFP released, $225,000 bids

JAN 2011

NSF releases funds!

APR 2011

Cooling system installation

MAY 2011

Hot aisles and systems shakedown

SEP 2011

Cooling phase completed

SOLAR SYSTEM CONSTRUCTION February – May 2012

Solar System Design Considerations • Cd/Te cadmium/telluride solar PV panels

• 50 kW output (78,000kWh/year) • 720 70-watt panels ($83/panel) • Flat-mount panels for higher density • High wind load (rated to 120mph) • No penetrations of roof (concrete block

ballast) • Acts as backup generator (powers SAN and

cooling system) • Automatic solar failover to battery array • 3 hours of cooling at maximum power • Data center is carbon-zero in daylight

Construction begins with racking system and lift of solar panels

Removed the old 20/10-ton

CRAC unit condensers

50 kW of Solar Panels are Next 720 Abound Solar 70W Panels

5 miles+ of wire

RESULTS

Perseverance Pays Off! • 91% reduction in cooling

energy consumption • New hot-aisle isolation

configuration • Solar array as backup

generator • 70% reduction in total

power consumption • CRAC used 9 hours as

dehumidifier per year! • Increased data capacity • Only maintenance cost is

filters (90% reduction)

0

5000

10000

15000

20000

25000

30000

Jun-

10

Jul-1

0

Aug-

10

Sep-

10

Oct

-10

Nov

-10

Dec-

10

Jan-

11

Feb-

11

Mar

-11

Apr-

11

May

-11

Jun-

11

Jul-1

1

Aug-

11

Sep-

11

Oct

-11

Nov

-11

Dec-

11

Jan-

12

Feb-

12

Mar

-12

Apr-

12

May

-12

Jun-

12

Jul-1

2

Ener

gy U

se (k

Wh)

New Cooling System Energy Use with Projection

Actual New CoolingSystem Energy Use

Projected NewSystem Energy Use(10°C cutoff)

Old CRAC CoolingSystem Energy Use

IT Energy

23,500 kWh Average CRAC Energy

2,560 kWh Average Coolerado Energy

Construction + Transition

Period

Solar Output

This system should generate about 78 megawatt-hours of energy/year

Even on this snowy April 2, we generated enough power to run the cooling system (~1kw) at midday

Energy and Dollar Savings Original Conditions NSIDC Data Center, through 2002-2009: 60-Tons Liebert DX cooling (30,20,10 ton) Average Cooling Power - 45kW Average UPS conversion – 6kW (87% efficient) Average IT Load - 45kW Average annual power consumption: 96kW*24*365 = 841,000 kWh/yr Average power cost at $0.11/kWh = $92,500/year

Current Conditions NSIDC Data Center, August 2011 to present: 40 tons Coolerado Cooling (8 5-ton) Average Cooling Power - 4kW Average UPS conversion – 4kW (87% efficient) Average IT Load - 32kW (Virtual servers) Average annual power consumption: 40kW*24*365 = 350,000 kWh/yr Average power cost at $0.11/kWh = $38,500/year

Average annual savings = 58% or $54,000 50kW Solar Array savings = $8580/year Total Annual Savings = $62,580 Total Energy Reduction = 70%

Project Participants • NSIDC • NASA • NSF • RMH Group • NREL • Coolerado • UCB FacMan • CU Dept of

Engineering • CIRES

• University of Colorado

• Xcel Energy • Raytheon • Abound Solar • SMA • Whitestone

Construction • Lighthouse Solar

All the critical components were built in locally

SUMMARY

Striving for Innovation: Keys to Success • Calculated Risk Taking. Strong desire by NSIDC to

consider new ideas and approaches. • Localized Think Tank. Generating ideas with NREL,

NSIDC, RMH and the University • Forming the Right Team. Pushing, encouraging and

supporting the design effort; true collaboration • Getting Support. Soliciting funding/support from the

NASA and NSF • Accountability. Test and measure performance. • Knowledge Sharing. Publicize success.

A few thoughts… 1. Don't kill a good idea just because there are those who feel (strongly at times)

that it is impossible. The fact that it has never been done is generally a virtue. 2. Build/show your vision by way of a tangible and easy-to-comprehend graphic.

A picture really is worth a thousand words. 3. Bring as many collaborators in as early as possible (government, universities,

associations, manufactures, sponsors) to spread the risk. 4. Failed projects die as orphans, successful projects have multiple fathers.

Accept this. 5. Engineering is inherently conservative and moves in small evolutionary steps.

The best engineers will break out of the box; don’t settle for less. 6. Be persistent and don't settle for lame answers. Be prepared to fire those who

do not deliver. 7. Manage risk. Do not attempt to eliminate it. The project that avoids all risks will

accomplish little and cost too much.

A few thoughts… 8. Have the full team understand the importance/significance of the project. This

will give them buy-in. 9. Have your contractors sign their work where everyone can see it (we had them

sign on the duct work). It improves quality of workmanship. 10. The Project Manager must be the project champion (push through the

difficulties). Do whatever it takes; turn the screw, mop the floors if you must. 11. Every project will have at least 3 crises, which could derail the project. Expect

them and do not delay in resolving them. 12. Scope creep: "As long as we are here, we might as well..." will kill any project.

Stay focused. 13. If successful, seek out awards and be prepared to show others what can be

done. You cannot change the world if no one knows what you did. 14. Celebrate a successful completion with all involved parties. Credit your

sponsors, subordinates and superiors.

And the best reward of all—daylight zero-carbon footprint

Conclusion • This is a prototype project of what can be done on a

small-scale operating data center • Cooling energy was reduced 91% (about what is takes

to run your car’s A/C) • Carbon and overall energy use were reduced by 70% • Payback on the cooling system is under 3 years

(difference of CRAC replacement vs. Coolerado solution and energy savings)

• Project won the Governor’s Sustainability Award, CU Sustainability Award, CIRES Service Innovation Award and the Uptime Institute Green Energy IT Retrofit Award

QUESTIONS?

Courtesy Ted Scambos, NSIDC Courtesy Andy Mahoney Courtesy Ted Scambos and Rob Bauer, NSIDC

CONTACT: David Gallaher: david.gallaher@colorado.edu