yogendra joshi ga_tech

TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 1

Characterizing Air Flows and

Thermal Responses to Help

Sustainable Data Centers

Yogendra Joshi

G.W. Woodruff School of Mechanical Engineering

Georgia Institute of Technology

Atlanta, GA 30332, U.S.A.

Acknowledgements:

Intel, IBM, Department of Energy, National Science Foundation (NSF), Triad Tiles, APC

TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

Measuring Air Flows in Data Centers

Designed and built by McKenney’s


42 U

1.98m.

(6'-6")

10 U

10 U

10 U

10 U

0.53m.

1'-9"

0.44m.

1'-5 1/2"

0.44m.

1'-5 1/2"

0.44m.

1'-5 1/2"

0.44m.

1'-5 1/2"

0.09m.

3 1/2"

Details of the CEETHERM data center and the

location of the server simulator

Power

distribution

unit

Power

distribution

unit

D

O

O

R

Storage

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

Network

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

Compute rack

35

71

3

24

14

Empty rack

Test rack

Test rack

Empty rack

Empty rack

Test rack

Test rack

Server

Simulatror

D

O

O

R

Empty rack

6

91

1

8

Test rack

10

12

Downflow

CRAC 1

Upflow

CRAC 2

Upflow

CRAC 3

Downflow

CRAC 4

Downflow CRAC 5Downflow CRAC 6

1

Power

distribution

unit

Experimental section Computational section

8.7

5m

6.4m 6.4m

PIV measurement plane

Note:

Perforated tile dampers

on tiles 1-4, 7,11-14 are

closed during the

experiments

Only Tiles 5,6,8,9,10 are open during the experiment. All other tiles are closed. CRAC 1 and CRAC 6 are simultaneously operated


• 3-D Stereoscopic PIV (Particle Image Velocimetry

system for room level air flow mapping.

• 22.8 kW Server Simulator with adjustable fan and

heater settings to simulate a variety of heat loads.

Experimental Facilities

3-D PIV system

Server Simulator

Fan speed

and Heat

setting

dials

Perforated floor

tiles with dampers


How much air flow rate is required to cool a 20kW rack?

Typical Raised floor air

distribution system Server Air requirements

A 20kW Server requires ~ 0.996 m3/s

(2,110 CFM for a ∆T=20°C)

A 2x2 (4 ft2, 0.3716 m2) opening with no

tile V ~ 2.68 m/s (~6 mph)

Only tile (no Damper) 56%opening; Area

= 0.208 m2 V ~ 4.788 m/s (~10.7

mph)

Tile with dampers (26% open); Area =

0.0977 m2 V ~ 10.2 m/s (~22.8 mph)

Server hot

air exit

Tout =47°C

Perforated floor

tile

Servers

Server pressure drop

Server cold

air inlet

Tin=27°C

∆T=20°C


Distance measured from the centre of cold aisle (mm)

Hei

ghtof

the

serv

ersi

mula

tor

mea

sure

dfr

om

the

per

fora

ted

floor

tile

(mm

)

150 300 450 600

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

1950

Velocity (m/s)

6

5.5

5

4.5

4

3.5

3

2.5

2

1.5

1

0.5

0

-2

-1

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700

Distance measured from the center of the cold

aisle (mm)

Velo

city

(m

/s)

U u-component v-component


Tile

Surf

ace

150 300 450 600

0 .5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Velocity (m/s)

Tile surface

Tile surface

Highly non uniform flow.

Reversed flow at the ends result in

air being sucked into plenum

Peaks and valleys in velocity field

relate to damper geometry Aisle Center

Hei

ghto

fthe

serv

ersi

mul

ator

mea

sure

dfr

omth

epe

rfor

ated

floor

tile

(mm

)

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

Velocity (m/s)

0.8

0.792

0.726

0.66

0.594

0.528

0.462

0.396

0.33

0.264

0.198

0.132

0.066

0

0

200

400

600

800

1000

1200

1400

1600

1800

2000

-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8

Velocity (m/s)

Rack H

eig

ht

(mm

)

U u-component v-componentAisle center

Aisle center

u-component suggests entrainment

Entrainment predominant at the tile surface and subsides at the aisle top

-ve v component is due to reversed flow at the tile surface

Rack air flow boundaries for a 22kW rack (0.754 m3/s of tile air flow)

Rack

inlet

Rack InletH

eigh

toft

hese

rver

sim

ulat

orm

easu

red

from

the

perf

orat

edflo

ortil

e(m

m)

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

1950

Velocity (m/s)

5.5

5.496

5.038

4.58

4.122

3.664

3.206

2.748

2.29

1.832

1.374

0.916

0.458

0

0

200

400

600

800

1000

1200

1400

1600

1800

2000

-2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5

Velocity (m/s)

Rack H

eig

ht

(mm

)


Rack inlet

Deviation from normal to slanting

Air flow at the bottom half of the rack is affected by the perf. tile flow

25% reduction in flow compared to case where of no air supply from tile


Ais

leT

op

0 150 300 450 600

0 1.25 2.5 3.75 5

Velocity (m/s)

-1

0

1

2

3

4

5

0 100 200 300 400 500 600 700


aisle (mm)

Velo

city

(m

/s)


Top aisle boundary

Aisle top 65% of perf. tile air flow is unutilized

and escapes from the top.

Ascending column of air is pushed

towards the rack inlet.

Compression due to entrainment

causes a steady increase in v-comp

velocity

Rack InletH

eigh

tof

the

serv

ersi

mul

ator

mea

sure

dfr

omth

epe

rfor

ated

floo

rtil

e(m

m)

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

1950

Velocity (m/s)

5.5

5.496

5.038

4.58

4.122

3.664

3.206

2.748

2.29

1.832

1.374

0.916

0.458

0

0

200

400

600

800

1000

1200

1400

1600

1800

2000

-2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5

Velocity (m/s)

Rack H

eig

ht

(mm

)


-2

-1

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700


aisle (mm)

Velo

city

(m

/s)



Tile

Surf

ace

150 300 450 600

0 .5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Velocity (m/s)

Tile surface


Ais

leT

op

0 150 300 450 600

0 1.25 2.5 3.75 5

Velocity (m/s)

-1

0

1

2

3

4

5

0 100 200 300 400 500 600 700


aisle (mm)

Velo

city

(m

/s)


Top aisle boundary

Aisle Center

Hei

ghto

fthe

serv

ersi

mul

ator

mea

sure

dfr

omth

epe

rfor

ated

floor

tile

(mm

)

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

Velocity (m/s)

0.8

0.792

0.726

0.66

0.594

0.528

0.462

0.396

0.33

0.264

0.198

0.132

0.066

0

0

200

400

600

800

1000

1200

1400

1600

1800

2000

-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8

Velocity (m/s)

Rack H

eig

ht

(mm

)

U u-component v-componentAisle center

62% of the air requirement;

about double at air inlet temp.

of 30 oC

TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 7 7


Hei

ghtof

the

serv

ersi

mula

tor

mea

sure

dfr

om

the

per

fora

ted

floor

tile

(mm

)

0 150 300 450 6000

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

1950

Velocity (m/s)

3

2.8

2.6

2.4

2.2

2

1.8

1.6

1.4

1.2

1

0.8

0.6

0.4

0.2

Comparison of cold aisle air distribution for high and low density racks

Study suggests that there is a limit to the amount of useful air discharged from

the perf tile.


Hei

ghtof

the

serv

ersi

mula

tor

mea

sure

dfr

om

the

per

fora

ted

floor

tile

(mm

)

150 300 450 600

0

150

300

450

600

750

900

1050

1200

1350

1500

1650

1800

1950

Velocity (m/s)

6

5.5

5

4.5

4

3.5

3

2.5

2

1.5

1

0.5

0

650 CFM (6.5 kW rack) 2500 CFM (22 kW rack)


Grid based Temperature Measurement

Consist of 256 thermo couple

sensors

Type T Copper Constantan

Thermocouples

Measurement accuracy of

±0.2°C

Response time of 4ms/channel

Complete temperature mapping

of the cold/hot aisle.


Complete 3-D temperature mapping

1

2

3

4

5

6

1 2 3

6 planes along the

height

5 planes along the

depth of the rack

10 planes along the

width of the cold aisle


• High velocity discharge

cause adverse pressure

gradients at rack inlet

• Pocket of hot air due to

reversed flow in the cold

aisle.

• Hot pocket in the

lower half of the

rack due to reduced

air intake and

recirculating flows.

Hot aisle

temperature

map

Cold aisle

temperature

map

Velocity profile at Aisle top

Effect of high perf. tile flow (0.667 m3/s, 1414 CFM)) on cold and hot aisle

temperature profiles (TCRAC Supply= 12.3°C)

Velocity profile at perf. tile

surface


Effect of low perf. tile flow (0.227 m3/s, 586 CFM)) on cold and hot aisle

temperature profiles (TCRAC Supply= 12°C)

• Absence of reversed flows

• Hot air pocket has vanished

• Hot air influx into the aisle at

the top

• Higher temperature

gradients in the hot aisle

• Temperatures increase in

the upper half of the rack

Hot aisle

temperature

map

Cold aisle

temperature

map

Velocity profile at Aisle top

Velocity profile at perf. tile

surface


12

Real time server measurements


• Each rack accommodates

6 Blade Center

• Each Blade Center has 14

Blades

• Each Blade has 2

processors (total 168

processors per rack)

• Processor temperature is

measured for each

processor

• Air inlet temperature is

measured for each blade

center

Rack Rear view with fan and the power supply details

CD

LS20

Rack Blade Center

Blade

Processor 1

and 2

BC-6

BC-5

BC-4

BC-3

BC-2

BC-1


100 % CRAC VFD setting (~12,000 CFM)

Effect of Perforated Tile Flow on CPU Temperature and Fan Speed

ComparisonRackD6.wmv


60 % CRAC VFD setting (~7,000 CFM)

Effect of Perforated Tile Flow on CPU Temperature and Fan Speed

CRAC_60_Celcius.wmv


16

Velocity

(m/s)

Static

Pressure

(Pa)

K=13.2

Multi-University NSF I/UCRC E3S Center: Compact Models of Contained

Aisles

yogendra joshi ga_tech

Business

rack air flow boundaries

compute rack compute

tile flow

rack height mm0

tile surface ucomponent

tile v

air flow rate

air requirement velocity