1

CPTE 433 – John Beckett

Chapter 17 – Data Centers

2 / 33

Basic Characteristics

• Large enough• Place for servers (racks)• Wiring infrastructure• Power• Environment• Fire suppression• Extra stuff you need• Access

3 / 33

Where

• 1950-1966 – “Goldfish bowl” model (LCCA server room)

• 1967-2001 – Not in their faces• 2002-? Somewhere nobody knows

– Perhaps even out-sourced– Dilemma: Avoiding accidents/terrorism

versus connectivity

4 / 33

The Room

• Large enough.• How many servers & lines &

workstations?– More!

• Routing for wires– Old way – in the floor (dusty but beautiful)– New way – in the ceiling (clean but ugly)– Ceiling routing is easier to re-do without

damaging what’s there before.

5 / 33

Access

• Control who goes in and out– Physical access is total access– Record entrances, use video camera

• Minimize goings in and out– Stabilizes temperature

• Emergency egress• Path in and out for equipment

– Controlled, ID required & logged

6 / 33

Power

• Enough• HVAC load = power input to room

+/- other heat inflows and outflows.• UPS / ATS / Generator• UPS Bypass switch

– UPS may need servicing– UPS may fail– Remember that flat roof!

7 / 33

Power Distribution

• Use “home run” circuits - one per rack.

• Never feed one rack from another.• Watch out for switches that can be

accidentally turned off.

8 / 33

How Long is a Power-Fail?

• < 10 seconds– Glitch corrected by power company

computer• < 1 minute

– Power company people had to experiment to see where the failure was

• < 1 hour– Truck had to drive out to fix it

• Longer – might as well send people home

9 / 33

How Old Is That Battery?

• Small UPS - replace the UPS, it’s cheaper and the UPS isn’t made to last anyhow.

• Larger UPS - May use external battery system.

10 / 33

Generators

• Tested regularly• Fuel supply maintained

– Can you get more fuel?• Properly mounted and ventilated• How do you switch over to them?

– Avoid back-feeding the power grid!• Can you refuel while in operation?

11 / 33

Power Transfer Systems

• Manual switch– Make sure it cannot be fed by two

sources at the same time• Automatic Transfer Switch (ATS)

– What is the protocol to switch back?– Does the generator match phases?

• “Float” battery power

12 / 33

Lightning Protection

• Room should have a central ground.• Wires coming in should go through

protectors that conduct transients to that ground.

• The more fiber you can use to communicate, the less trouble you’ll have with equipment.

• Consider a rule that nothing but power connects to your data center except through fiber.

13 / 33

Emergency Lighting

• Strategically located lights in room & hallways, should come on automatically

• Flashlights with batteries– NiCD rechargeable are nearly useless for

this– NiMH work better

• Cycle them through a charger– My preference: Alkaline, due to shelf life– A headlamp is also nice to have

• How many flashlights? One to lend, one to use – maybe more

14 / 33

Earthquake Protection

• Fundamental issue is: will equipment fall over or out of racks?– Secure it.

• What might things fall onto?• Whatever you do to handle

earthquakes will help in case of an accident

15 / 33

Fire Suppression Systems

• Can you localize the system?– Otherwise it will take down the whole

room.• Once activated, the system will not

protect you until recharged.• Any gas (Halon, CO2) that doesn’t

support fire, won’t support people.– It might even be poisonous

16 / 33

Avoiding Accidents• Aisles should be at least 36” wide• Nothing hanging into the aisles

– No places that require someone to work under stuff that sticks out

• Use proper procedures– “Run your fingers around the sides and

back of a laptop before taking it away.”• Use common sense• OSHA has rules that make sense

17 / 33

HVAC• An at-rest person emits about 200

watts (700 BTU/hr) in heat dissipation• Ignore square-foot rules of thumb –

pay attention to actual power consumption

• Windows, walls, etc., contribute heat or cold and possibly humidity– From adjacent rooms– From the environment

• Three aspects:Temperature, humidity, clean air

18 / 33

Racks

• 19” horizontal distance across front panel of equipment.

• Vertical units are 1-3/4”• Telecomm equipment may be in 21-inch racks (but

that’s changing to 19”).• Depth - enough for servers, data, power wiring.• Rails used to support back of servers.• Use cage nuts because they wear better.• Standardize on a rack and keep them all the same.

– Otherwise you’ll have a pile of hardware that doesn’t fit the rack you’re needing it for!

19 / 33

Rack Orientation

• Rows of racks should be back-to-back• Front of racks: cool aisle• Back of racks: hot aisle• HVAC should be “plumbed”

accordingly• Support natural convection• Avoid feeding one server’s hot air

into another server• Our method: Use a plenum curtain

20 / 33

Fans

• Fans keep things cool.– Does that fan actually send the heat

elsewhere?– Does that fan actually pull in air from a

cooler place?• Fans eventually die.• What’s your plan to:

– Check them?– Replace them?

Convection

• Your HVAC airflow should be designed to work with convection, rather than against it

• Convection devices can get clogged

21 / 33

22 / 33

Cable Management

• Overhead is best• Horizontal “ladders”• Curved adapters• Vertical channels into your racks• Velcro is wonderful for bundling

– Don’t use wire ties - they slow down signals if too tight, and can’t be adjusted.

23 / 33

Don’t Make Cables

• Failure rate is very high on hand-made cables

• Factory-made cables are inexpensive in quantity

• Cost of failures, especially intermittent failures, is very high

• Exception: Specialized cables in non-critical runs

24 / 33

Non-Rackmount Equipment

• UPS• Dehumidifier• Roll-around cart with diagnostic PC

25 / 33

Color-Coding Wires• Use sparingly

– Half of male people are measurably color-blind

– You want important stuff to stand out– You may not have the right color handy

when wiring something• Suggestion: Use color coding for only

2 functions– Backbone– Server management subnet

26 / 33

Pre-Wiring

• Standardize your layout of racks & datacomm

• Fully wire everything at the outset• Consistent wiring scheme

– less headaches• Faster implementation• Make interconnections at the master

patch panel the only variable

27 / 33

Inter-Cable Interference

• Data cables belong with data cables• Power cables don’t belong with data

cables• Wire power down one side of the

rack, and data on the other

28 / 33

Cable Marking

• Copper cable has footage markers - 2’

• Fiber cable has meter markers - 1m– (This may change)

• Difficult to mark ends of cable– Brady is the company of choice for this

(not as bad as the others)• Looseness in cables allows “wiggle

testing.”

29 / 33

Cable Testing

• Common procedure is “frogging”• Switch both cables at both ends• See if:

– problem moves (cable trouble)– or doesn’t (equipment trouble)

• Or…use OTDR• Or…use cable certifier

30 / 33

Communications

• Need a phone line that does not depend on network to work!

• Wireless phone - best quality you can find

• Some way of using the phone while keeping your hands free

31 / 33

Workbench• Connections to all subnets

– Perhaps using a soft switch of some sort

• Plenty of power outlets• Anti-static gear• Label maker• Monitors• Spare keyboards & mice• Tools - neatly on a rack so they go

back

32 / 33

Spare Parts

• Inventoried• Status known and marked clearly• Valuable parts should be:

– Visible – Locked down– (See cages at Sam’s Club)

33 / 33

Documentation and Media• Book shelves• Media and licenses for each purchased

piece of software, identified as to which computer it resides on.– May have a single disk set with a list of licenses

• Reference works as needed• Logs of changes• Internally-generated documentation (KB or

website)• Database for your jumper wiring

34 / 33

The Care You Give…

• First UNIX system on campus– Used junk PC components– Kept in my office– Managed as if it were important

• Result: mainframe-style uptime• Just a thought: If you treat people as

if they are important, they might act as if they are worthy.