canheit- data center - 02-g - university of · pdf filecabling • was preventing ......

18/06/2012

1

Cost effective data centre: How uOttawa leveraged a

relocation to deliver a modernized data centre

Joel Breton

Paul Mercier

University of Ottawa 1

Agenda

1. uOttawa Facts

2. Previous Data Centre

3. Relocation Process

4. Challenges

5. New and improved Data Centre

6. Accomplishments

7. Questions?

2

18/06/2012

2

uOttawa Facts

• World’s largest bilingual university (French/English)

• Located in Ottawa, Ontario

• 40,000+ students

• 250 undergraduate programs and 200 graduate programs in 10 faculties

• One of Canada’s top 10 research universities

• Over 5,100 staff members

• 280 IT staff in central IT (CCS) and in faculties and services (50-50 split)

3

Out with the Old, In with the New

4

18/06/2012

3

Previous Floor Plan

5

Some pictures…

6

18/06/2012

4

Power and Cooling• No Hot/Cold aisles concept

• UPS at Maximum Capacity

• No Power resiliency

• No standard way of distributing power

• Old equipment (Liebert, MGE, etc.)

• Weak Documentation on power

distribution (diagrams, labels, etc.)

7

Summer 2010 – It was HOT…

8

18/06/2012

5

Cable Distribution and Rack Location

• Years of technology changes

• No importance was given to the upkeep of the

cabling

• Was preventing good airflow and cooling

distribution

9University of Ottawa

Physical Server Access

• No standards for Remote Access to servers

• Multiple KVM and Console

• Incomplete Management Network


18/06/2012

6

Networking

• Switches random location

• Distinct group for network and storage area

network

• No Central IP Management

• Maximum 1 Gbps


Storage Area Network (SAN)

FastT 700 DS4800 DS5020 DS5100 EMC FCBR

Switch00 Switch01 NT1A NT1B Switch04 Switch05 Switch06 Switch07

BC07 BC07

BC01 BC02 BC04 BC05 Switch02 Switch03 BC06


18/06/2012

7

Storage Area Network (SAN)

Storage Cabinet Server Cabinet


Storage

• Distributed randomly

• Lack of technical expertise, documentation

and standards.

• Unable to retire Legacy

• No structure

• No tiered strategy


18/06/2012

8

Strategy for the New Data Centre

• Address critical weaknesses: power and

cooling, capacity, scalability, sustainability, etc.

• Improve the efficiency and the reliability of

the Data Centre by modernizing the

equipment and by retiring legacy

15

Planning

• Multiple iterations of the floor plan to optimize

space and usage

• Hiring of a dedicated Project Manager

• Documentation and Inventory

• Server Virtualization

• Storage Virtualization

• Network Infrastructure

• Fabric Design & Renewal

16

18/06/2012

9

Relocation Process

• Server Relocation

• Storage Relocation

• Network/Fabric

Relocation

• Mainframe Relocation

• Change Management

• Defined Responsibilities

• Pilot the process

• Control Centre

• Check List

17

Challenges during the move

• Building still under construction

• Organizational issues: roles and staffing,

physical location of staff, multiple re-orgs

before and during the move.

• Operational issues

• Technical issues with fabric/SAN, load

balancers, etc.

• Process issues. Ex: forgot to turn on a server.

18

18/06/2012

10

New Building

19

Floor Plan of New Data Centre

20

18/06/2012

11

Hot Aisle Containment System

(HACS)

21

HACS.MOV

Power and Cooling

• Based on APC InfraStruxure

• Capacity of 500 kW, redundant

• Under-the-floor power distribution

• In-Rack Power resiliency (A/B Design)

• Current UPS Battery Time: ~110 minutes

• In-Row Chilled Water Cooling based on existing Campus

infrastructure, with redundant chillers for the building

• Cooling pipes are hidden under the floor to prevent leak

damages

22

18/06/2012

12

Top of Rack and Middle of Row

Cable Distribution


Patch Panels

24

18/06/2012

13

Server Access

• IP KVMs to replace all local and KVM consoles

• End or Row Console Access

• Remote Access for geographically dispersed

staff


Servers• Virtualization of servers

• Increased density of racks and servers

• FCoE implementation for virtual hosts

0

200

400

600

800

2008 2009 2010 2011 2012

Number of Virtual Servers


18/06/2012

14

Networking


Improved SAN

Edge Edge

Core Core

Edge Edge Edge Edge Edge Edge

DS4800

ServerA ServerB ServerC

DS5020 DS5100 DS5300


18/06/2012

15

Storage Improvement

• Storage Physical Layout

• Storage Virtualization

– Thin provisioning

– Legacy retirement

– Storage tiers

• Storage growth


Accomplishments

• More then 15,000 hours of labour

• Over three years for the planning, building of the

new data centre and execution of the move

• Documentation of existing infrastructure

– Labelling of all assets, including cables

– CMDB in service-now

• Cable Distribution

– 6 km of fibre

– 6 km of ethernet


18/06/2012

16

Some Pictures

31

Some Pictures

32

18/06/2012

17

Overall: Success

• Energy Recycling: up to 80% of the heat generated by the

equipment is recuperated to heat up the building during

Winter months

• Energy Savings (power and cooling): from spending $700 per

year for a physical server to $20 for a virtual one

• Increased efficiency in the use of electricity

• Reduced data centre floor space

• Increased efficiency of data centre assets

• Standardized Infrastructure Architecture


Lessons Learned

• Start virtualization as early as possible and virtualize

everything you can

• Start small, for testing

• Ensure Commitment throughout the organization

• Communication

• Involve all stakeholders early in the process

• Some things can and will go wrong. Ex: switch failure

• Minimize changes happening during the move.

34

18/06/2012

18

Next Challenges

• Foundation for the Cloud

• Disaster Recovery & Business continuity

• Co-location option

35

Questions?

Joel Breton

[email protected]

(613)562-5800 x4192

Paul Mercier

[email protected]

(613)562-5800 x3947

University of Ottawa 36

canheit- data center - 02-g - university of · pdf filecabling • was preventing ......

Documents