Download - CSC Modular Datacenter
![Page 1: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/1.jpg)
Kajaani Datacenter Case-‐Study
Peter Jenkins System Specialist & Project Manager
CSC – IT Centre for Science
![Page 2: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/2.jpg)
![Page 3: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/3.jpg)
Contents
! About CSC – High Performance Computing
! Drivers – Why a new datacenter? – Why Kajaani?
! The project ! What we built ! Construction and commissioning ! Lessons learned
![Page 4: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/4.jpg)
I’m from there
![Page 5: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/5.jpg)
Who am I?
! Computer Science background ! 6 years at Sun Microsystems ! HPC consultant in Financial Services ! 3 years in Finland working for CSC ! I’m not a datacenter specialist!
– It helps!
![Page 6: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/6.jpg)
My role at CSC
IT • Rapid change • New hardware
challenging datacenters
Facilities • Traditionally very stable • Disruptive innovations in
last 5 years
• Traditionally separate departments must now work together
• I try to bridge the gap
![Page 7: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/7.jpg)
About CSC • 100% owned by ministry of educaGon
– Public sector, Non profit • Major service areas:
– High Performance CompuGng (HPC) – Storage and archive – FUNET – Academic network in Finland – Managed services and IaaS cloud – Server hosGng and co-‐lo
![Page 8: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/8.jpg)
CSC and High Performance Computing
![Page 9: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/9.jpg)
![Page 10: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/10.jpg)
CSC CompuGng Capacity 1989–2011
![Page 11: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/11.jpg)
![Page 12: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/12.jpg)
338
227
146 141
124
95
42 37
35 34 167
Users of CSC resources by discipline 2011 (total 1386 users)
Biosciences
Physics
Nanoscience
Language research
Chemistry
Grid usage
ComputaGonal fluid dynamics
Structural analysis
ComputaGonal drug design
Earth sciences
Other
![Page 13: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/13.jpg)
WHY A NEW DATACENTER?
![Page 14: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/14.jpg)
0
1
2
3
4
5
6
7
8
9
10
2005 2006 2007 2008 2009 2010 2011
Energy GWh
CSCs Datacenter Energy Use 2005 -‐ 2011
DC 2 Infra DC 2 IT
DC 1 Infra DC 1 IT
![Page 15: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/15.jpg)
-‐ €
10 €
20 €
30 €
40 €
50 €
60 €
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
Elspot prices FI in EUR/MWh
![Page 16: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/16.jpg)
New site drivers
• Capacity limiGng growth – Super computers – Managed hosGng
• Costs increasing – Tax – Energy – Carbon neutral
• New site focused on efficiency and capacity
0
1
2
3
4
5
6
7
8
9
10
2005 2006 2007 2008 2009 2010 2011
Energy GWh
CSCs Datacenter Energy Use 2005 -‐ 2011
DC 2 Infra DC 2 IT
DC 1 Infra DC 1 IT
![Page 17: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/17.jpg)
2011 Energy cost per MW/hour
Espoo Kajaani
VAT Tax Transfer Electricity
![Page 18: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/18.jpg)
WHY KAJAANI?
10/10/2012 18
![Page 19: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/19.jpg)
Where is Kajaani?
![Page 20: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/20.jpg)
![Page 21: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/21.jpg)
![Page 22: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/22.jpg)
![Page 23: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/23.jpg)
national grid
fibre
fibre
![Page 24: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/24.jpg)
![Page 25: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/25.jpg)
![Page 26: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/26.jpg)
![Page 27: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/27.jpg)
![Page 28: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/28.jpg)
![Page 29: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/29.jpg)
Electricity reliability
29
Finnish naGonal grid Renforsin Ranta substaGon
![Page 30: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/30.jpg)
Fingrid connections to Kajaani
![Page 31: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/31.jpg)
Power capabilities
! Within perimeter fence: – National grid connection access to 340 MW – 110 kV / 10 kV main transformer capacity
! Current capacity 240 MW
– Biopower on site
! Green power options – 3 hydro power plants within 3 km
! feeding directly to site.
! Diverse power supply = reliable power
![Page 32: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/32.jpg)
![Page 33: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/33.jpg)
Related datacenter sites
Facebook: Luleå 120 MW Free air cooling
CSC: Kajaani 1.4 MW Free air cooling 0.9 MW Water cooling PUE design
Google: Hamina ??? MW Sea water cooling
CSC: Espoo 1.6 MW ConvenGonal PUE 1.4 & 1.8
![Page 34: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/34.jpg)
Government support
! Google Hamina = wakeup call – Unused assets ideal for fast growing industry – Jobs, skills, international competitiveness
! Government acted – Regional development money – Extra money to CSC to build a new site
! Site selection: long story short – Initial concept study 2010 – Several former paper mills considered – Kajaani was successful in bidding
![Page 35: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/35.jpg)
THE PROJECT
![Page 36: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/36.jpg)
Approach
! Design goal: multi-MW facility PUE <1.2 ! Leverage features of site
– Matched to business requirements – Avoid redundancy and backup
! Only 100kW UPS from day one – < 5% of load – core network, management, automation – Emphasis on monitoring and rapid recovery
! No generators day one ! Option to add 100% UPS and generators
![Page 37: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/37.jpg)
Approach continues
! Free cooling year round ! Use modular to right-size and scale quickly ! Green
– CSC buys certificates of carbon neutral energy – 100% Finnish hydro power
![Page 38: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/38.jpg)
Outside
Timeline
Paper mill
2010 2011 2012
Site selecGon
Planning
Warehouse
2013
1st ITT
Datacenter
Supercomputer
October
2nd ITT
Analysis
Delivery Conven&onal
build
Modular build
![Page 39: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/39.jpg)
WHAT DID WE BUILD?
![Page 40: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/40.jpg)
CSC’s new Kajaani datacenter
! Renovated paper warehouse ! 12 000 m2 of space ! ~ 1.7 x Old Trafford football ground
![Page 41: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/41.jpg)
Warehouse has two DC tenants
![Page 42: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/42.jpg)
Hub datacenter
SGI Ice Cube R80
TWO DATACENTERS IN ONE
![Page 43: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/43.jpg)
3D WALKTHROUGH VIDEO
![Page 44: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/44.jpg)
Specification
! 2.4 MW combined hybrid capacity ! 1.4 MW modular free air cooled datacenter
– Upgradable in 700kW factory built modules – Order to acceptance in 5 months – 35kW per extra tall racks – 12kW common in
industry – PUE forecast < 1.08 (pPUEL2,YC)
! 1MW HPC datacenter – Optimised for Cray super & T-Platforms
prototype – 90% Water cooling
![Page 45: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/45.jpg)
Hub building and DC
![Page 46: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/46.jpg)
Hub datacenter
Hub building
! Redundant network rooms ! 100kW UPS – upgradable to MW’s ! 10KV switchgear ! Fire suppression ! Storage rooms – warehouse is cold ! Hub Datacenter
![Page 47: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/47.jpg)
Hub Datacenter
Chillers 2 x 450 kW
External dry coolers
![Page 48: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/48.jpg)
![Page 49: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/49.jpg)
Hub DC facts
! Due November ! 900kW water cooling
– + 100kW air from hub
! Purpose built for water cooled HPC
![Page 50: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/50.jpg)
IT summary
• Cray “cascade” supercomputer – 10M€ -‐ five year contract – Fastest computer in Finland due mid-‐November – Phase one 385 kW 2012 of Intel processors – Very high density, large racks
• T-‐Plamorms prototype – Very high density hot-‐water cooled racks – Intel processors, Intel and NVIDIA accelerators – TheoreGcal 400 TFlops performance
![Page 51: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/51.jpg)
SGI Ice Cube R80
![Page 52: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/52.jpg)
![Page 53: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/53.jpg)
SGI Ice Cube R80
![Page 54: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/54.jpg)
SGI Ice Cube R80
! One head unit and two expansion modules ! More modules can be added ! Fully automated free cooling system
– Dozens of cooling fans, louvers and sensors ! Extremely energy efficient – pPUE 1.08 ! Set point allowed to vary (10-27C for us) ! Adiabatic cooling on warm days ! Exhaust heat used to warm incoming air
![Page 55: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/55.jpg)
Factory visit August 2012
![Page 56: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/56.jpg)
![Page 57: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/57.jpg)
![Page 58: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/58.jpg)
![Page 59: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/59.jpg)
! “Super Cluster” – 4.5M€ five year contract – 1,152 Intel CPUs – 190 TFlop/s – 30kW 47U racks
! HPC storage – 3PB of fast parallel storage – Supports Cray and HP systems
IT summary
![Page 60: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/60.jpg)
CONSTRUCTION
![Page 61: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/61.jpg)
Site in January 2012
![Page 62: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/62.jpg)
MDC TIME LAPSE VIDEO
![Page 63: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/63.jpg)
Head load test
• 700kW of load banks – 20 racks – 120 x 6kW load banks – 2 days to rack mount – Half MDC capacity
• pPUE 1.05 • Very useful • Silenced the criGcs
![Page 64: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/64.jpg)
Commissioning
! For MDC it took longer than construction!
! MDC internal workings and drawings are sensitive – Test plan process challenging – Training required for new free
cooled system type ! Where possible plan in
advance ! Allow plenty of time
![Page 65: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/65.jpg)
LESSONS LEARNED
![Page 66: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/66.jpg)
Lessons learned
! Simplify your procurement! ! Share every detail of your site with bidders ! A modular datacenter is still a datacenter
– Even if design fixed is expect many decisions ! Pay special attention to integration
– fire, security, power, electricity, etc. – Any small issue can cause major delay
! Get expert advice – don't assume!
![Page 67: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/67.jpg)
Reference or define every ’standard’ Don't assume
! Racks – Define depth, height, RU, posts – Strongly consider alternatives like
opencompute ! Door heights ! Make a list of required standards
– Fire suppression – insurance, building permit ! Get a datacenter contract
– Modular buildings contracts != IT contracts – If possible multiple phases of acceptance
![Page 68: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/68.jpg)
Things we got right
! Make the selection based on 10 year TCO – IT TCO was 5 years per phase – highlights energy efficiency without defining 'green’ – could also use an artificially high energy price
! Require that IT vendors list – PDU efficiency use 80Plus – Temperature and humidity ranges – ASHRAE TC 9.9 2011
! All our suppliers support A2 range (5 to 35C)
![Page 69: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/69.jpg)
Datacenter operations team
![Page 70: CSC Modular Datacenter](https://reader034.vdocument.in/reader034/viewer/2022051513/546e9e22b4af9fc3268b472c/html5/thumbnails/70.jpg)
QUESTIONS? Thanks