energy dissipation is a severe limit to push further the
TRANSCRIPT
© 2011 IBM Corporation
Energy Dissipation is a Severe Limit to Push Further the Computation Performance of ICT Devices: Towards Zero-Emission Datacenters
Dr. Bruno Michel, IBM Research – Zurich, Manager Advanced Thermal Packaging May 06, 2011
© 2010 IBM Corporation2 IBM Research Zurich, Advanced Thermal Packaging
Green Datacenter Market Drivers and Trends
Increased green consciousness, and rising cost of power
IT demand outpaces technology improvements – Server energy use doubled 2003-2008; temporary slowdown due to
economic crisis; resumed growth is not sustainable – Koomey Study: Server use 1.2% of U.S. energy
ICT industries consume 2% world wide energy– Carbon dioxide emission like global aviation
Real Actions NeededBrouillard, APC, 2006
Future datacenters dominated by energy cost; half energy spent on coolingSource IDC, 2009
© 2010 IBM Corporation3 IBM Research Zurich, Advanced Thermal Packaging
From Individual Transistors to the GlobeMorteratsch Glacier, Switzerland, 1985
Morteratsch Glacier, Switzerland, 2007
- Thermal issues propagate up to the world climate- Global length- and decade time-scales involved - A holistic view is required to solve these problems
IT to become part of the solution to climate challenge!
100
kmDa
y
5’00
0 km
Year
40’0
00 k
m10
0 ye
ars
Glacier recession 1985-2007: 400 meters!
Pow
er
Stat
ion
Cont
inen
tal
Pow
erNe
ts
Glo
bal
Tem
pera
ture
© 2010 IBM Corporation4 IBM Research Zurich, Advanced Thermal Packaging
Motivation for Chip Liquid Cooling
Superior thermal properties of liquids compared to airDisadvantage: Increased complexity
Biological vascular systems optimized for mass transport at low pressure
>700 W/cm2 cooling capacity: ΔT = 65K, Δp <0.25 Bar, and branching manifold <1.5 mm
Impinging liquid jets for high heat transfer from hot chip
Thermal conductivity [W/(m*K)]
Volumetric heat capacity [kJ/(m3*K)]
Air 0.0245 1.27
H2 O 0.6 4176
© 2010 IBM Corporation5 IBM Research Zurich, Advanced Thermal Packaging
Zero-Emission Data Centers
High-performance chip-level cooling improves energy efficiency AND reduces carbon emission:– Cool chip with ΔT = 20ºC instead of 75ºC– Save chiller energy: Cool with T > 60ºC hot water– Re-use: Heat 700 homes with 10 MW datacenter
Need carbon footprint reduction– EU, IPCC, Stern report targets– Chillers use ~50% of datacenter energy – Space heating ~30% of carbon footprint
Zero-emission concept valuable in all climates– Cold and moderate climates:
energy savings and energy re-use– Hot climates: Free cooling, desalination
Europe: 5000 district heating systems– Distribute 6% of total thermal demand– Thermal energy from datacenters absorbed
© 2010 IBM Corporation6 IBM Research Zurich, Advanced Thermal Packaging
Aquasar Hot-Water Cooled HPC Cluster for ETH
Aquasar connected to space heating at ETH
Experimental validation: Air vs. cold, vs. hot water cooling
World record in facility-level Efficiency ERE: 7.9 TFLOP/gCO23x smaller datacenter energy cost (saved energy and sold heat)Operative since May 6th, 2010 (Anniversary today)
– Two chassis liquid cooled (green) and one air cooled (red)– Storage server and Infiniband switch (cyan) – Closed cooling loop with 20 liters water and 30 liters per min. flow– Recover approximately 80% of waste heat @ 60ºC
33 QS22 and 9 HS22 Servers
© 2010 IBM Corporation7 IBM Research Zurich, Advanced Thermal Packaging
SuperMUC 3 Petaflop Supercomputer at LRZ In 2012, LRZ (Leibnitz Rechenzentrum) builds a supercomputer based on IBM System x iDataPlex with 19476 CPUs / 157464 Cores, 324 TB Memory, and 10 PetaByte File Space
“SuperMUC” provides previously unattainable energy efficiency by exploiting new Intel processors and innovative IBM hot water cooling.
– Hot Water Cooling technology invented for Aquasar
First general available Hot Water Cooled iDataPlex clusterMost Energy Efficient high End HPC System:
– PUE 1.1 – Green IT– 40% less energy consumption compared to air-cooling– 90% of waste heat will be reused
The system is part of the Partnership for Advanced Computing in Europe (PRACE) HPC infrastructure
Cooling at Rack Level with Rear Door Heat Exchanger
Cooling at Component Level with Hot Water Cooling Technology
© 2010 IBM Corporation8 IBM Research Zurich, Advanced Thermal Packaging
“Value” of Heat in Different Climates
0
10
20
30
40
50
60
70
80
90
-5 20 45
savi
ngs
[%]
Electricity consumptionCO2-emissionEnergy costNet energy cost
Montreal / Anchorage Dubai / Las VegasBerlin / Washington
Basic concept – Moderate climates: energy savings and energy re-use – Hot climates: free cooling on hottest day ever recorded (57ºC)
Longer term developments– Drive desalination with datacenter heat– Drive adsorption chiller with hot water cooled part – Best in cold climate: Space heating– Best in hot and dry climate: Desalination– Best in hot climate: Sorption cooling
Heating degree days (red) and cooling degree days (blue) for different cities
PUE and reduction of energy cost – a) reference (red), direct heat reuse (green),
and with cooling (blue) – b) Energy cost reduction with heating (green)
and cooling (blue) Emission reduction (light green / blue)
0.0
0.5
1.0
1.5
2.0
2.5
New York Washington Texas
PUE
eff [
-]
0
10
20
30
40
50
60
70
80
New York Washington Texas
redu
ctio
n [%
]
© 2010 IBM Corporation9 IBM Research Zurich, Advanced Thermal Packaging
Paradigm Change: Vertical Integration
Global wire lengths reduction
Meindl 05 et al.
System on Chip
3D Integration
Multi-Chip Design
Benefits:High core-cache bandwidth
Separation of technologies
Reduction in wire length
Equivalent to two generations of scaling
No impact on software development
Brain: synapse network
© 2010 IBM Corporation10 IBM Research Zurich, Advanced Thermal Packaging
Limits of Traditional Back-Side Heat Removal
Microchannel back-side heat removal
Heat removal limit constrains electrical design
© 2010 IBM Corporation11 IBM Research Zurich, Advanced Thermal Packaging
Scalable Heat Removal by Interlayer Cooling3D integration requires interlayer cooling for stacked logic chips Bonding scheme to isolate electrical interconnects from coolant
Through silicon via electrical bonding and water insulation scheme
A large fraction of energy in computers is spent for data transportShrinking computers saves energy
cross-section through fluid port and cavities
Test vehicle with fluid manifold and connection
MicrochannelPin fin
© 2010 IBM Corporation12 IBM Research Zurich, Advanced Thermal Packaging
Key Messages and Outlook
Scale up to full size data centersVision of a Zero-Emission DatacenterIT driven solutions for the climate challenge
Reduce emission up to 85% through heat re-useSave 40% energy and reduce energy cost by 2-3 x Roadmap for large efficiency increase in next 15 years
– Interlayer cooling of 3D chips
(149 F)
(140 F)
© 2010 IBM Corporation13 IBM Research Zurich, Advanced Thermal Packaging
Summary
Paradigm changes– Energy will cost more than servers– Coolers are million fold larger than transistors
Moore’s law goes 3D– Single layer scaling slows down– Stacking of layers allows extension of Moore’s law– Reach functional density and connectivity of Human brain
Future computers to look different– Liquid cooling and heat re-use: Aquasar / SuperMUC– Interlayer cooled 3D stacks– Bionic designs Smart Energy
Energy aspects and communication are key– Cooling - power delivery – photonics– Shrink a rack to a “sugar cube”: 50x efficiency– Synergy with solar power Smart Planet
© 2011 IBM Corporation
Acknowledgment
Advanced Thermal Packaging: Thomas Brunschwiler, Ingmar Meijer, Stephan Paredes, and Werner Escher, Javier Goicochea, Patrick Ruch, and Brian SmithAquasar FOAK: Bruno Battaglia, Peter Buhler, Marcel Bürge, Ralph Heller, Daniel Jehle, Markus Kirschner, Urs Kloter, Thorsten Kramp, Marcus Oestreicher, Walter Riess, Daniel Ronzani, Sandra Rufener, Hansruedi Steinauer, Erich Rütsche, Jörg Schanze, Gerhard Schoor, Jürgen Stumpp, HeinerTschopp, Kurt Wasser, and Martin WitzigIBM Boeblingen (Germany): Martin Bachmaier, Gottfried Goldrian, Michael Malms, Jürgen Marschall, Oliver Rettig, Manfred Ries, and Walter WeberCCEM and CMOSAIC Partners: Dimos Poulikakos, Peter Kasten, Fabio Alfieri, Ming Hu, Igor Zinovik, Manish Tiwari, Severin Zimmermann, Petros Komoutsakos, and Wendelin Stark (ETH), and John Richard Thome, Jonathan Olivier, Navid Borhani, and Yassir Madhour (EPFL) Microfabrication: Rene Beyeler, Daniele Caimi, Ute Drechsler, Urs Kloter, Richard Stutz, Kurt Wasser, and Martin WitzigIBM Research Yorktown (USA): Paul Andry, Evan Colgan, Claudius Feger, Winfried Haensch, HendrikHamann, Theodore vanKessel, Ken Marston, Yves Martin, John Maegerlein, and Thomas TheisIBM Server and Technology Group in East Fishkill and Poughkeepsie (USA): Pepe Bezama, Kamal Sikka, Michael Ellsworth, Roger Schmidt, and Madhu IyengarIBM other locations: Dave Frank, Vinod Kamath, and Hannes EngelstaedterFinancial Support: IBM Zurich Research Laboratory, IBM Research FOAK program (Aquasar Project), Swiss Government KTI Projects, Swiss Government CCEM Project, Swiss Government NanoTera Project, EU FP7 project NanoPack