planetary scale computing towards planetary scale...
TRANSCRIPT
June 2003- 1
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Towards Planetary Scale Computing
next generationinternet computing
Rich Friedrichdirector
internet systems and storage labhp laboratories
June 2003http://www.hpl.hp.com/research/internet/
June 2003- 2
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Outline
● The rise of the Internet Data Center● Why scale to the planet?● Planetary Scale Computing at HP Labs● From research to reality● A research platform
June 2003- 3
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
The rise of the internet data center
● Confluence of web based distributed applications, Linux based rack servers, and internet data centers has enabled applications unimaginable a decade ago.
● However, these new environments are problematic: ■ dedicating hardware to specific applications limits flexibility,■ varying application demands result in poor server utilization, ■ rising complexity escalates operational costs, and■ emerging applications will consume 10-100X more resources■ IPv6 world with 2128 nodes ranging from nanobots to 128 way
servers ■ increasing server density generates energy and cooling issues
! Consequently, a new conceptual model for large-scale computing is required that addresses flexibility, utilization and cost while providing performance, security and fault isolation.
June 2003- 4
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Why IT services on a planetary scale?
● Centralization is the bane of tomorrow■ Simple scaling arguments dictate that “machine rooms” will
be dispersed■ Ownership of academic, business and consumer content will
be dispersed, but not necessarily public■ A sea of interconnected resources now exists■ Rich media from the masses
● Geographic dispersion of virtual teams■ Few people only interact with colleagues in the next office■ Workload demand follows the sun■ Supply chains■ 50% of professionals in 2006 may telecommute■ Online entertainment
! Consequently, new conceptual model must take into account large scale distributed services
June 2003- 5
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
planetary scale computing
vision
● We envision a world where distributed services execute on a utility that dynamically and securely allocates globally connected resources on demand
● …a global commercial GRID
● …will do for resources what the Web did for documents
■ provide uniform, ubiquitous access to globally connected server and storage resources
■ provide infrastructure on demand
■ eliminate resource shortages
June 2003- 6
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
planetary scale computing
requirements
● Large-scale and federation● Self-adapting for varying workload demands and resource capacity
● Resilient in the face of failure or attack
● Trust and privacy● Dynamic and evolutionary events
● Self-describing, verifiable policy model
● Mobile clients and storage● O(105) elements per data center
● Economical to deploy and operate
● Supporting emerging 21st
century applications such as rich media, bioinformatics, massive personalization, sensor networks
June 2003- 7
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
“Programmable Data Center”
Distributed, Shared Resources:
SecureFlexibleDynamicTraded
thesis: next generation of computing: the data center is the computer
“Internet Data Center”
Dedicated Resources:SecureInflexibleStatic
2001
Supplychain
SalesHR
R&D Mfg
Supplychain
SalesHRR&DMfg
2007
June 2003- 8
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
internet systems
architecture research
the data center is the computer
● What high density, low power, high performance computing architectures most economically support pervasive computing?
● What are the simple building blocks of processing, communications, storage and power that support dynamic allocation of virtualized resources?
● Where should processing occur? Where should data reside?
● Can ethernet provide a single server/storage fabric?
● What is the science of large scale dynamical systems that estimates probabilistic behavior based on small scale experiments?
June 2003- 9
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
“Programmable Data Center”
Distributed, Shared Resources:
SecureFlexibleDynamicTraded
thesis: programmable data center requires a program: the data center OS
Ford HPP&GYahooebay
2007“Data Center OS”
Automatically allocate distributed, shared resources:
SecureFlexibleDynamicFederationSelf-adapting
“Data Center OS”
Automatically allocate distributed, shared resources:
SecureFlexibleDynamicFederationSelf-adapting
June 2003- 10
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
automated resource control
research
the data center OS
! All physical resources controllable; i.e. servers, routers, storage and energy.
● What decentralized mechanisms support distributed resource allocation?
● What automated reasoning systems can eliminate the complexity of controlling large scale systems?
● What sensors and actuators are necessary?
● What control techniques are applicable to reactive and predictive events?
● What time scales are appropriate for control?
● How is security enforced?● How are control measures
and decisions coordinated across federated systems?
June 2003- 11
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
basic idea: consumer/supplier
applications
virtualized resource pool
Demandintelligent brokering:matchservice demandwithresource capacity
Supply
intelligent provisioning:effective use of physical resources
June 2003- 12
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Early insights: Internet workload characterization and control
Applications
virtualized resource pool
Supply Demand
User Demand for a Fortune 100 WWW Server
120000
160000
200000
240000
280000
Req
uest
s pe
r Day
Web Request Size Distribution
0
20000
40000
60000
80000
100000
120000
2 4 8 16 32 64 128
256
512
1024
2048
4096
8192
1638
4
3276
8
6553
6
1310
72
2621
44
5242
88
1048
576
2097
152
4194
304
8388
608
1677
7216
Request Size (bytes)
Freq
uenc
y
Fortune 100 daily demand web pages ..
… file size evolution …
OL
TP
E-C
omm
Web
97
Web
2001
New media types:
voice, audio, video
Growth 10-15% per month3x in 1 year10x in 2 years
characterization
M. Arlitt, D. Krishnamurthy, J. Rolia, "Characterizing the Scalability of a Large Web-based Shopping System," ACM Transactions on Internet Technology, June 2001.
Server QoSadaptable control
● Offered Load = 300% (at t=13)● Desired utilization = 85%
Nina Bhatti and Rich Friedrich, "Web Server Support for Tiered Services" IEEE Network, 13(5):64-71, September 1999.
June 2003- 13
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
• For the six data centers we looked at:• More than 50% of servers have a utilization <= 10%; 85% are <= 25% • All the data centers have servers that are very busy• The identity of the servers that are busy varies with time
Data center server utilization
June 2003- 14
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
A reference architecture frameworkfor a data center OS
Non-stoptrusted utility
Real time data msmt & mining
IntelligentResourcecontrol
Programmable resource utility
Resource management system
Service Spec &deployment system
GRID Services(OGSI)
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling trust privacy
Business processes
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
automating operational processes based on business priorities
June 2003- 15
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Adaptive internet data center circa 2000
accesstier
webtier
applicationtier
databasetier
edge routers
routingswitchesauthentication, DNS,
intrusion detect, VPNweb cache
1st level firewall
2nd level firewall
load balancingswitches
web serversweb page storage
(NAS)
databaseSQL servers
storage areanetwork(SAN)
applicationserversfiles
(NAS)
switches
switches
switchedfabric
processingelements
storageelements
infrastructure on demand
internet
intranet
wire infrastructure once...
rewire programmatically,
dynamically re-provision resources
J. Rolia, S. Singhal, R. Friedrich, “Adaptive Internet Data Centers,” SSGRR 2000, European Computer and
eBusiness Conference, L’Aquila, Italy, July 2000
An architecture for tomorrow
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 16
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
From research to reality
● HP announced the Utility Data Center (UDC) Nov 2001
■ incorporates key HP Labs concepts and technologies
● Based on HP Labs research on adaptive internet data center
■ ability to direct resources to any application dynamically
■ self healing, policy driven.■ Heterogeneous environments:
Windows, Linux, HP-UX, Sun Solaris, Cisco, ProCurve, EMC
switchedfabric
processingelements
storageelements
infrastructure on demand
internet
intranet
… to create a dynamically configurable utility fabric that
can be programmed per service or customer, based on
SLAs and demand…
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 17
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
creating a service with the UDC
1. Architect new service:
LB
WEB WEB WEB
APP
FWSvc “A”
Svc “A”
2. Build a service template:
APP
FW 1U Linux
2U NT
HP-UX
applianceLB
WEB WEB WEB
• Install apps
3. Ignite the service
Free
Discover and apply free resources
• Specify connectivity• Auto-configure
network and storage• Auto-load OSes
Svc “A”
June 2003- 18
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
IT Service ConsumerBusiness Impact Analysis2
2Business Process Cockpit; M. Sayal, F.Casati, U.Dayal, M.C.Shan, VLDB 2002
IT Service ProviderResources
Linking service management and business processes
ServiceSLO/SLA
ServiceSLO/SLA
ServiceSLO/SLA
ServiceSLO/SLA
IT manager
1Towards regulating electronic communities with contract; M.Morciniec, M.Salle, B.Monahan, ICAIL 2001
CIO
Management by Contract1
LOB manager
June 2003- 19
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
SmartFrog: service description and deployment
• Flexible configuration description language
• precise, desired configuration of applications composed of sets of components running across a distributed system
• declarative, prototype-based• predicates, constraints
• Service deployment architecture for massive systems
• realize application description• monitor and manage the resulting
applications through their lifecycles• No single point of control
applicationapplicationapplicationapplicationdescriptiondescriptiondescriptiondescription
SmartFrogSmartFrogSmartFrogSmartFrog distributed distributed distributed distributed deployment systemdeployment systemdeployment systemdeployment system
SmartFrogSmartFrogSmartFrogSmartFrog notationnotationnotationnotation
realizes running, realizes running, realizes running, realizes running, distributed distributed distributed distributed
applicationsapplicationsapplicationsapplications
managed, managed, managed, managed, monitored through monitored through monitored through monitored through
lifecyclelifecyclelifecyclelifecycle
• which application which application which application which application components?components?components?components?
• running where?running where?running where?running where?• how is each component how is each component how is each component how is each component
configured?configured?configured?configured?• how are the component how are the component how are the component how are the component
lifecycles sequenced?lifecycles sequenced?lifecycles sequenced?lifecycles sequenced?• how are components how are components how are components how are components
related?related?related?related?
Patrick Goldsack, "SmartFrog: A framework for configuration", from the Workshop on Large-Scale System Configuration, Edinburgh, November 2001. (Online proceedings available at www.dcs.ed.ac.uk/home/paul/wshop)
Managing the sea of software versions and dependencies
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 20
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
CPU allocations
Case Study- information on cpuutilization of 48 servers in a data center- simulated traces of cpuusage based on profiles from raw data
4 8 12 16 20 240
100
200
300
400 Static (util=0.5)
Hour of Day 4 8 12 16 20 24
0
100
200
300
400 Guaranteed (util=0.56)
Hour of Day
4 8 12 16 20 240
100
200
300
400Prob Best Effort (util=0.87)
Hour of Day
p=0.999
4 8 12 16 20 240
100
200
300
400Prob Best Effort (util=0.95)
Hour of Day
p=0.99
Resource Access Management for Computing Utilities
• Enterprise applications: continuously available, demands vary with time and user request loads, high peak-to-mean ratio
• Adaptive infrastructure: programmable data centers that offer shared resources
• Our focus: increase asset utilization and enable QoS for resource access
• Capacity planning/admission control/resource allocation --- use demand profiles, exploit time based allocation and statistical multiplexing
• Class of Service (CoS) --- static, guaranteed interval based access, best effort, probabilistic best effort: offers resources on demand with a specific probability p
Publications- J. Rolia, X. Zhu, M. Arlitt and A. Andrzejak, “Statistical service assurance for applications in utility Grid environments,” MASCOTS 2002. - J. Rolia, X. Zhu and M. Arlitt, “Resource access management for a utility hosting enterprise applications,” to appear at IM 2003.
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 21
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
0
2000
4000
6000
8000
10000
12000
14000
0 100 200 300 400 500 600 700 800 900
Nu
mb
er
of
Se
ssio
ns/D
ay
Day Number
HPC
Sessions
dynamics of media sites
how do we deal with bursty loads?
1 1.2 1.4 1.6 1.8 2Risk
0.1
0.12
0.14
0.16
0.18
0.2
nr
ut
eR
Computation as economicsUsing dynamic portfolio approaches
how do we manage risk and return?
market components: computational analogs:
resources hardware and software
agents programs’ choices
preferences computational needs
Bernardo Huberman: journal of econ. dynamics and control, 22, 1169 (1998)
June 2003- 22
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Turning the UDC into a “power station” for the commercial GRID
● Use an offline tool to design application topologies
● Use the Globus toolkit to■ submit resource
requests to a UDC■ create a “farm” (a multi-
tier topology) request■ return the access
information for the farmto the user
● UDC provides secure, dynamically allocatableresources for the GRID
Sven Graupner, Jim Pruyne, Sharad Singhal, Making the Utility Data Center a Power Station on the Commercial Grid, GlobusWorld 2003.
June 2003- 23
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Scalable commodity open source platform
http://www.gelato.org
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
• Automated application optimization• Secure kernel platform• stack unwind and perfmon
June 2003- 24
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
High Performance Computing
Dual Processor PEAK Bytes/FLOPSmall is BAD
-
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1H03 2H03 1H04 2H04 1H05 2H05
IA-32IA-64
Godiva
June 2003- 25
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Appia: automated SAN fabric designThe problem: Given flow requirements between hosts and storage devices, find a minimum cost reliable network to support all flow requirements simultaneously.
hosts
storage devices
flow requirements
(MB/s)
links& ports
fabric nodes (hubs & switches)
& a path for each flow
Ward, J., O'Sullivan, M., Shahoumian, T., Wilkes, J. Appia: automatic storage area network fabric design. File and Storage Technologies (FAST) Conference, 2002.
The solution: a software tool that produces provably correct, reliable and cost-effective designs in minutes.
3 days$4m
10 mins$1.4m
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 26
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Federated Array of Bricks (FAB):building an affordable and scalable disk array
● Goals■ drop in replacement for
disk array■ “infinite” scale■ incrementally upgradeable■ self-managing: no people
● Benefits■ smooth, incremental
capacity+performance scaling
■ enterprise-class reliability, functionality, management
■ low entry price■ best-in class absolute
price
Clients (typically servers, blades, etc)
storage bricks
iSCSI, FC, SCSI, SAS, …(read and write)(direct connect or SAN)
FAB protocols
Back End Network – Ethernet+RDMA, … (our choice)
Research: quorum-basedreplication scheme, dynamicload balancing and online
reconfiguration
Ref: “FAB: enterprise storage systems on a shoestring”, HotOS 2003
June 2003- 27
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Hippodrome: automatic storage management
Hippodrome: running circles around storage administration Eric Anderson, Michael Hobbs, Kimberly Keeton, Susan Spence, Mustafa Uysal, and Alistair Veitch. Conference on File and Storage Technology (FAST'02) January 2002
AnalyzeAnalyzeAnalyzeAnalyzeworkloadworkloadworkloadworkloadAnalyzeAnalyzeAnalyzeAnalyze
workloadworkloadworkloadworkloadImplement Implement Implement Implement
designdesigndesigndesignImplement Implement Implement Implement
designdesigndesigndesign
Design newDesign newDesign newDesign newsystemsystemsystemsystem
Design newDesign newDesign newDesign newsystemsystemsystemsystem
Design system to meet workload requirements
Configure devices & migrate data
Learn workload performance characteristics
benefit: lower operational cost for storage
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 28
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Corporate Immune Systems
Time
ResponseIn
fect
ed M
achi
nes
Problems
Response
Prevention
Speed
Fast
Slow
ResilientInfrastructure
ResilientInfrastructure
Virus ThrottlingSlowing down the spread by throttleing new connections
Matthew M. Williamson "Throtting viruses: restricting propagation to defeat malicious mobile code", proceedings of ACSAC
conference 2002, Las Vegas, NV
June 2003- 29
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
• Microprocessor Power – 100 W (tenfold growth in ten years)
• Microprocessor Power Density - 200 W/cm2 (by 2003, today 60 W/cm2)
• High System Power Density – 300 W, thin 1U form factor
• High EIA Rack Power Density – 10 to 15 KW per EIA Rack foot print
• High Room Power Density - 2700 W/m2 (~300 W/ft2, today at 70 W/ft2)
Energy to Remove Heat
.100 KW 1000+ KW10 - 15 KW
500 KW1 KW0.005 KW
Heat Generated
FlowThermo-dynamics
Thermal Challenges from Chips to Data Centers
June 2003- 30
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
HeatExchanger
pump
• Precise spray commensurate with the heat load on the chip
Evaporative Spray CoolingConductive Interface in high power processing and communication devices (>200 W/cm2, total power of 75 W) will not work
Why? How?
Inkjet assisted spray cooling
June 2003- 31
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Dynamic thermal management in large scale data centers
Patel, C.D., Sharma, R.K, Bash, C.E., Beitelmal, A, Thermal Considerations in Cooling Large Scale High Compute Density Data Centers, ITherm 2002 – 8th Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems” May 2002, San Diego, California
• Power Density becoming critical -
• Microprocessor: 200 W/cm2
(by 2003, today 60 W/cm2)
• System – 300 W, thin 1U form factor 10 to 15 KW per EIA Rack foot print
• Room- 2700 W/m2 (~300 W/ft2)
• Affects reliability and cost
! Use 3D modeling to understand thermal characteristics of data centers
! Sensor networks and robotic in situmeasurements
! Exploit this for dynamic resource allocation and proper provisioning
Programmable resource utilityTrusted
resources
Resource management system
Service Spec &deployment system
UDC UDC-lite horizontal-scale cluster SMP blades energy cooling
Real timedata mining (ZLE)
Non-stop
trusted utility
IntelligentResource
control
GRIDServices(OGSA)
BAM Business applications& web services (SLAs)
Non-stoptrustedservices
Business processes
June 2003- 32
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Summary
● HP Labs believes systems research needs to address
■ A new conceptual model for large scale internet computing and storage built upon commodity components and resource virtualization
■ Utility computing architectures, mechanisms and policies
■ Intelligent, decentralized resource control to simplify operation and reduce lifecycle costs
■ Pervasive, unified security model to ensure privacy and mitigate denial of service attacks
June 2003- 33
Planetary Scale
Computing
© 2001-2003 Hewlett Packard Company
Twin UDCs in HP Labs● Built the first large Planetary
Computing infrastructure in Palo Alto (US) and Bristol (UK)
■ Learn what it takes to build a solution■ Move HPL IT services to the UDC
● The first Virtualized Data Center ■ From Server, storage, networks to energy management
● Platform for HP experimentation● Raise the level of research platform
collaboration■ HPL, Academia, Partners■ Interested?" [email protected]