grid market - scut market dr simon c.w. see director grid and high performance computing technology...
TRANSCRIPT
Grid MarketDr Simon C.W. SeeDirectorGrid and High Performance Computing TechnologySun Microsystems Inc.
Sun Proprietary/Confidential: Internal Use Only
Grid Economic or Grid Market
•Motivation>to make grid to be the mainstream computing model,
Sun Proprietary/Confidential: Internal Use Only
“Utility” & “Grid” Definitions
A computing infrastructure that someone else owns and manages. You access this infrastructure only whenyou need to, pay only what for you use, and have nolong-term commitment.
A specific way systems are linked in a horizontalfashion. Sun is offering a grid infrastructure (or gridfarm) for customers to access on a pay-as-you-gobasis (versus a high-end, mission-critical data center.)
UTILITY
GRID
Sun Proprietary/Confidential: Internal Use Only
Grid (Background)• Current Practice of Grid
> voluntary sharing of compute resources,> establishing small-scale grid dedicated to a
specific purpose> FREE !!!?????
• BUT > this is simply not sustainable for global level
generic grid > WHO is paying for it?
•Footnote position, 12 pts.
Sun Proprietary/Confidential: Internal Use Only
Grid Market• Approach
> an efficient supporting platform and mechanism to encourage> resource owners to issue their idle resources > customers can get the satisfying resources.
> Using markets in Grid computing to commercialize their grid resources
Sun Proprietary/Confidential: Internal Use Only
Definition• Traditional Market
> a social arrangement that allows buyers and sellers to discover information and carry out a voluntary exchange.
• Grid Market > a software platform with the business
mechanisms to support trading between grid users.
e-Service Market ConceptA Market Place and Exchange provides economic incentive to trade
“Resource Owners”
1. Pricing Model2. Catalog 3. Billing Support –
Trading Agents Accounting Agents
1. Utility Model – The Contract
2. Broker to fulfill the contract
“Resource Users”
Enablers to Trade Enablers to Trade
Exchange
Broker
Broker
Broker
UserUser
User
User
ServiceProvider
User
ServiceProvider
ServiceProvider
Broker
Resource Provider
ServiceProvider
E-Service Market Place
Sun Proprietary/Confidential: Internal Use Only
Market Advantage• Encourages more users to adopt Grid technology,
especially in enterprises.• Provides incentives for resource owners to provide their
idle resources, which is helpful for establishing large-scale, mature grid system.
• Enforces efficient utilization of grid resources> buyers who value a resource most highly will buy from sellers
most willing to sell. > Provide access for even small businesses to temporary grid
resources ( too expensive to acquire on their own, or just to meet their short term peak demand).
• Both buyer and seller can easily design their trading policies based on their current status so as to maximize resources’ utilization and their benefits.
Sun Proprietary/Confidential: Internal Use Only
Industry Grid or Utility• Sun Grid Compute Utility,• IBM’s On-demand Computing (ODC), • Platform Enterprise Grid Orchestrator (EGO),• HP Adaptive Enterprise
Sun Proprietary/Confidential: Internal Use Only
IBM’s On-demand computing (ODC (1) Autonomic computing.• (2) Fast response to external business-affecting
changes. • (3) Adaptive business processes to protect
revenues and contain costs. • (4) Complex interactions inside and outside of
organizational boundaries. • (5) Resilience against external threats such as
viruses, intrusions, and power outages
Sun Proprietary/Confidential: Internal Use Only
HP Adaptive Enterprise
• Agile, Adaptive• Technology
> Tycoon
Sun Proprietary/Confidential: Internal Use Only
Sun Grid: Worlds Only Grid Based Compute Utility
Technical Complexity Managed by Sun and Hidden from ViewProvides Simple Plug-and-Play Experience for End Users
Easy, Secure, Affordable, Powerful
• Standard Infrastructure• Reliable• Scalable• Predictable pricing• Metered pricing• Multi-source• Multi-tenancyNetwork.com
Sun Proprietary/Confidential: Internal Use Only
The Face of Utility ComputingEasy Access to Compute Resource to Anyone on the Network
Sun Proprietary/Confidential: Internal Use Only
Customer ChoiceBuy the System, Consume the Utility, or Both
Sun Grid Private Utility
Commercial Sun Grid
Public Sun Grid
Consume it:Sun's Utility PortfolioBuy it:
Sun's Hardware & Software
Common Technology and IPCommon Architecture and DesignCommon Building Blocks
Sun Proprietary/Confidential: Internal Use Only
The Evolution of Public Sun Grid Utility
Phase 3
InfrastructurePlatform
Services Platform
2004 2005 2006
Continuous Innovation
Continuous Innovation
2007
Portal AccessHPC and Batch Apps
Programmatic AccessMiddleware ServicesISV interfaces
Multi sourceService Provider Packaging
2008
Service Aggregation
Sun Proprietary/Confidential: Internal Use Only
What Can Customers Do Today?Expedite success running compute intensive batch jobs
Digital Movie RenderingQuantum Chromodynamics
Radiation Cancer TherapyOffload Peak Traffic Flow
Stellar Evolution
Econometrics
Dow-Jones Forecasting
Oil Well Exploration
Very Large Scale Integration (VLSI) Design
CAE Military Simulation ModelingThe Synthetic Environment
Digitalization & Network ModelingSimulation Training
Monte Carlo Analysis
Molecular Drug Testing
Thousands of subscribers!
> 1,000 Developers Registered!
Sun Proprietary/Confidential: Internal Use Only
GESA-WG (Closed)• an open extensible economic service
architecture (based around OGSA Grid Services) to enable the provision and consumption of Grid Services.
• a Grid Banking Service (GBS) that records these financial transactions.
• Chargeable Grid Service (CGS) that encapsulates an existing Grid Services with the functionality necessary for it to be purchased.
• Enable a variety of economic models with these services but not to explore the economic models themselves.
●
Sun Proprietary/Confidential: Internal Use Only
GRACE (Gridbus)• Grid Resource Broker (e.g., Nimrod/G) • Grid Resource and Market Information Server
(provided by Globus MDS and extended services for resource access price)
• Grid Open Trading Protocols and API
Sun Proprietary/Confidential: Internal Use Only
The Computational Markets ●funded under the DTI e-Science Core Technology programe
Sun Proprietary/Confidential: Internal Use Only
Compute Power Market (CPM)
• Repository of information on providers
• Agents for consumers and providers
• Mechanisms for updating the information
• Interaction with other markets
Sun Proprietary/Confidential: Internal Use Only
Businss Grid (Japan)● Increasing the efficiency of in-house IT systems● Managing expected load increases● Enhancing the availability of the e-Japan system● Features of the Business Grid middleware are:● All information relating to a business application can be described
and retained in a defined format.● The IT resources used by business applications are virtualized as
hosting environments.● business applications can be deployed automatically on distributed
IT resources● a business application operating on an IT resource can be moved
to another resource or more resources can be added to a business application.
Sun Proprietary/Confidential: Internal Use Only
Business and Pricing Models
• Commodity Market Model• Posted Price Model• Bargaining Model• Tendering Model• Auction model
Sun Proprietary/Confidential: Internal Use Only
• Time Mode> Instant> Subscription> Agreement
• Trading Object thru Proxy> Commission Proxy> Confirmation Proxy> Non-Porxy
Sun Proprietary/Confidential: Internal Use Only
Dynamic Pricing• the terms on which the trading objects ( products or
services) are exchanged. • a reflection of the current state of the market• Price Dispersion
> Spatial and/or temporal• Price Discrimination
> 1st Degree – Perfect Differentiation> 2nd Degree – Nonlinear Pricing> 3rd Degree - the producer sells products to different
people for different prices, but every unit of product sold to a given person sells for the same price.
Sun Proprietary/Confidential: Internal Use Only
Model of Dynamic pricing• Is an optimization problem• Type of Model
> Inventory-based Model> Data-driven Model
> e.g Customer-based Model> Auction-based Model> Game Theory Model> Machine-based Model
Sun Proprietary/Confidential: Internal Use Only
Research Work• Customer Behavior Mode
> SunLab• Re-enforce Learning• Trading Grid Services
> University of Karlsruhe• Grid Market Framework
> NTU, APSTC• SORMA
> 8 Univ + Sun(APSTC) + 3 Commericial companies• Simulation Technology
> HUST + APSTC
Grid Market Framework
Grid Exchange, Grid Market, etc...
Grid Middleware (Globus, UNICORE…)
Grid Service / Grid Resources / Grid Applications
Data Grid Computational Grid Visualisation Grid …...
Business Development Toolkit
Workflow Engine Data Engine Visual Engine
Pricing Formation Charging Tools
Registry management
Identity management
Negotiation management
License mangement
Business Management Toolkit
Billing Accounting Banking
Trading management
Contract management
Service Settlement
Secure Trust Model Reporting Metering
Business Support Toolkit
Loyalty management
Monitoring Toolkit
Service Routing Portal Tool
…...
Customer layer
Grid Market Layer
Product Layer
Grid Fabric
Customer Toolkit
Decision Support Search Engine Service Broker ...
Sun Proprietary/Confidential: Internal Use Only
Grid Service Market Framework Prototype v1.0 Portal
Portal
Contract Mgmt Publish Payment Search
Searcher Agent
Relational Database
Trader Agent
Authentication Sign In/Out
XML Database
Authentication Sign In/Out
Authentication Sign In/Out
Banker Agent
Sun Proprietary/Confidential: Internal Use Only
The Problem...
“Only 5% of the applications are today suitable for unfettered
deployment to any utility computing environment...”
How do we accelerate the development of a library of
suitable applications?
Sun Proprietary/Confidential: Internal Use Only
Categorizing WorkloadsData Flow Defines System Requirements
Application DevelopmentCompute GridWorkgroup
Electronic Design SimulationStructural AnalysisEAI Servers
J2EE Application ServersDirectory
Security
Streaming Media
Web Serving
Proxy Caching
Thermodynamics
Seismic Analysis,Reservoir Modeling
File ServerOLTP Database
Nuclear Simulation/Weapons Modeling
Data WarehousingMeteorology/Climate Simulation
Financial Risk/Portfolio AnalysisMonte Carlo Simulation
Genomics, Cheminformatics
Batch
WebData
HPCApplication
Data Analysis
Compute
NetworkIntensive
StorageIntensive
Highly Threaded
Single Threaded
SAP R3
CommercialTechnical
Sun Proprietary/Confidential: Internal Use Only
One Size Does Not Fit AllDatabase, file systemSAN connectedBlock storageData servicesThread level parallelismVertical and horizontal scaleBig I/O bandwidthBig memoryHighly statefulHighly reliable nodesClustering/RACData replicationGeo distributionResources to workloadPartitioningDynamic resource mgt.Cautious change mgt.In depth monitoringActive management
Web serverData from the networkSmall local storage Thread level parallelismHorizontal scaleBig network bandwidthMedium memoryStatelessHorizontal replicationDurable nodes Workload to resourcesLoad balancerPacket allocationSystem image and app.ProvisioningHealth monitoring
MPI libraries, Queuing, Job CntlData from the networkSmall local storage Instruction level parallelismFP and integer intensiveEnough memoryHorizontal scalingStatelessHorizontal replicationDurable nodes Workload to resourcesGrid softwareJob allocationSystem image and app.ProvisioningHealth monitoring
Key middlewareData flow, dataData persistence
ApplicationThroughput ServiceAvailability
VirtualizationUtilization
ProvisioningManagement
Data Web ComputeSystem Attribute
Sun Proprietary/Confidential: Internal Use Only
Application Utility Defined?• A platform that enables Sun and ISV partners to
offer application subscription/utility services from the grid.
• The Sun Grid platform provides dynamic provisioning, observability and lifecycle management of services> Provides a mechanism for federated, policy-based
provisioning across the grid> Captures data used for
> Measurement of resource utilization> Service quality management
> Provides a mechanism for development and deployment of systemically-distributed/peer-oriented systems
> Enabled through Sun Connection telemetry
Sun Proprietary/Confidential: Internal Use Only
The New Application Paradigm?
Rearden Commerce has created a fabric for the hosting of web services that enables various services to be integrated in order to complete a unit of work. They have organized these services by vertical market, and provide a common management layer to ensure security and availability. This is the paradigm that we want to ultimately both provide for, and in the case of the ecosystem, fully enable.
Sun Proprietary/Confidential: Internal Use Only
Application Lifecycle Challenges• Component
> Tools> Frameworks> Patterns> Partners
• Test Harnesses• Submission• Validation• Placed in Library• Change Management
• System> Tools> Frameworks> Patterns> Partners
• Integration• Verification• Change Control• Submission• Provisioning• Execution
Sun Proprietary/Confidential: Internal Use Only
Phase 1 Hosted Development Environment:Technical Approach
Phase 1 has minimal impact on Sun Grid infrastructure engineering as majority of development done against Servlet Facade.
Sun Proprietary/Confidential: Internal Use Only
Conclusion• Grid Market has attracted a lot of attention both
academically and the industry• Still there are still of LOT of work to do. Many
issues and questions are not answered yet.• We need not just computer scientists but a whole of
others such as economists, mathematician, domain expert.
Sun Proprietary/Confidential: Internal Use Only
Invitation of participate• http://apstc.sun.com.sg• http://java.net• Pls send email to me if you are interesting to
work/collaborate with us> [email protected]