andrea fornaia consortium garr - infn › event › 603 › session › 13 › ... ·...

CLEVER

Andrea Fornaia

Consortium GARR

A light middleware for Private/Hybrid Cloud

Grid and Cloud workshop. Peking University, Beijing (China).

Monday, 23 April 2012

Cloud: a definition

“Cloud Computing is a large-scale distributed computing paradigm that is driven by economies of scale, in which a pool virtualized resources are delivered on demand to external customers over the Internet.”

[I. Foster et al. (2008)]

Grid and Cloud workshop. Peking University, Beijing (China). Monday, 23 April 2012

Three different cloud service levels

Virtualization

Hardware


A Cloud classification: Public, Private and Hybrid Clouds

» Public Clouds » Management of Virtual Machine instances within a proprietary infrastructure.

» Many different customers can run and control their own applications.

» Access from a remote interface using a specific protocol.

» Private Clouds » Infrastructure owned by a single organization offering its internal computing

resources to local users: do not “sell“ computing capacity.

» Open Source tools employment, dedicated operating environment offered to local users with high trust level

» Hybrid Clouds » A private cloud which adds to the local infrastructure more computing

capacity with resources coming from an external public clouds.

» External resources access allowed over the Internet, using remote interfaces.


Private/Hybrid Cloud middlewares: a reference stack » Features of Private/Hybrid cloud middlewares:

External Cloud Interfaces

Security Federation Contextualization

Autoscaling

Dynamic Resource Scheduling

VE Deployment

Data Management Networking

Configuration

Disk Image Management

Resource Monitoring

High-level Management

Virtual Infrastructure Management

» Middleware for Virtual Infrastructure Management: essentially dynamic orchestrator of Virtual Environments (VEs).

» Middleware for High-level Management: transforms existing infrastructures into an IaaS clouds with cloud-like interfaces; adds Security, Contextualization, Federation and other ”high-level” mechanisms.


Private/Hybrid Cloud middlewares: existing solutions » Virtual Infrastructure Management Middlewares: OpenQRM and

OpenNebula ˃ Deploy and manage VEs: individually or in groups needing parallel

scheduling on local resources or external public clouds.

˃ Automate VE setup regardless of the underlying virtualization layer.

˃ Lack mechanisms for building hybrid IaaS clouds: public cloud-like interfaces, the ability to deploy VMs on external clouds and other High-level functionalities.

» High-level Middlewares: Globus Nimbus and Eucalyptus ˃ Transform existing infrastructure into an IaaS cloud with cloud-like

interfaces.

˃ Compatible with the Amazon EC2 or Web Services Resource Framework (WSRF) interfaces and offers self-configuring virtual cluster support.

˃ Include Cloud-like interfaces and higher-level functionalities for security, contextualization.

˃ Limited VI management capabilities: lack the features of middlewares specialized in VI management.


A new cloud computing middleware: CLEVER » Acts as a middleware for the management of Private and Hybrid

cloud computing infrastructures.

» Specifically integrates VI Management layer functionalities.

» Provides simple and easily accessible interfaces:

˃ Integration of security, contextualization and other high-level functionalities made available from higher level software components;

˃ Interconnection of different heterogeneous cloud computing infrastructures.


Main features of CLEVER

» All inside a JAR, no installation required » Easy to deploy, easy to remove » Light in requirements » Fully pluggable and customizable » High scalability, easy to add new nodes to you cloud » Firewall pass through » Fault tolerant aware » Auto configuration: you only customize you middleware in

order to run in any node of you infrastructure. The middleware will auto generate the configuration required

» Manage distributed and heterogeneous resources, even it they are in separated networks

» Federation of different CLEVER cloud achieved


General Architecture on the reference scenario

» N computing nodes containing one host level Management module: Host Manager.

» One node includes a cluster level Management module: Cluster Manager.

» External components: XMPP Server and Distributed Database.

» Middleware entities “talks” in an XMPP chat room exploiting the presence feature.


CLEVER Architecture

» Host Manager (HM)

˃ Communicates with the hosts’ OS, hypervisor and distributed file-system on which the VE disk-images are stored.

˃ Performs both physical resources and VEs monitoring.

˃ Runs VEs on the physical hosts even performing their migration.

» Cluster Manager (CM)

˃ Coordinates the HMs and performs operations on the Distributed Database.

˃ Acts as an interface between the clients and the HM.

˃ Performs the user VE disk-images management and the monitoring of the overall cluster state.

˃ At least one CM has to be deployed on each cluster: many of them should exist to enable fault-tolerance.

˃ A master CM will be in active state while the other ones will remain in a monitoring state: automatic active CM re-election.


XMPP for the communication layer

» Administration XMPP Room ˃ Admin consoles join this room in order to interact whit the unique CM active in the CLEVER

private cloud.

˃ CM will forward the Admin requests to the properly HMs within the External Communication Room.

» External Communication XMPP Room ˃ Each HM joins this room in order to receive commands from the CM.

˃ In case of failure of the CM, the other HMs will start an algorithm in order to create a new CM


XMPP & Distributed Database

» XMPP (ejabberd)

˃ Offers a decentralized communication channel: more XMPP server could exists.

˃ Using more servers avoids a central point of failure.

˃ Both CMs and HMs “talks” in a chat room exploiting the presence feature.

˃ Information about the cluster state (connected hosts) offered directly by the protocol.

˃ It is easy to add new nodes in the infrastructure, the new node only needs to connect to the chat room to notify his presence: scalability.

» Distributed Database (sedna)

˃ Database containing the overall set of information related to the middleware: the current state of the VEs, data related to the XMPP connection.

˃ Developed according to a well structured approach, for enabling fault tolerance features.

˃ Used by both the Active/Idle CMs and XMPP server(s).


Host Manager Components: Host Coordinator

» Each HM component mapped an a different OS process: high modularity and fault tolerance.

» The core of the Host Manager: it coordinates all the HM internal components using a specific Internal Communication protocol (D-BUS or JMS).

» Through the CM interface communicates with the CMs exchanging XMPP chat messages on the specific room (VEs allocation, Monitoring State, etc.).

All inside a JAR


Host Manager: Monitor and Low-level components

» Monitor: Resource usage monitoring for each host. Information organized and made available for the HM coordinator.

» Hypervisor Interface: middleware back-end to the host hypervisor. Different virtualization technologies could be employed using different plug-ins structure has to be developed.

» Image Manager: supply to the Hypervisor Interfaces the needed VE disk-image corresponding to a specific VE. Different plug-ins associated to different data access/transfer method.

» Network Manager: Gathers information about the host network state. Manages host network (OS level) according to the guidelines provided by the HM Coordinator: dynamic creation of network bridges, routing and firewalling rules.

All inside a JAR


Cluster Manager Components: Cluster Coordinator

» Database Manager: interacts with the database used to store information needed to the cluster handling. Database Manager must maintain the data strictly related to the cluster state.

» Performance Estimator: Analysis of the set of data collected from the Coordinator, in order to compute and provide a probable trend estimation of the collected measures.

» Image Manager: manages registration and upload within the Cluster Storage System of the VEs disk-images. The Storage Manager is used to perform the registration process of such files and manage the internal cluster distributed file system.

All inside a JAR


Agent based: fully pluggable and fault tolerant

» Separated process for Each Agent/Plugin » Re-spawn of the Agent/Plugin in case of failure » Agent/Plugin loaded and removed at runtime (eaven new features) » CLEVER can be customized in order to achieve not only cloud task,

but even more.

JVM JVM JVM JVM

IPC (D-BUS)

Cluster Coordinator

Image Loader

Hypervisor Interface

Image Loader


Federation process using XMPP


Future Works: CLEVER on gLite Grid middleware


Thank you for attention


andrea fornaia consortium garr - infn › event › 603 › session › 13 › ... ·...

Documents