One Click Cloud Orchestrator (OCCO): bringing Complex Complete

Infrastructures Effortlessly to IaaS Clouds

Peter KACSUK, Adam VISEGRADI, Jozsef KOVACS andMárk GERGELY

MTA SZTAKI

Common national pools of resources

Developing an Open Science Backbone

Developing an Open Science Commons

Governan

ce

Operatio

ns

Securit

y

Shared capabilities based on open standards

Research Infrastructures and long tail of science

Core capabilities• Open Science

Cloud (e.g., VM management, Data storage/access/discovery)

• PID• Service registry

and marketplace

• Research platform built on top of shared capabilities plus community owned resources

• Data products, tools, scientific gateways, virtual labs

Multi-level governance with community participation• Local• National• European

Federated operations and support• Service desk• Monitoring and

accounting• Capacity management• Service level

management

• Network of CSIRT• Federated IdPs,

Auth and Authz• Management of

different levels of assurance

From Member States• Capacity dedicated to

large RIs• Free pools for long tail

researchers• Both pubicly funded

and commercial providers (all supporting open standards and no lock-in)

Credit to Sergio Andreozzi

BOINCproject server

BOINCclient

The whole infrastructure should be created by 1 click(Number of BOINC

clients is a parameter)

BOINCclient

• Build an institutional desktop grid for universities

• Required BOINC expertise

• Took several weeks to establish

• Goal: setup the desktop grid in ~10 minutes in the cloud

• Easy creation of institutional desktop grids

• Used in IDGF-SP project

Motivations for OCCO (Example 1)

Univ. cloud (or EGI cloud)

SZTAKI Desktop Grid server

Gateway(WS-PGRADE)

To run PS applications on BOINC

3G Bridge BOINCproject

DC-API plugin

Metajob plugin

BOINCclient

Virtualisationby

GBACGBAC

BOINCclient

Virtualisationby

GBAC

Univ. cloud or EGI cloud

Build the BOINC infrastructure with a gateway to submit parameter sweep applications into BOINC

Motivations for OCCO (Example 2)

SZTAKI Desktop Grid server

Autodock Gateway 3G Bridge BOINC

project

DC-API plugin

Metajob plugin

BOINCclient

Virtualisationby

GBACGBAC

BOINCclient

Virtualisationby

GBAC

Univ. cloud or EGI cloud

Setting up an Autodock SaaS (gateway with BOINC infrastructure):

Motivations for OCCO (Example 3)

Biologist, chemist can use it without any learning curve

Using Autodock SaaS service

E-mail notification of the user

Admin page showing the setup of the service

E-mail notification of the user

The gUSE portal can be reached at

http://dgdemo5.lpds.sztaki.hu/

Motivations for OCCO

• The creation of this 1-click infrastructure took several weeks -> we want a solution that enables the construction in several days

• There are many other use cases where the goal is to set up a complete infrastructure on demand in the cloud

• Such an infrastructure typically consists of multiple virtual machine applications• These should be instantiated and managed in an automated way• We need an offline description how such an infrastructure should look like• Then the infrastructure can be instantiated automatically, or with a single click of a

user. Hence the name: One-Click Cloud Orchestrator• The infrastructure typically defined as a graph where nodes are infrastructure

services.

• Automation possibilities in a cloudo Node instantiation

• Through an API or a UIo Node configuration management

• Via Chef, Puppet, etc.o Our goal is to extend these to infrastructure

Problems to be solved

• Complex, multi virtual machine applications need special care that are not offered by current infrastructure cloudso Users frequently must fine tune their virtual infrastructures manually to meet

their applications needs (scaling, inter-vm dependencies, error resilience)o Users of multiple clouds are usually restricted to the use of a particular provider

• Users of complex, multi virtual machine applications currently need low level understanding of application subcomponents and clouds for:o Configuration and deployment of subcomponents in a cloud context (this is

partially offered by configuration management tools like Chef or Puppet)o Such VMs must be crafted for the subcomponents that are capable to interface

with other dynamically created VMs

• We are developing a tool for the EGI communities to provide easy-to-use management for multi VM applications

State of the Art

feature\System Juju OneFlow Cloud- Formation Heat SlipStream OpsWorks"OneClick" plans "OneClick" plans

"OneClick" “vision”31-Jul-14 31-Jul-15

OS Support Ubuntu Hypervisor dependantFixed list of Linux

distributionsWindows Server

FreeBSDHypervisor dependant Supports wide variety

of OS

Amazon Linux,Supports wide variety

of OSSupports wide variety

of OSSupports wide variety

of OS

Ubuntu 12.04 LTS

Supported cloud backends

AWS EC2, HP Cloud Services, Windows Azure, Openstack

OpenNebula AWS EC2 OpenStackOpenStack, AWS EC2, OCCI, Microsoft Azure,

etc.AWS EC2

General EC2, General EC2,

All widely accepted Cloud Interfaces

could be easily extended later OCCI,

could be easily extended later

Node management method Service Composer Image Based Service Composer Service Composer Image Based Service Composer

Service Composer, Service Composer, Chef, other SotA

composer(s)

Generic Service Composer, Integrated

Abstract image managementChef

Open source Yes Yes No Yes Partly No Yes (?) Yes (?) Yes (?) Hosted service Hosted No Hosted No Hosted Hosted No No No + SaaS

Infrastructure Auto healing No Yes Yes Yes No Yes Hopefully Yes Yes

Manual Infrastructure Scalability Yes Yes Yes Yes Yes Yes No Yes Yes

Automatic Infrastructure Scaling No Yes Yes Yes No Yes No Yes Yes

UI CLI, GUI Web frontend, CLI management console, CLI, API

CLI, API, Horizon dashboard Web UI management console,

CLI, SDK API, CLI API, CLI, HTML5 Web frontend

API, CLI, HTML5 Web frontend, SaaS

One Click UI No No No No No No Yes Yes Yes

Service composer support Juju — Chef, Puppet Chef, Puppet — Chef loose integration with

Chefloose integration with

Chef, other SotA composer(s)

Generic Service Composer, Integrated

Abstract image management 

Multi-cloud provisioning Yes No No No Yes No Yes Yes Yes

Maturity (subjectively between 1-5) 3 1 5 2 2 1 1 2 4

OCCO in general

• Instead of basic components like virtual machines, OCCO can provide complete infrastructures of services in an on-demand, self-service fashion

• Automatic scaling and error recovery will be included.• OCCO aims both providing

o a high-level service with UI o exposing the underlying architecture as a framework to be built upon

• OCCO orchestrates o resource provider backends (e.g. IaaS clouds) o and configuration manager backends (e.g. Chef) based on statically defined infrastructure descriptions (“infrastructure-as-code” paradigm).

• The OCCO framework is developed to be highly versatile and backend-agnostic.

One Click Cloud Orchestrator (OCCO)

• Analogous to IaaS clouds: users can simply request complex infrastructures via a simple UI/API.o Instead of choosing from virtual machine types, users choose from virtual

infrastructure typeso They still face the same pay as you go model, and they can still utilize the

flexibility of the IaaS’s just on larger scale.

OCCO

Node resolution

OCCO

16

View of the OCCO Administrator

• OCCO Administrators are expected to create node definitions and infrastructure descriptions (templates)o Based on existing configuration management descriptions (e.g. Chef recipes)o The templates are stored inside the OCCO provided template store in order to

allow reuse

17

View of the OCCO User

• Can browse and customize offered infrastructure description templates• Can receive notifications about the state of his/her infrastructure (to be

implemented)o Through email or an automated service (allowing immediate use after creation or

changes)o If a change is applied to the template then the new updated deployment descriptor can

be pushed to OCCO internals

Internal Architecture

18

• Automated Infrastructure maintenanceo infrastructure descriptor processing and

VM management initiator

• Infrastructure Processoro internal depiction of a virtual

infrastructure

• Cloud Handlero abstracts IaaS functionality (e.g., VM

creation) for federated and interoperable use of clouds

• Service Composer (vm reshaper)o ensures awaited functionalities for VMs

• Information Dispatchero de-couples the information producer and

consumer roles across the architecture

Definition of an infrastructure

Definition of an infrastructure

VM1 Cloud-init

Deployment example: Wordpress-mysql

MySQL

CHEF

OCCO

VM2Cloud-init

Wordpre

ss

EC2

Register MySql

Register Wordpress

Deploy Wordpress

DeployMySQL

Instantiate new VM with MySQL ipaddress

Instantiate new VM

Get MySQL ipaddress

Infrastructure description

Development environment of OCCO

• In alpha stage• Python 2.7• Git [https://gitlab.lpds.sztaki.hu/groups/cloud-orchestrator]• Jira [https://jira.lpds.sztaki.hu/browse/OCD] – agile sprint dev.• Documentation [http://c153-33.localcloud/util-doc/util.html]

o Sphinx, docstrings: automatic documentation generation• Testing: Python nosetests: automatic testing• Deployment: setuptools

o Development package repo: [http://c153-86.localcloud:8080/packages]o Works best with virtualenv

• Package dependencieso PyYAML, argparse, python-dateutil, pika (for RabbitMQ), Boto, etc.

• Will be open-source after first release in 2015 Q3

Documentation of OCCO

Conclusions

• Prototype version of OCCO is working• First version of OCCO is close to be released• Short term goal: to create OCCI plugin for EGI

FedCloud • New use cases are continuously developed• We are looking for partners to realize their use cases