managing cloud services with conpaas guillaume pierre...

48

Upload: others

Post on 12-Jul-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Heterogeneous Cloud Computing for ResearchManaging Cloud Services with ConPaaS

Guillaume Pierre

http://www.conpaas.eu/

Heterogeneous Cloud Computing for Research 1 / 34

Page 2: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Table of Contents

1 Introduction

2 ConPaaS usage

3 ConPaaS architecture

4 Conclusion

Heterogeneous Cloud Computing for Research Introduction 1 / 34

Page 3: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Typical Cloud Applications (according to AWS)

Application Hosting

Backup and Storage

Content Delivery

E-Commerce

High Performance Computing

Media Hosting

On-Demand Workforce

Search Engines

Web Hosting

Heterogeneous Cloud Computing for Research Introduction 2 / 34

Page 4: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Applications running at Amazon Web Services

Sample: 50 applications from the

AWS Customer App Catalog.

Heterogeneous Cloud Computing for Research Introduction 3 / 34

Page 5: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Many Cloud applications are alike

Many cloud applications are alikeI Web serversI Application serversI Database serversI High-performance frameworks (MapReduce, MPI, Work�ows)I . . . and every application has something di�erent

To be deployed on a wide variety of cloud resources

Cloud application developers often rebuild

the same types of frameworks again and again and again. . .

Heterogeneous Cloud Computing for Research Introduction 4 / 34

Page 6: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Deploying an application in the Cloud can be easy

1 Choose a cloud provider

2 Start one virtual machine using a ready-made image

3 Install your software in the virtual machine

4 Snapshot the VM

Heterogeneous Cloud Computing for Research Introduction 5 / 34

Page 7: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

What about a slightly more complex application?

Heterogeneous Cloud Computing for Research Introduction 6 / 34

Page 8: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Heterogeneous Cloud Computing for Research Introduction 7 / 34

Page 9: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Can the Cloud help support common types of applications?

Infrastructure-as-a-Service provides basic computing resourcesI Absolute �exibility: you can build anything you wantI But it can be very complex and time consuming

F DeploymentF Software upgradesF Fault-toleranceF Performance monitoringF Resource provisioningF Dynamic recon�guration orchestrationF etc.

Platform-as-a-Service provides high-level servicesI Each PaaS service targets a speci�c family of applicationsI Provide a simple deployment environment for applicationsI Provide high-level guarantees for applications using these services

Heterogeneous Cloud Computing for Research Introduction 8 / 34

Page 10: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Main PaaS environments

Google AppEngine

Microsoft Azure

Amazon Beanstalk

RightScale

OpenShift

ConPaaS

etc.

Di�erent PaaS environments address

di�erent types of applications

Heterogeneous Cloud Computing for Research Introduction 9 / 34

Page 11: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS takes care of your applications

Heterogeneous Cloud Computing for Research Introduction 10 / 34

Page 12: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS takes care of your applications

Heterogeneous Cloud Computing for Research Introduction 10 / 34

Page 13: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS in a nutshell

Broad range of functionalitiesI Application servers, databases, high-performance computing, other

Fully integratedI Applications can compose any set of services together

Easy to use but also very powerfulI Simple Web GUI + powerful command-line toolI Services are highly customizable

Cutting-edge SLA enforcement technologiesI Elasticity and resource provisioning techniques to guarantee

performance at the lowest possible cost

Runs over Amazon EC2, OpenNebula, OpenStack

and HARNESS, of course :-)

Open-source

http://www.conpaas.eu

Heterogeneous Cloud Computing for Research Introduction 11 / 34

Page 14: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Table of Contents

1 Introduction

2 ConPaaS usage

3 ConPaaS architecture

4 Conclusion

Heterogeneous Cloud Computing for Research ConPaaS usage 12 / 34

Page 15: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS Applications

A ConPaaS application is de�ned as a composition of multiple service

instances

For example: web hosting service + MySQL database + logging

service (to store access logs)

PHP service

(using 1 or more

machine instances)

Logging service

(using 1 or more

machine instances)

End user

SQL data service

(using 1 or more

machine instances)

Heterogeneous Cloud Computing for Research ConPaaS usage 13 / 34

Page 16: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS Services

Web hosting (PHP, Java)

Databases (MySQL, Scalarix)

File system (XtreemFS)

High-performance computations (MapReduce, TaskFarming)

Functional testing (Selenium)

Heterogeneous Cloud Computing for Research ConPaaS usage 14 / 34

Page 17: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS Operation

Users access ConPaaS thanks to a Web interfaceI LoginI Start new services (i.e., start a standard VM image with the service

implementation)I Manage existing services (i.e., communicate with the service's manager

to issue commands)I Stop services (i.e., stop all service instances except the service

manager)I Terminate services (i.e., destroy a service completely)

An extended set of functionalities is available through a command-lineinterface

I All commands from the Web interface are available (except starting anew service)

I Additional commands may be implemented for expert usersI The command-line interface makes it easy to script service management

Heterogeneous Cloud Computing for Research ConPaaS usage 15 / 34

Page 18: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 19: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 20: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 21: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 22: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 23: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 24: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 25: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Front-End

Heterogeneous Cloud Computing for Research ConPaaS usage 16 / 34

Page 26: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Uploading new application versions

Heterogeneous Cloud Computing for Research ConPaaS usage 17 / 34

Page 27: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Uploading new application versions

Heterogeneous Cloud Computing for Research ConPaaS usage 17 / 34

Page 28: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Uploading new application versions

Heterogeneous Cloud Computing for Research ConPaaS usage 17 / 34

Page 29: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Scaling a service

Heterogeneous Cloud Computing for Research ConPaaS usage 18 / 34

Page 30: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Scaling a service

Heterogeneous Cloud Computing for Research ConPaaS usage 18 / 34

Page 31: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS command-line interface

Heterogeneous Cloud Computing for Research ConPaaS usage 19 / 34

Page 32: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS command-line interface

Heterogeneous Cloud Computing for Research ConPaaS usage 19 / 34

Page 33: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS command-line interface

Heterogeneous Cloud Computing for Research ConPaaS usage 19 / 34

Page 34: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Table of Contents

1 Introduction

2 ConPaaS usage

3 ConPaaS architecture

4 Conclusion

Heterogeneous Cloud Computing for Research ConPaaS architecture 20 / 34

Page 35: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS Services

A ConPaaS service is composed of 1 Manager and n AgentsI One Python class running in the Manager VMI One Python class running in the Agent VM

Heterogeneous Cloud Computing for Research ConPaaS architecture 21 / 34

Page 36: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

The ConPaaS Director

The single entry point to the system

Keeps tracks of users, credentials, services

Handles the life-cycle of ConPaaS applications

Heterogeneous Cloud Computing for Research ConPaaS architecture 22 / 34

Page 37: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

ConPaaS clients

Clients make it easy to send commands to the DirectorI Command-line interfaceI Web-based GUI

Heterogeneous Cloud Computing for Research ConPaaS architecture 23 / 34

Page 38: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Lifecycle of a ConPaaS service

No service

Service

created(1 manager VM)

Service

started(1 manager VM

+ N agent VMs)

Scale

up/down

Service−specific

operations

Terminate

Create

Stop

Start

Heterogeneous Cloud Computing for Research ConPaaS architecture 24 / 34

Page 39: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Example: the Web hosting service

The service exists in two versions: PHP and Java

Initially the service has 2 VMsI 1 VM running the managerI 1 VM running a load balancer, a web server and a PHP backend

When adding VMs each VM becomes specialized (load balancer VMs,

web server VMs, PHP backend VMs)

Proxy

Web

PHP

ManagerProxy

Web PHP

Manager

+1 Web

+1 Web

+1 P

HP

+1 P

HP

Manager Proxy

Web

PHP

Manager Proxy

PHP

Web

Heterogeneous Cloud Computing for Research ConPaaS architecture 25 / 34

Page 40: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Session handling in the PHP service

PHP has built-in support for sessionsI Normally stored in main memory of the PHP server

We must share session state between multiple PHP backends(otherwise users would logout at each request)

I We use the Scalaris key-value store for thatI One Scalaris server inside the manager VM

Making use of the Scalaris session storage is totally transparent to the

applications

Heterogeneous Cloud Computing for Research ConPaaS architecture 26 / 34

Page 41: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

What we want to avoid

Heterogeneous Cloud Computing for Research ConPaaS architecture 27 / 34

Page 42: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Service recon�guration

When the user scales the service up:1 The front-end sends a request to the service manager to scale up2 The service manager creates a new VM with proper contextualization

information, then starts polling3 The agent VM boots, then starts its manager process4 When the manager establishes a connection with the agent, it requests

it to start one or more roles5 The manager uploads code/data as necessary6 The manager recon�gures other VMs as necessary

When scaling down:I Same story in opposite order

Heterogeneous Cloud Computing for Research ConPaaS architecture 28 / 34

Page 43: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Building new ConPaaS services

Building new ConPaaS services from scratch was::::::HARD

I Build a proper VM image with contextualizationI Develop new manager and agent deamonsI Implement a standardized protocol between the front-end and the

agentsI All communication goes over SSL with custom security checks

Solution: the service coreI All ConPaaS services use a single VM imageI All ConPaaS services use the same manager and agent deamonsI The service core implements shared functionality between all services

F Start/stop/contextualize virtual machinesF Secure communication primitivesF Performance monitoring mechanisms

I Each service can specialize the service coreF Implement the service-speci�c parts

Heterogeneous Cloud Computing for Research ConPaaS architecture 29 / 34

Page 44: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Structure of a service implementation

Building a new ConPaaS service from the service core is:::now

::::::EASY:

I (optional) Provide shell scripts to be executed when VMs start and stopI Write a manager and an agent class in PythonI Extend one Python �le to register the new serviceI Extend the front-end with one service-speci�c page in PHP

Adding functionality to all services is quite easy as wellI Secure internal communicationI Support for new cloudsI Performance monitoring

Heterogeneous Cloud Computing for Research ConPaaS architecture 30 / 34

Page 45: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Our latest baby: the Generic service

Package any application, provide 5 scripts:I init.sh: install the applicationI run.sh: start the applicationI interrupt.sh: stop the applicationI cleanup.sh: uninstall the applicationI notify.sh: handle the arrival/departure of another VM in this service

Heterogeneous Cloud Computing for Research ConPaaS architecture 31 / 34

Page 46: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Table of Contents

1 Introduction

2 ConPaaS usage

3 ConPaaS architecture

4 Conclusion

Heterogeneous Cloud Computing for Research Conclusion 32 / 34

Page 47: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Conclusion

ConPaaS is a platform-as-a-service environmentI Designed to facilitate elastic application hosting in the cloudI Designed to be easily extensible

ConPaaS addresses two major classes of applications:I Web applicationsI Scienti�c applicationsI Combinations of both

We will use the public ConPaaS installation for practical assignments:

https://online.conpaas.eu

Heterogeneous Cloud Computing for Research Conclusion 33 / 34

Page 48: Managing Cloud Services with ConPaaS Guillaume Pierre …harnesscloud.github.io/2015-07-15-feltham/conpaas/slides.pdf · Microsoft Azure Amazon Beanstalk RightScale OpenShift ConPaaS

Credits

Ismail El Helw(services,

managers, agents)

Adriana Szekeres(security)

Emanuele Rocca(director,

networking)

Francesco

Allertsen(manifests)

Claudiu

Gheorghe(web frontend)

Yann Radenac(multi-cloud)

Genc Tato(nutshell, appmanagers)

Teodor Criv µ(nutshell, disk

volumes)

Heterogeneous Cloud Computing for Research Conclusion 34 / 34