cloudmesh feb-2016 2 (1)

PowerPoint Presentation

CloudmeshAn Introduction

Gregor von Laszewskilaszewski@gmail.com1

laszewski@gmail.com, http://cloudmesh.github.io/client/

ResourcesDocumentationhttp://cloudmesh.github.io/client/

Source RepositoryCloudmesh_base: https://github.com/cloudmesh/baseCloudmesh_client:https://github.com/cloudmesh/client

laszewski@gmail.com, http://cloudmesh.github.io/client/2

Independent StudyI accept a limited number of talented students for independent studies. Projects: extending cloudmesh or other interesting cloud related workEffort: The usual effort 3 credit hours = 12 hours weekly at minimumWhy? E.g. 36 hours per week /3 courses = 12 hoursWill not accept students that have more than 9 credit hours per week.Grading: Based on working projectJob: No. You must at least have done an independent study with A or A+ grade.

OutlineMotivationHybrid CloudsPrevious EffortsIntroductionCurrent FocusMany OSClient FocusedLayersAPI, Shell, PortalCloud UsageOpenstackAzureAWSComet

HPC UsageQueuesExperimentsCoordinationSimple workflowPlatformsLaunchersVirtual ClusterHadoop...StatusConclusionlaszewski@gmail.com, http://cloudmesh.github.io/client/4

MOTIVATION

Hybrid CloudCloud Computing has many different facets and offers the ability to leverage IaaS, PaaS and SaaS on resource and services offered by others.However, some application require due to privacy concerns, cost, or functionality the use private clouds or a mix hosted at different providers.This leads us to the term hybrid cloud.Definition:A hybrid cloudis acomputing environment which uses a mix of on-premises, privatecloudand third-party, publiccloudservices with orchestration between the two platforms. It can offer bare metal, traditional HPC as a service, or virtualized resources offered in a cloud.von Laszewski, Gregorlaszewski@gmail.com, http://cloudmesh.github.io/client/6

Federated CloudsCapacity Federation

Technology Integration

laszewski@gmail.com, http://cloudmesh.github.io/client/7AWSAzureRackspaceHP CloudHybridCloudFutureSystemsChameleonCloud.HybridCloudIaaSPaaSCloud, ClusterSaaSBareMetalaaS aaS

Technology Robustness

Technology Federation

laszewski@gmail.com, http://cloudmesh.github.io/client/8AWSAzureHPEucalyputusHybridCloud

AWSAzureOpenStackEucalyputusHybridCloud

Federated CloudsMotivationPriceAvailabilityFault ToleranceCapacityResource LimitationsFeatures within the cloudHybrid CloudsIndependence:Avoid vender lock in

RequirementsAccessibleEase of useIntegratedFlexibleSupport multiple user community typesEnduserAdministratorDevelopers

Gartners Analysis

http://www.zdnet.com/article/microsoft-and-google-rise-while-ibm-sinks-in-gartners-latest-magic-quadrant-for-cloud-providers/

IaaSAmazon Web Services (AWS)Bulk of market share in the IaaS market, 10 times more cloud IaaS compute capacity in use than the aggregate total of the other 14 providers Gartner studied.MicrosoftAzure Infrastructure Services, second place in market share, with more than twice as much cloud IaaS compute capacity as the other providers Gartner studied (excluding AWS).GoogleThe Google Cloud Platform combines Compute Engine (its IaaS offering), App Engine (an application PaaS or aPaaS offering) and complementary services like object storage and a Docker container service.CenturyLinkGlobal communications service provider CenturyLink acquired Web hoster Savvis in 2011 and pure-play cloud IaaS provider Tier 3 in 2013 and merged them to create its CenturyLink Technology Solutions business unit, which now offers CenturyLink Cloud (CLC).VMwareVMware has traditionally focused on virtualization technologies, but made its vCloud Air cloud IaaS service generally available in 2013.IBM (SoftLayer)IBM acquired SoftLayer in 2013, and by 2014 it migrated its existing SmartCloud Enterprise cloud IaaS customers to SoftLayer and shut SmartCloud down.RackspaceRackspace offers a multitenant, OpenStack-based public cloud and three flavors of hosted private cloud: vCloud Director-based and VMware-virtualized, Microsoft Cloud OS-based and Hyper-V virtualized or OpenStack-based and KVM-virtualized. It also offers an on-premises private cloud.

HP and Rack space have since stopped serving public clouds and focus on specialized customers.

http://www.cio.com/article/2947282/cloud-infrastructure/top-cloud-infrastructure-as-a-service-vendors.html

Academic cloud providersFutureSystems.orgComet Cloud (not OpenStack)CloudLabChameleonCloud.orgCybera.org (Canada)UBuffalo Cloud (Eucalyptus)RedCloud (Cornell)Many many more laszewski@gmail.com, http://cloudmesh.github.io/client/12

Need for managed cloud services?Hybrid cloud brokerage - managing services you use on clouds.Centralized visibility and control.Optimising cost/performance and staying compliant.Hybrid CloudsOn Premise - eg: OpenStackPublic Clouds - eg: AWS, Azure, SoftLayerCredential management & Access control across multiple clouds.Policy based cataloguing & orchestration.

CLOUDMESH

INTRODUCTIONCloudmesh is a project to easily manage virtual machines and bare metal provisioned operating systems in a multi cloud environment. With Cloudmesh, it is possible to use Azure, Amazon Web Services, and other clouds such as chameleon.Cloudmesh also provides an interface to manage thousands of jobs on High Performance Computing (HPC) clusters, such as XSEDEs comet supercomputer

How cloudmesh fills the gap?Matches cloud consumer needs with multiple providers services.Researchers can use one platform to manage their clouds.Orchestrates provisioning and allocation of cloud resources.Local copy of your cloud data is created, so new clouds with similar configurations can be created easily.Default attributes allow easy control of cloud artefacts.

Previous EffortFocus was on portal for FutureGrid that accesses hybrid clouds within FutureGrid Portal while also managing the users credentialsCons: Portals are time consuming to developManaging credentials for others is dangerousPros:Baremetal provisioning before others did itIt actually worked

Effort is known as cloudmesh

Current Effort: Cloudmesh ClientKnown under the code name cloudmesh clientFocus on user installable and hosted serviceFocus on command shell first, than portal.The portal is also hosted on the client machineEasy installation while eliminating Unix specific components such as LDAPTarget OS right from the start: OSX, Linux, WindowsProvide most useful featuresProvide access to SDSC comet

Cloudmesh Client and PortalSimplify accessQueueing SystemsContainersIaaS

ParallelismTaskComputeData

(GPU)XSEDEHPCCloudsCometJetstreamBridges

Cloudmesh Client and PortalData RepositoryAccessCommand Shell & LineAPIRESTPORTALComputeVMs, Containers, Batch ChoreographyGroups, Workflows, PaaS

$ cmcm> register remotecm> default cloud=kilocm> vm boot

Integrated Access InterfacesIntegrated State Management!E.g. you start a VM in the commandline, you can see it in the Web interfaceCommunication through a NoSQL database

InstallWe recommend python 2.7.10pip 8.0.2

Will install on LinuxOSXWindows

See:http://cloudmesh.github.io/client/ Do not just do what we present here on the slides

pip install cloudmesh_client

One line install with curl

In case we find resources for it (good student projects):rpm, dpg, ..dmgexe

Overview: ShellShell$ cmcm> register remotecm> default cloud=kilocm> vm boot

Yes that is it! Defaults for images and flavors can be set

Command Linecm register remotecm default cloud=kilocm vm boot

Integration of the cybera (Alberta, Canada) took 5 minutes

Basic UsageCommandDescriptioncm helphelpcm manmanual pagescm script.cmexecute cm commands in script

Cloud CommandsCloudCommandDescriptioncm image listlist imagescm flavor listlist flavorscm vm listlist vmscm vm bootboot vmcm vm boot cloud=kiloboot vm on cloud kilocm default cloud=kiloset default cloud to kilocm select imageselect interactively the default image (not implemented yet).cm select flavorselect interactively the default flavor (not implemented yet).cm select cloudselect interactively the default cloud (not implemented yet).

Shell CommandsCommandDescriptioncm color onsets the shell colorcm color offswitches off the colorcm refresh onautomatic refresh from the cloudscm refresh offdata is only read from the database. Useful for managing thousands of VMs or limit your access to the cloud.var a=xyxdeclares a variablevar username=cloudmesh.profile.usernamereads the variable from the cloudmesh.yaml filevar time=nowgets the time and store it in the variable time

HPC CommandsCommandDescriptioncm helpHelpcm hpc queue info about the queue cm hpc infoinformation about the queues on the HPC resourcecm hpc run uname -aruns the command unamecm hpc run listprints the ids of previously run jobscm hpc run listprints the ids of previously run jobscm hpc run list 11prints the information regarding the job with the id 11

PaaS ManagementVirttual Cluster$ cm cluster n 10 group=myCluster

Hadoop$cm hadoop n 10 group=myHadoop

Launchers: commands via GUI

29Specific LaunchList of Launchers

Homeworkhttp://cloudmesh.github.io/client/exercises.html

Install CloudmeshStart vms on one cloudStart groups of vms

QuestionsGregor von Laszewskilaszewski@gmail.comlaszewski@gmail.com, http://cloudmesh.github.io/client/31

cloudmesh feb-2016 2 (1)

Internet

monthly performance report february 2016 · 4.22 average...

feb. 24, 2016

accessing and managing multiple clouds (infrastructures)...

caltrux feb. 2016

exemptions granted jan2014 - feb 2016 exemptions feb...

home & community education jan./feb. 2016...feb. 2016 feb. 3...

feb 18- feb 19- feb 20 feb 21- feb a summer with what lack...

19 feb .to 25 feb.2016

feb 2016 crossface

kreate feb 2016

feb 2016 full

15 feb, 2016

elizabeth city state university reu site: arctic and...

gramarathnam feb 2016

4sight feb 2016

fierce feb 2016

selected lessons learned from futuregrid resulting in a...

alchemy feb 2016

btl pm informbtl pm information - feb 2016.docxation - feb...

laser feb 2016