Multi-Cloud Management &

Application Customized Scaling

with CloudFoundryEdison Ting

Solutions Architect / Field EngineeringOctober 23, 2014eting@pivotal.io

Motivation:

Change Capture:• On Average = 1,500 – 5,500

ops/sec (inserts/updates/deletes) captured

• Highly variable and bursty• Real time data flow problem

A few months ago, we built a distributed replication solution for a customer, using distributed processes that read from queues populated with change capture data

Each Replicator would read from a set of queues, then write data into a in-memory data grid, while periodically retiring batches of changes into a distributed database

Replicator:On Average can handle 1,000+ ops/sec

On average each Replicator can process 1,000+ operations per second and so to keep up with the change capture process, we would add as many Replicator processes as needed to support the highest rate observed

Adding Replicators allows the overall task to scale linearly

Distributed Replication

We created a performance dashboard for monitoring of the overall throughput of the system, as replicators are manually added/removed to process the changing demands of the system.

Distributed Replication Monitoring

We could not dynamically scale our replication processes to react fast enough to the demands of the incoming transactions

We couldn’t take advantage of resources that might be available from other clusters to scale out the replication processes

We needed an automated way to monitor whether the replication processes had reach peak loads so that we can begin to decide how to add or remove replication processes that are not needed

We needed a way to allocate resources properly to processes so as to not to take away resources that can be used by other processes replicating other parts of the system

Challenges:

Goals

Define multi-cloud management and application customized scaling and how it begins to solve some of the challenges

What is CloudF0undry: a PaaS that provides a polyglot and cloud agnostic environment with application lifecycle recovery and resource management capabilities

Demo of Multi-Cloud Management and Custom App Scaling Prototype

Discuss the architecture of the MCC and Custom App Scaling solution using CloudFoundry capabilities and APIs.

Pivotal - Our Mission

mccController can analyze resources globally, across multiple clusters managed by a PaaS, to determine where applications can run or where resource can be adjusted for provide optimal use of the system

Application #1

PaaS (CloudFoundry)

Multi-Cloud Controller Architecture

mccMonitor

mccController

Application #2

Application #N

Application #1

PaaS (CloudFoundry)

mccMonitor

Application #2

Application #N

mccController tracks the resource utilization of each PaaS that it manages, and based on the data can provide recommendations on where to deploy new applications and scale resources

Application #1

PaaS (CloudFoundry)

Multi-Cloud Controller Architecture

mccMonitor

mccController

Application #2

Application #N

Application #1

PaaS (CloudFoundry)

mccMonitor

Application #2

Application #N

Applications periodically reports resource utilization to mccMonitor as well as changing workload requirements. mccMonitor will manage the information and report to mccController for workload analysis

Application #1

PaaS (CloudFoundry)

Application #2

Application #N

mccMonitor

mccController

Multi-Cloud Controller Architecture

mccController communicates with mccMonitors periodically while mccMonitors runs locally on each PaaS to allows application to communicate workload status as well as scaling requirements.

mccController then communicates with the PaaS (CloudFoundry) to manage resources of applications and scale the number of application instances.

Application #1

PaaS (CloudFoundry)

Application #2

Application #N

mccMonitormccController

Multi-Cloud Controller Architecture

mccController will communicate with multiple PaaS instances with different load characteristics, so that it can scale applications and add or drop app instances as necessary to achieve desired throughput and balance resource utilization between different PaaS instances.

Application #1

PaaS (CloudFoundry)

Multi-Cloud Controller Architecture

mccMonitor

mccController

Application #2

Application #N

Application #1

PaaS (CloudFoundry)

mccMonitor

Application #2

Application #N

Application #1Application

#1

Application #2

CloudFoundry: An Operating System for the Cloud

CloudFoundry and Bosh provides an IaaS Agnostic Infrastructure

+

Bosh manages other distributed processesbesides CloudFoundry, such as Hadoop

CloudFoundry (PaaS) Architecture

Containers

Containers

Containers

Containers

What does CloudFoundry do with an Application ?

Containers

Containers Containers Containers Containers

+

+

KernelContainers

CloudFoundry manages the applications placement for effective scaling , high

availability and recoveryContainers

CloudFoundry promotes loose coupling of applications and services

Containers

Containers

Environmentvariables

ProvisionServices

CloudFoundry provision services and make them available for use to applications. Applications access services using environment variables configured thru CloudFoundry

How CloudFoundry enables the Multi-Cloud Controller Architecture

mccMonitor

mccController

Environmentvariables

mccController can (1) deploy mccMonitor’s to each CloudFoundry instance, and have applications bind to the mccMonitor, enabling the application to communicate performance metrics, then (2) enable mccController to dynamically scale their resources

mccMonitor

Environmentvariables

1

2

1

2

Our Tools: CloudFoundry Client APIsHow can we use these to accomplish what

we need ?

Package: org.cloudfoundry.client.lib CloudFoundryClient CloudCredentials

Package: org.cloudfoundry.client.lib.domain CloudInfo CloudApplication CloudService ApplicationStats InstanceStats InstanceInfo InstanceState

Demo

Pushing (deploying) Application(s)

Containers

Containers

+

+

CloudFoundryClient client = new CloudFoundryClient(new CloudCredentials(user, pw), cfURL, org, space,..); client.login();List<CloudDomain> domains = client.getDomains();String appURL = “myApp” + domains(0).getName(); // myApp.cfapps.io

client.createApplication( “myApp”, new Staging(), memSize, appURLs, services );client.uploadApplication( “myApp”, file.getCanonicalPath() );client.startApplication( “myApp” );

1

2

3

Pushing (deploying) Application(s)

Containers

Containers

+

+

1

2

3

Create CloudFoundry client object

Assign application name and URL

Upload (push) and start application

Creating Service(s)

Containers

CloudFoundryClient client = new CloudFoundryClient( … );String appURL = “myApp” + client.getDomains(0).getName(); // myApp.cfapps.io

CloudService service = new CloudService(metaData, “myService”);Map<String, Object> credentials = new HashMap<String, Object>();credentials.put( “myServiceAPI”, appURL );client.createUserProvidedService( service, credentials );

1

2

3

myApp.cfapps.io

Creating Service(s)

Containers

1

2

3

myApp.cfapps.io

Assign application name and URL

Create Service object

Provide credentials information to allow applications to use the service

Binding Service(s) to Application(s)

Containers

CloudApplication yourApp = client.getApplication( “yourApp” );

client.bindService( yourApp, “myService” );

client.stopApplication( “yourApp” );client.startApplication( “yourApp” );

Environmentvariables

Containers

1

2

3

Binding Service(s) to Application(s)

Containers

Environmentvariables

Containers

1

2

3

Get application object related of the app to be boundBind application to service

Stop and Start application

How can Applications find ‘bound’ Services ?

Containers

String myEnv = System.getenv( “VCAP_SERVICES” );

JSONObject obj = JSONValue.parse( myEnv );JSONArray arrySrv = obj.get( “yourService” );for( JSONObject srvObj: arrySrv ) { JSONObject credentials = srvObj.get( “credentials” ); String serviceUrl = credentials.get( “yourServiceAPI” );

Environmentvariables

Containers

1

2

How can Applications find ‘bound’ Services ?

ContainersEnvironmentvariables

Containers

1

2

Get environment variable VCAP_SERVICES

Find the service’s entry and get credentials information to be used to communicate with the service

How do we gather application statistics ?

CloudFoundryClient client = new CloudFoundryClient(..);Map<> env = client.getApplication().getEnvAsMap();

int instCnt = client.getApplication().getInstances();ApplicationStats stats = client.getApplicationStats(“yourApp”);for( InstanceStats is : stats.getRecords()) { InstanceStats.Usage usage = is.getUsage(); long memuse = usage.getMem(); long cpuuse = usage.getCpu(); long diskuse = usage.getDisk(); long memQuota = is.getMemQuota(); long diskQuota = is.getDiskQuota(); …}

mccController

1

2

3

How do we gather application statistics ?

mccController

1

2

3

Create CloudFoundry Client Object

Get Application Stats Object

Iterate Applications Instances Stats object to get memory, disk, and instance count information

How do we scale application resources ?

CloudFoundryClient clnt = new CloudFoundryClient();

clnt.stopApplication(“yourApp”);

clnt.updateApplicationEnv(“yourApp”, Map<> env);

clnt.updateApplicationMemory(“yourApp”, newVal);clnt.updateApplicationDiskQuota(“yourApp”, nDisk);clnt.updateApplicationInstances(“yourApp”, nInstns);

clnt.startApplication(“yourApp”);

mccController

1

2

3

How do we scale application resources ?

mccController

1

2

3

Create CloudFoundry Client Object

Update application memory, disks, and instances

Stop and Start application

How can we apply these APIs to theMulti-Cloud Controller Architecture ?

mccMonitor

mccController

Environmentvariables

mccMonitor

Environmentvariables

Let us deploy the monitor and bind it to applications

running on each CloudFoundry instance

mccMonitor

mccController

Environmentvariables

/* Push mccMonitor to CloudFoundry Instances */pushMccMonitor( cloudFoundryURL );/* Bind CloudFoundry Applications to MccMonitor */bindApplicationsToMccMonitor( cloudFoundryURL, MccMonitorUrl );

/* Next Applications will provide workload specfiic statistics to MccMonitor… */

1

2

How do we enable the application to customize its scaling characteristics ?

mccMonitor

Environmentvariables

/* Applications get mccMonitor URL via VCAP environment variable */mccURL = getMccURL( System.getenv( “VCAP_SERVICES” ) );

/* Application provides known workload thresholds for each instance to mccURL */URL( mccURL + “&invokeThreshold=1000&updateThreshold=500” );

/* Application periodically reports accumulated statistics about specific workloads */URL( mccURL + “&binc=1&invokeCount=27&updateCount=4” );

App1 :{ invokeCount: 34invokeThreshold:

1000updateThreshold:

500 }

Let us follow the protocol between the applications and mccMonitor

mccMonitor

Environmentvariables

/* Applications get mccMonitor URL via VCAP environment variable */mccURL = getMccURL( System.getenv( “VCAP_SERVICES” ) );

/* Application provides known workload thresholds for each instance to mccURL */URL( mccURL + “&invokeThreshold=1000&updateThreshold=500” );

/* Application periodically reports accumulated statistics about specific workloads */URL( mccURL + “&binc=1&invokeCount=27&updateCount=4” );

App1 :{ invokeCount: 34invokeThreshold:

1000updateThreshold:

500 }

1

2

3

mccMonitor

mccController

Environmentvariables

Let us follow the protocol between mccMonitor and mccController

/* mccController loops thru all applications and checks last pollDateTime environment variable that it sets on every application to compute elapsed time since last poll */CloudApplication appl = client.getApplication(..);Map<String,String> env = appl.getEnvAsMap(); prevPoll = env.get(“lastpoll”); prevPollDateTime = new Date(prevPoll); elapsedTime = currDate - prevPollDateTime

/* mccController get applStats from monitor */String mccMonitorUrl = cfMonitormap.get(curCF);String jsonStruct = post2Monitor(mccMonitorUrl);// … continued

App1 :{ invokeCount: 34invokeThreshold:

1000updateThreshold:

500 }

1

2

mccMonitor

mccController

Environmentvariables

Protocol between mccMonitor and mccController (contd…)

/* mccController takes variables ending in Count and Threshold and computes overall rate */String jsonStruct = post2Monitor(mccMonitorUrl);JSONObject obj = JSONValue.parse(jsonStruct);Iterator it = obj.entrySet().iterator();while( it.hasNext() ) if( it.next().getKey().endsWith(“Count”)) { thres = thresKey(key); … // compute rate achieved per instance ratePerInst = getRate(elapsed,numIns); // Add instance if threshold exceeded addInst( ratePerInst, thres ); // Remove instances if below n% thres reduceInst( ratePerInst, thres );

App1 :{ invokeCount: 34invokeThreshold:

1000updateThreshold:

500 }

3

4

5

mccMonitor

mccController

Environmentvariables

Protocol between mccMonitor and mccController (contd…)

/* mccController also reads in pctGrowTrigger and pctShrinkTrigger variables to scale memory */String jsonStruct = post2Monitor(mccMonitorUrl);JSONObject obj = JSONValue.parse(jsonStruct);

pctGrowTrigger = obj.get(“pctGrowTrigger”)pctGrowAmount = obj.get(“pctGrowAmount”);pctShrinkTrigger = obj.get(“pctShrinkTrigger”);pctShrinkAmount = obj.get(“pctShrinkAmount”);

growOrShrinkMemory( pctGrowTrigger, pctGrowAmount, pctShrinkTrigger, pctShrinkAmount);

App1 :{ invokeCount: 34invokeThreshold:

1000updateThreshold:

500 }

6

7

8

Multi-Cloud Controller gathers resource utilization data

directly from multiple CloudFoundry Instances

for( cloudFoundryUrl : cfUrls ) { CloudFoundryClient client = new CloudFoundryClient(cloudFoundryUrL); List<CloudApplication> apps = client.getApplications();

for( CloudApplication app: apps ) { ApplicationStats stats = client.getApplicationStats( app.getName() );

for( InstanceStats is : stats.getRecords()) { InstanceStats.Usage usage = is.getUsage(); long memuse = usage.getMem(); long memQuota = is.getMemQuota(); …..

mccController

1

2

3

Multi-Cloud Controller provides global view of how each

CloudFoundry Instance utilize their resources

mccController

mccController

This CloudFoundry Instance shows a large portion of overall memory not being used, possible candidate for scaling down some instances ?

mccController

This CloudFoundry Instance shows hosts only a few low footprint applications, should these be moved to the other instance that can host more instances ?

How do we adapt these tools to the distributed replication solution ?

mccMonitor

mccControllermccMonitor

How do we adapt these tools to the distributed replication solution ?

mccMonitor

mccController

mccMonitor

How do we deploy the distributed apps shown below ?

Summary

Multi-cloud management and App customized scaling architecture makes applications active participants in the way they are scaled and how they consume resources within a collection of PaaS instances

Applications automatically scale up and down based on specific workloads they need to process

Distributed processes and Micro-services can self deploy and self bind to one another achieving loose coupling and independent scaling

Applications have full control of their destiny

Where can you find more information ?

CloudFoundry Client Java APIs https://github.com/cloudfoundry/cf-java-client

CloudFoundry Documentation http://docs.cloudfoundry.org/ https://github.com/cloudfoundry

Bosh Documentation http://docs.cloudfoundry.org/bosh/ http://www.think-foundry.com/cloud-foundry-bosh-introducti

on/

https://github.com/cloudfoundry/bosh-lite

MicroServices Architectures http://www.activestate.com/blog/2014/09/microservices-

resources

Thank you!