cloud usage with cloudstack

© 2014 Sungard AS. All rights reserved.

Cloud Usage with CloudStackSept 2014

Chandra Prakash Mishra

2

In next 45 minutes …

Agenda

• CloudStack Usages

• Cloudstack configuration to enable Usage Capture

• Usage Data Generation Process

• Usage Type

• Usage Event

• Usage Records

• How can Access Usage Data

• Use Cases

3Footer – Apply across document

Usage Requirements

• Meter their cloud's consumption and monetize the cloud offerings by charging for usage

• Report on the cloud’s resource usage for monitoring/capacity planning

• Existing billing system, just need access to the usage data

• Organizations tend to have different billing criteria/charging models and require access to usage data for their specific needs


Usage in CloudStack today

• CloudStack does not offer any native usage monitoring or billing capability

• Root Admin can view current System Capacity

• Domain Admin and Users can view VM statistics

5

Usage Server

• Optional

• Separately installed

• Provide aggregated usage records

• Run at least once per day (Default):1440

• Take data from the events log in cloud DB “cloud.usage_event”

• Create summary usage records in cloud_usage DB “cloudusage.usage_event”

• Access usage records using listUsageRecords“http://localhost:8096/client/api?command=listUsageRecords&startdate=2013-05-25&enddate=2013-06-05”

http://localhost:8096/client/api?command=listUsageRecords&startdate=2013-05-25&





6

Usage Server Config

Name Description value

enable.usage.server Flag for enabling usage true

usage.stats.job.exec.time Time when the usage server will start processing

00:15

usage.execution.timezone The timezone to use for usage job execution time

Usage server time zone

usage.stats.job.aggregation.range Usage is aggregated for specified number of minutes

1440

usage.aggregation.timezone The time zone to user for usage stats aggregation

GMT


Usage Server Config

Usage Server

Name Value

usage.stats.job.exec.time 00:15

usage.execution.timezone PST

usage.stats.job.aggregation.range 60

usage.aggregation.timezone GMT

• Usage Job will run at PST 00:15

• Generate usage records for 1hrs from 07:15:00 GMT to 08:14:59 GMT

• Next exec.time is 01:15

Usage Type

8

ID Usage Type ID Usage Type

1 RUNNINGVM 8 ISO

2 ALLOCATEDVM 9 SNAPSHOT

3 IPADDRESS 11 LOADBALANCERPOLICY

4 NETWORKBYTESSENT 12 PORTFORWARDING

5 NETWORKBYTESRECEIVED 13 NETWORKOFFERING

6 VOLUME 14 VPNUSERS

7 TMEPLATE

9

CloudStack Event

• Usage events are stored in usage_event table. Any resource for which usage has to be tracked needs to log usage events when the the resource is created/allocated and when its deleted/released

• Network usage is the only exception which doesn’t use usage events• Once these events are logged usage server will process them and generate Cloudstack usage

records

Description of various columns in usage_event table:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| 12 | VM.START | 2 | 2011-02-09 11:16:18 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 || 15 | VM.STOP | 2 | 2011-02-09 12:00:24 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 |+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +| 2 | NET.IPASSIGN | 2 | 2011-02-09 11:16:18 | 1 | 10 | 10.91.30.121 | NULL | NULL | 1 | 0 || 9 | NET.RELEASE | 2 | 2011-02-09 12:00:24 | 1 | 21 | 10.91.30.121 | NULL | NULL | 0 | 0 |+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Running VM

Public IP address


Usage Record Generation

• CloudStack events have been logged into cloud.usage_event table

• The cloudstack_usage job gets a list of the latest usage events

• Inserts these events into cloud_usage.usage_events table

• Parses cloud_usage.usage_event and populates Helper tables

• Helper table data used to populate cloud_usage.cloud_usage table with aggregation range wise data

11

Network Usage Record Generation

• Network usage is processed differently and is not driven by usage events

• In Virtual Networking case Iptables rules are set in domR to track public traffic

• Management server queries for Network stats periodically and populates them in user_statistics table account-wise

• Cloud.user_statistics is queried and populated to cloud_usage.user_statistics with latest stats

• Network helper entry is created in usage_network table

• Network usage for current event is calculated as:

bytesSent = (total bytes sent) – (bytes sent prior to this event);bytesReceived = (total bytes received) – (bytes received prior to this event);

Usage Calculation

12

1-1-VM Event Timedeployed 2013-06-05 12:00

start 2013-06-05 12:00

stop 2013-06-05 18:00

start again 2013-06-05 23:00

startdate 2013-06-05 00:00:00

enddate 2013-06-05 23:59:59

RUNNINGVM(Usage Type 1)

6+1 = 7 hours

ALLOCATEDVM(Usage Type 2)

12 hours

The Usage Record


Usage Record Format

Virtual Machine Usage Record Format: For running and allocated vm usage

account name of the account

accountid ID of the account

domainid ID of the domain in which this account resides

zoneid Zone where the usage occurred

description A string describing what the usage record is tracking

usage String representation of the usage, including the units of usage (e.g. 'Hrs' for VM running time)

usagetype A number representing the usage type (see Usage Types)

rawusage A number representing the actual usage in hours

virtualMachineId The ID of the virtual machine

name The name of the virtual machine

offeringid The ID of the service offering

templateid The ID of the template

usageid Virtual machine

type Hypervisor

startdate, enddate The range of time for which the usage is aggregated; see Dates in the Usage Record

14

Usage Record Format

account name of the account

accountid ID of the account

domainid ID of the domain in which this account resides

zoneid Zone where the usage occurred

description A string describing what the usage record is tracking

usagetype A number representing the usage type (see Usage Types)

rawusage A number representing the actual usage in hours

usageid Device ID (virtual router ID or external device ID)

type Device type (domain router, external load balancer, etc.)

startdate, enddate The range of time for which the usage is aggregated; see Dates in the Usage Record

Network Usage Record Format:

For more details Please refer this link:https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/API_Developers_Guide/usage-record-format.html


How we can access usage data


CloudStack API

• The usage records can be accessed through the CloudStack API

• This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed

• CloudStack provides the following Root Admin API calls and responds with XML and JSON responses

• listUsageTypes – lists available Usage Types

• listUsageRecords - provides Usage records for a date range• • generateUsageRecords - asynchronous usage record generation


ListUsageRecords API

The base API request is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature • Optional request parameters:

18

<listusagerecordsresponse> <count>1816</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description> <usage>2.34 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.34</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord> ………</listusagerecordsresponse>

Example response from listUsageRecords


CloudMonkey CLI

• CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API

• It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing

• Filtering can be used to limit the result set • It supports argument passing and shell automation

• The host, port, apikey and secretkey can be configured on setup


CloudMonkey list usagerecords

The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10 enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07


SQL

• The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack

• Usage Data can be exported in a CSV format from a SELECT query

• The SQL query can become complex if the ID fields in the data need to be de-referenced

• Typically multiple passes of the query results may be required to extract the required information


CloudStack Message Queue

• CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers

• Use case: A third-party cloud usage solution can subscribe to CloudStack events and generate usage data which can be consumed by their billing software

For more details Please refer this link: https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus


Customer Use Cases


MS ExcelMS Excel

·MS Excel can be used to analyse CloudStack Usage Records ·The usage records are imported from the cloud_usage table into MS

Excel via an MySQL ODBC connection ·Either PivotTables or reports can be used to present the data ·This is convenient and great for integration testing between CloudStack

and external systems


Splunk

• Splunk is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs

• We then connected splunk to the CloudStack Usage database and automated the collection and indexing of Usage Data

• Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting


Amysta

Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity Mgmt Amysta consolidates public cloud expenses e.g. AWS and private cloud

consumption in one dashboard view

Fore more details please refer this link:https://www.youtube.com/watch?v=ZA-VPb7WAy8http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing


SunGard Cloud Portal

Thank You!

cloud usage with cloudstack

Documents