cloudstack metering working with the usage data metering working with the usage data tariq iqbal...

CloudStack MeteringWorking with the Usage Data

Tariq Iqbal

Senior Consultant

[email protected]

Twitter: @TariqIqbal_ @ShapeBlue

@ShapeBlue #CloudStack #CCCNA14

Involved with CloudStack before donation to Apache

Built and deployed CloudStack / CloudPlatform based clouds for Enterprises and Service Providers globally - SunGard, Ascenty

Specialise in integrating CloudStack with Business Support and Operational Support systems, which include:

CloudPortals (including Payment Gateways, Fraud Control solutions)

CRM/ERP Systems – SAPB/1, Salesforce.com

Monitoring Solutions – Nagios, Splunk, Zenoss

Ticketing Systems – ServiceDesk, Email

About Me


“ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform

integrator & consultancy”

“First to offer professional 24x7x365 support of Apache CloudStack and now CSForge”

“ShapeBlue is absolutely one of the top experts on deploying CloudStack. Great company and very deep skill set”

About ShapeBlue


CloudStack Usage

CloudStack Configuration

Usage Data Generation

Usage Data Access

Customer Use Cases

Overview


Organisations need to be able to meter their cloud's consumption to:

Monetize the cloud offerings by charging for usage

Report on the cloud’s resource usage for monitoring/capacity planning

Majority of organisations already have existing BSS/OSS and just need access to the usage data

Organisations tend to have different billing criteria/charging models and require access to usage data for their specific needs

Usage Requirements


CloudStack does not offer any native usage monitoring or billing/chargeback capability

Root Admin can view currentSystem Capacity

Domain Admin and Users can view VM statistics

Usage in CloudStack today


CloudStack includes a Usage Server:

Optional Component

Separately installed service called ‘cloudstack-usage’

Runs once a day (Default)

Creates aggregated usage records for the various resources consumed in CloudStack

The usage records are stored in a separate database called ‘cloud_usage’

Usage Server


Usage Server Configuration


Usage Server Configuration Examples


As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in cloud.usage_events

There are 110 different events. Full list in the Programmer Guide

Network usage is the only exception which doesn't use Events

CloudStack Events


1) CloudStack events are logged into cloud.usage_event table

2) The cloudstack_usage job gets a list of the latest usage events

3) Inserts these events into cloud_usage.usage_events table

4) Parses cloud_usage.usage_event and populates Helper tables

5) Helper table data used to populate cloud_usage.cloud_usage

table with aggregation range wise data

Usage Record Generation


Usage Types


Usage Types

API: http://<HOSTIP>:8096/client/api?command=listUsageTypes


Usage Records show the amount of resources consumed by guest instances.

The 8 Usage Record formats are:

Allocated & Running VMs

Network Usage

IP Address

Disk Volume

Template, ISO and Snapshot

Load Balancer or Port Forwarding

Network Offering

VPN User

Details of the Usage Record formats can be found in the Developers Guide

Usage Records


VM Usage Record Format


To Create, Start, Stop and Restart a VM:

Usage Record Example


Accessing Usage Data

CloudStack Usage Data


The usage records can be accessed through the CloudStack API

This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed

CloudStack provides the following Root Admin API calls and responds with XML and JSON responses:

listUsageTypes – lists available Usage Types

listUsageRecords - provides Usage records for a date range

generateUsageRecords - asynchronous usage record generation

CloudStack API


The base API request is:http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=yyyy-MM-dd&enddate=yyyy-MM-dd&signature

Optional request parameters:

ListUsageRecords API


<listusagerecordsresponse><count>1</count><usagerecord>

<account>user5</account><accountid>10004</accountid><domainid>1</domainid><zoneid>1</zoneid><description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description><usage>2.95288 Hrs</usage><usagetype>1</usagetype><rawusage>2.95288</rawusage><virtualmachineid>4</virtualmachineid><name>i-3-4-WC</name><offeringid>1</offeringid><templateid>3</templateid><usageid>245554</usageid><type>XenServer</type><startdate>2009-09-15T00:00:00-0700</startdate><enddate>2009-09-18T16:14:26-0700</enddate>

</usagerecord>

</listusagerecordsresponse>

ListUsageRecords Response


CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API

It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing

Filtering can be used to limit the result set

It supports argument passing and shell automation

The host, port, apikey and secretkey can be configured on setup

CloudMonkey CLI


CloudMonkey list usagerecordsThe command to retrieve Usage Records is:

cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f-080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2


The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack

Usage Data can be exported in a CSV format from a SELECT query

The SQL query can become complex if the ID fields in the data need to be de-referenced

Typically multiple passes of the query results may be required to extract the required information

SQL


CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP-

based brokers Use case: A third-party cloud usage solution can subscribe to CloudStack

events and generate usage data which can be consumed by their billing software

Great blog from Chip Childers on configuring a CloudStack management server to publish events, and some sample code that prints each event to the console as they are received

http://www.chipchilders.com/blog/2013/7/16/tapping-into-apache-cloudstack-events-via-amqp.html

http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Admin_Guide/events.html

CloudStack Message Queue

http://www.chipchilders.com/blog/2013/7/16/tapping-into-apache-cloudstack-events-via-amqp.html

http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Admin_Guide/events.html


Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system

Considerations:

Do you actually need all the Usage Data for all Resources?

Any UUID values of the CloudStack resources should also be imported into the target system for reference purposes

Will you need to synchronise accounts and resources in CloudStackwith the target system?

Mediation


Customer Use Cases

CloudStack Usage Data


MS Excel can be used to analyse CloudStack Usage Records

The usage records are imported from the cloud_usage table into MS Excel via an MySQL ODBC connection

Either PivotTables or reports can be used to present the data

This is convenient and

great for integration

testing between CloudStack

and external systems

MS Excel


Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform

The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM

Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the

users billing cycle

Citrix CloudPortal


Splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs

We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data

Setup feature-rich dashboards

and the capability to drill down into

the Usage Data for BI reporting

Splunk>


Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds

Dashboard Integrated in the CloudStack UI

Pricing, Alerting, Reporting, Capacity Mgmt

Amysta consolidates public cloud expenses

e.g. AWS and private cloud consumption in

one dashboard view

Amysta


Slides: www.slideshare.net/shapeblue

Blogs: www.shapeblue.com/blogs

CloudStack Programmer Guide, API Documentation & Wiki:http://cloudstack.apache.org/

CloudStack Mailing Lists (Users, Development, Marketing):http://cloudstack.apache.org/mailing-lists.html

CloudMonkey:https://cwiki.apache.org/CLOUDSTACK/cloudstack+cloudmonkey+cli

How to Use MS Excel to Analyze CloudStack Usage Records:http://support.citrix.com/article/CTX132030

Further Information

http://www.slideshare.net/shapeblue

http://www.slideshare.net/shapeblue

http://cloudstack.apache.org/

http://cloudstack.apache.org/mailing-lists.html

https://cwiki.apache.org/CLOUDSTACK/cloudstack+cloudmonkey+cli

http://support.citrix.com/article/CTX132030

Working with CloudStack Usage Data

Tariq Iqbal

Senior Consultant

[email protected]

Twitter: @TariqIqbal_ @ShapeBlue

cloudstack metering working with the usage data metering working with the usage data tariq iqbal...

Documents