Performance measurement for cloud computing applications

using ISO 25010 standard characteristics

Anderson Ravanello, Jean-Marc Desharnais, Luis Eduardo Bautista Villalpando, Alain April, Abdelouahed Gherbi

ravanello@gmail.com, jean-marc.desharnais@etsmtl.net, luis.bautistav@gmail.com,alain.apri@etsmtl.ca, Abdelouahed.Gherbi@etsmtl.ca

Background of cloud computing

Cloud computing is an emerging technology.

This technology is being broadly adopted by the industry as means to achieve mobility, reduced costs and ubiquity. (Voas and Zhang, 2009)

One of the most important challenge in delivering Cloud Services is to ensure that they are fault tolerant (Bautista et al., 2013)

The system performance is unreliable due to the complexity of the infrastructure

Characteristics of cloud computing

• Cloud computing is expanding

• Measuring performance for this infrastructure is complex

• Measuring performance from log data involves evaluating large amounts of data

Rapid processing of query results to user is important (ex. Google) and is a part of the performance

Objective of this research

The main objective of this research is to show how base and derived measures can be map to reveal the performance of a cloud-based applicationThis will be tested in the context of a large Microsoft

Exchange application installed in a private cloud and its distributed clients (10000 servers around the world)

To take in account the complexity of the infrastructure we will implement the measurement framework developed by Bautista. This measurement framework is using quality characteristics of ISO 25010 (ISO SQuaRE series)

Background: comparative complexity of standard computing X client computing

Client server (simpler) (IBM, 2013)

Cloud Computing (more complex) (Lemay, 2012)

Figure 1: Comparative complexity between client server and cloud computing infrastructure

Background: The studied private cloud

Our case study was conducted on commercial software running on the CC infrastructure of a private cloud that mainly hosts and enables access to a company’s email services.

VM host

Lan

DNS

Firewall

WAAS Router

DMZ Verizon MPLS Backbone

Router WAAS

DMZ

Firewall

Lan

AD

Unix Filer

Virtualization server

CAS Mbx/db

Figure 2: Studied private cloud

Performance Measurement Framework

To achieve a performance measurement for this private cloud, we faced 2 challenges:1. How to determine the performance criteria?

We choose a limited number of characteristics and sub characteristics from ISO 25010 (e.g. time behavior)

2. How to choose and link the 'measures' to the sub characteristics and characteristics?

Choice of the 'measures' from the logs generate by the nodes and apply to a specific characteristics (see methodology)

Methodology

1. Data collection: automated from the performance logs generated by the 12 nodes (Figure 2). In this presentation only 2 were used (CAS, MBX/DB

2. Focus on the time behavior from ISO 25010 characteristics and sub-characteristics and apply each 'measures' to the pertinent sub characteristics

3. Data organization: physical location (North America, Europe, Asia) and day time (business and non-business hours)

4. Data analysis: statistical analysis results (averages, variances, kurtosis, and skewness), and the results of a visual examination of the time behavior graphs (see radial graph, figure 4)

Data collectionThe data collected from the logs of two nodes are mainly

'low level derived measures' (*).

The total private cloud is composed of approximately 80,000 client machines and 10,000 servers and network devices. In this study we collected data from 10,000 servers and display the data of 12 servers for the duration of 1 week – this section of data is 600 MB and is represented on figure 6.

With the total number of clients and servers, around 800,000 data points per minute are generated. To visualize this situation, imagine a spreadsheet which grows by 800,000 lines of data every minute, with each line made up of approximately 1,000 characters.

(*) There are 159 low level derived measures. Low level derived measures are the most atomic andgranular measures that are available in operational systems from this Cloud Computing Application.

Measurement of Time Behaviour

With the data available in this case study, we were able to assess the time behaviour quality characteristic via the email service usage low level derived measures presented partially (only 3 - see table) for the transmission function:

Measurement and Bautista framework

Bautista suggest a number of derived measures based on different characteristics.

Time behaviour and Bautista framework

It is possible to reverse the previous table using Time behaviour characteristics:

Time behaviour

Task function

Task executed

Task passed

etc.

Time function

Down time

Maximum response

time

etc.

Transmission function

Transmission error

Transmission capacity

etc.

Data organisationThree physical locations (North America, Europe, and

Asia),

Processed over a 1 week period during business and non business hours.

A sub grouping of the data was created per host status (machine level) to allow us to compare usage on a machine by machine basis.

Finally a data grouping is necessary to analyse the results. For example, base statistics for one measure is divide in business hours (ON) vs. non business hours (OFF). Will be presented in the analysis of the data next.

Results analysisAnalysis of the data (comparison)

Visual interpretation (global)

Analysis: single point measure and index creation

Analysis of the dataWeb Service \ Bytes Sent/sec \

_Total (On hours)Web Service \ Bytes Sent/sec \

_Total (Off hours)Mean 43 KBps Mean 28 KBpsStandard Error 1.5 KBps Standard Error 979.83Median 20 KBps Median 18 KBpsMode #N/A Mode #N/AStandard Deviation 84 KBps Standard Deviation 35 KBpsSample Variance 7GBps Sample Variance 1 GBpsKurtosis 27.77 Kurtosis 49.97Skewness 5.01 Skewness 5.73Range 821 989.44 Range 428426.46Minimum 2KBps Minimum 3.1 KBpsMaximum 829 MBps Maximum 431MBpsSum 124978002.8 Sum 37266022.34Count 2857 Count 1309Largest (5) 691MBps Largest (5) 343MBpsSmallest (5) 3 MBps Smallest (5) 3.2 KBpsConfidence Level (95.0%) 3 MBps

Confidence Level (95.0%) 1.9MBps

Example of the statistical data calculated (one characteristic, business VS off hours)

Analysis: Visual interpretationLow Level Derived measures (LLDM)

System \ Processor Queue Length \

Web Service \ Bytes Sent/sec \ _Total

Web Service \ Connection Attempts/sec \ _Total

Web Service \ Current Connections \ _Total

Figure 4: Four LLDM, 8am to 6pm. Higher numbers mean more resources used.

Analysis: single point measure and index creation

Network Interface \ Bytes Received/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _51

Network Interface \ Bytes Received/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _52

Network Interface \ Bytes Sent/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _51

Network Interface \ Bytes Sent/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _52

Network Interface \ Bytes Total/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _51

Network Interface \ Bytes Total/sec \ HP NC382i DP Multifunction Gigabit Server Adapter _52

System \ Processor Queue Length \

Web Service \ Bytes Sent/sec \ _Total

Web Service \ Connection Attempts/sec \ _Total

Web Service \ Current Connections \ _Total

Figure 5 is the measure of a single point in time; on this second, all these LLDM had different values. If we calculate the area of this figure, we get to a value that could be an indicator of the LLDM of that second.

This process was then replicated to all the data points, leading to the graph on figure 6 (next)

Figure 5: Time behavior at a single moment in time

Analysis: Index Creation

On this figure, we see peaks on the 11th and 13th, on different hours; these are moments on the equivalent area of the radial figure would be bigger, indicating a possible degraded performance for the whole system. The inverse is valid for the end of the week, where the index is lower.

Figure 6: Evolution of the time behavior index

Tim

e

2/10

/14

4:00

PM

2/11

/14

11:0

0 AM

2/12

/14

9:00

AM

2/12

/14

3:00

PM

2/13

/14

1:00

PM

2/14

/14

8:00

AM

2/14

/14

5:00

PM

2/15

/14

12:0

0 PM

2/16

/14

9:00

AM

2/16

/14

3:00

PM

0.00E+00

5.00E+05

1.00E+06

1.50E+06

2.00E+06

2.50E+06

3.00E+06

3.50E+06

4.00E+06

Index

Conclusion and future research

Demonstrated the possibility of using the framework to measure quality characteristic

Challenges regarding data collection, data processing and data representation

Future research: improved statistic techniques, larger time frame, different quality measures, real time processing.