Measuring PerformanceChapter 12 CSE807

Performance Measurement

• To assist in guaranteeing Service Level Agreements

• For capacity planning

• For troubleshooting performance problems

• For Performance modeling

Step I

• What are the information sources of performance data

• What are the monitoring tools available for measuring response time and resource usage

• What techniques are used to calculate model input parameters

Sources of Performance Data

• Monitors– Provide raw data– Transformation and analysis

• Product specifications– Manufacturers– Benchmarks

Performance Measurement Framework

• Application oriented workload– Workload classes

• Should be tied to SLAs– Responsiveness– Availability– Reliability– Throughput

Capacity Planning Frameworkper application

• Measure and chart as a proactive analysis as a factor of time– Bandwidth usage– CPU and Memory utilization– End-to-end Response Time for frequently used

transactions– Number of concurrent users

Troubleshooting Framework

• Monitor collect data for problem detection– Paging– I/O hot spots– Network collisions, error rates– CPU usage– Queue depths

Troubleshooting Framework SystemAccounting Data Example

• sar -u • HP-UX FIADGMS1 B.11.00 B 9000/804 02/26/01 • 00:00:00 %usr %sys %wio %idle • 01:00:00 37 8 19 35• 02:00:00 22 5 34 39• 03:00:00 9 2 16 73• 04:00:00 7 2 14 77• 05:00:00 18 4 27 51• 06:00:00 6 2 14 78• 07:00:00 2 2 5 91• 08:00:00 1 1 4 94• 08:20:00 36 2 2 60• 08:40:00 51 1 1 46• 09:00:00 43 2 9 46• 09:20:01 35 4 15 45• Average 33 4 13 5088

Performance Tuning Framework

• Operating System Level– Establish kernel maximum and minimum

values - resource allocation per process

• Application and Database Level– Oracle SGA– Priority values for classes of workload– Application code tuning

Performance Management of Network-based Environment

• Clients– Application Response Monitor (ARM)– Workstation Performance Monitors

• Networks– Sniffers, Network Monitoring software

• Openview, Tivoli, CiscoWorks

• Servers, Proxies– OS monitoring tools

Measuring C/S Performance

Client Server

LANNetwork

LAN

Measurement Techniques

• Specify Reference Points– e.g. where to start the measurements from

• Specify Measurements– e.g. Packets, collisions, error rates statistics

• Instrument and Collect Data– Install & configure measurement tools

• Analyze and Transform Data– Accounting data, web logs are too detailed– Transform into useful information

A Representation of Measurement Process

Application Programs-------------------------------

Server Hardware------------------------------

Operating System ------------------------------

Hardware

Specify Reference

Point

Specify Measurements

Instrument &collect data

Analyze &transform

data

Type of Measurements

• Event Mode– State of the system defined by a set of variables– e.g. I/O request, Completion of a transaction

• Tools– Special code inserted at specific point of OS– Records generated with date, time, event and

other requested information– High overhead

Type of Measurements(cont.)

• Sampling Mode– At pre-defined time instants– Sampled at timer interrupts based on hardware

clock– Overhead depends upon

• # of variables

• Size of sampling interval

• large sampling interval will reduce statistical significance of variable

Data Collection Tools

• Hardware Monitors– Collects data at pre-defined signals– Electronic probes to sense the state of hardware– Registers, memory locations, I/O channels– Do not create overhead– Do not collect any software related information– Portable

Data Collection Tools(cont.)• Software Monitors

– Event or interrupt driven– Can record information regarding software and

operating system– Use resources– Easy to install and use– System Level - Global CPU, I/O, memory,

queue etc.– Program Level - Application oriented, program

identification, elapsed time, memory usage

Software Monitors

• Accounting Systems– Application level granularity– Most popular

• Program Analyzers– Used for measuring software related data– TP monitoring– Transaction counts, CPU/transaction

Software Monitors(cont.)

• Logs– NT Performance Monitor log can be used to

graph the usage statistics– HP-UX Measureware log data can be extracted

for historical trend analysis– Web Servers logs (Example 12.1)

• Access Logs

• Referrer Logs

• Agent Logs

• Error Logs

Performance Model Parameters

• Queues or Devices

• Workload Classes

• Workload Intensity

• Service Demands

Performance Model Parameters

• Queues– Determine what is the point of view– Determine the relevant queues– Queues in a System– Jobs waiting for components

• CPU, Memory, I/O

Performance Model Parameters

• Workload Classes– Logical grouping of workload– OPEN vs. CLOSED– Open useful for forecasting– Realistic for web servers modeling

Performance Model Parameters

• Workload Intensity - define the amount of work system needs to perform– Arrival Rate - # of cars arriving at the toll plaza

per minute– Number of customers - # of customers in bank

per hour– Pages requested per transaction– Average packet size

Performance Model Parameters

• Service Demand– For hardware resources– Service time for satisfying a request in the

queue– Does not include waiting time– Non-linear scaling for additional CPU caused

by OS overhead to manage CPU allocation– Example 12.2

Performance Model Parameters

• Parametric Estimation - combination of information provided by program analyzers– Identify the type of execution environment– Specify the measurement process– Monitor the system and collect the data– Estimate input parameters– Example 12.3

Collecting Performance Data

• Networks– SNMP - Simple Network Management Protocol– Manager and Agent– Management Information Database (MIB)– e.g. Openview, Tivoli– LAN analyzers - packets/sec, collision rate– RMON - Remote Monitoring MIB

Collecting Performance Data

• Servers– NT - Performance Monitor

• Counter by object types - CPU, Memory, disk etc.

• Averaging Counters

• Difference Counter - last and previous

• Chart, report, alert and logs

Collecting Performance Data

• UNIX– SAR - System Activity Report, a sampling tool– ps– vmstat– iostat– netstat– logfiles

Summary

• Guidelines for performance measurement

• Types of monitors

• Measurement techniques– Specify reference point– Specify measurements– Instrument and collect data– Analyze and transform