cpe 619 monitors aleksandar milenković the lacasa laboratory electrical and computer engineering...
TRANSCRIPT
CPE 619Monitors
Aleksandar Milenković
The LaCASA Laboratory
Electrical and Computer Engineering Department
The University of Alabama in Huntsville
http://www.ece.uah.edu/~milenka
http://www.ece.uah.edu/~lacasa
2
Part II: Measurement Techniques and Tools
Measurements are not to provide numbers but insight - Ingrid Bucher
Measure computer system performance Monitor the system that is being subjected to a
particular workload How to select appropriate workload
In general performance analysis should know1. What are the different types of workloads?2. Which workloads are commonly used by other analysts?3. How are the appropriate workload types selected?4. How is the measured workload data summarized?5. How is the system performance monitored?6. How can the desired workload be placed on the system in a
controlled manner?7. How are the results of the evaluation presented?
3
Outline
Introduction Terminology Software Monitors Hardware Monitors Monitoring Distributed Systems
4
Monitors
A monitor is a tool used to observe activities on a system
Observe performance Collect performance
statistics May analyze the data May display results May even suggest
remedies
Monitors are used not only by performance analysts
Systems programmer may profile software
System manager may measure resource utilization to find bottleneck
System manager may use to tune system
System analyst may use to characterize workload
System analyst may use to develop models or inputs for models
That which is monitored improves. – Source unknown
5
Outline
Introduction Terminology Software Monitors Hardware Monitors Monitoring Distributed Systems
6
Terminology
Event – a change in the system state E.g.: cache miss, page fault, process context switch, beginning of
seek on a disk, arrival of a packet, Trace – a log of events, usually including the time of the event,
and other important parameters Overhead – most monitors perturb the system operation
Use CPU or storage; Sometimes called artifact. Goal is to minimize artifact
Domain – the set of activities observable by the monitor E.g.: accounting logs record information about CPU time, number
of disks, terminals, networks, paging I/O’s, the number of characters transferred among disks, terminals, networks, and paging devices
7
Terminology (cont’d)
Input rate – the maximum frequency of events that monitor can correctly observe Burst mode: the rate at which an event can occur for a
short period of time Sustained mode: the rate the monitor can tolerate for
long durations Resolution – coarseness of the information observed Input width – the number of bits recorded for each
event. Input rate x width = storage required
8
Monitor Classification
Implementation level Software, Hardware, Firmware, Hybrid
Trigger mechanism Event driven – activated only by occurrence of certain events;
Low overhead for rare event, but higher if event is frequent
Sampling (timer driven) – activated at fixed time intervals by clock interrupts
Ideal for frequent events Display
On-line – provide data continuously. E.g.: tcpdump Batch – collect data for later analysis. E.g.: gprof.
9
Outline
Introduction Terminology Software Monitors Hardware Monitors Monitoring Distributed Systems
10
Software Monitors
Monitor operating systems, and higher level software,
e.g., networks, databases At each activation, several instructions are executed
In general, only suitable for low frequency event or overhead becomes too high
Overhead may be OK if timing does not need to be preserved;
Lower input rates, lower resolutions, and higher overhead than hardware
But, they have higher input widths, higher recording capacities
Easier to develop and modify
11
Issues in Software Monitor Design - Activation Mechanism
How to trigger the data collection routine? 1) Trap – instrument the system software with trap instructions
at appropriate points. Collect data. Like a subroutine. E.g.: to measure I/O service time, trap before I/O service routine
and record time, trap after, take diff 2) Trace – each instruction is followed by data collection
routine (trace mode). Enormous overhead. Time insensitive. E.g., instruction-trace monitor to produce a PC histogram
3) Timer interrupt – a timer interrupt service provided by the OS is used to transfer control to a data collection routine at fixed intervals.
Overhead is independent of the event rate If sampling counter, beware of overflows
12
Issues in Software Monitor Design – Buffer Size
Store recorded data in buffers in memory, which are later written to hard disk
Buffers should be large To minimize the need to write frequently to hard disk
Buffers should be small Don’t have a lot of overhead when write to disk Doesn’t impact performance of system
(or reduced memory availability is not observable) Optimal buffer size is a function of the input rate,
input width, and emptying rate
13
Issues in Software Monitor Design – Number of Buffers
Usually organized in a ring Allows recording (buffer-emptying) process to
proceed at a different rate than monitoring (buffer-filling) process Monitoring may be bursty
Since cannot read while process is writing, a minimum of two buffers required for concurrent access
May be circular for writing so monitor overwrites last if recording process too slow
May compress to reduce space, but adds overhead
14
Issues in Software Monitor Design – Buffer Overflow
In spite of a ring, all buffers could become full Two options (both result in information loss)
Overwrite a previously written buffer Old information is lost
Stop monitoring until a buffer becomes available New information is lost
Trade-off: old vs. new information importance Counter overflows
15
Issues in Software Monitor Design – Misc
Data Compression or Analysis Online compression/processing before storing to
reduce storage requirements On/Off
Most hardware monitors have an on/off switch Software can have “if … then” but still some overhead.
Or can “compile out” E.g.: remove “-pg” flag E.g.: with #define and #ifdef
Priority Asynchronous, then keep low. If timing matters, need
it sufficiently high so doesn’t caus skew
16
Outline
Introduction Terminology Software Monitors Hardware Monitors Monitoring Distributed Systems
17
Hardware Monitors
Hardware monitors -- separate pieces of equipment attached to the system being monitored via probes No system resources are consumed in monitoring
Generally, lower overhead, higher input rate, reduced chance of introducing bugs
Can increment counters, compare values, employ timers, record histograms of observed values … Range from simple logic elements and counters to
sophisticated computer systems Usually, gone through several generations and
testing so is robust
18
Software vs. Hardware Monitors What level of detail to measure?
Software more limited to system layer code (OS, device driver) or application or above
Hardware may not be able to get above information What is input rate? Hardware tends to be faster Expertise?
Good knowledge of hardware needed for hardware monitor Good knowledge of software system (programmer) needed
for software monitor Most hardware monitors can work with a variety of
systems, but software may be system specific Most hardware monitors work when there are bugs,
but software monitors brittle Hardware monitors more expensive
19
Firmware and Hybrid Monitors
Firmware monitors fall between hardware and software monitors
Implemented by modifying the processor microcode Hybrid: combines hardware, firmware, software
monitoring E.g., use hardware components to capture events and
software modules to compress/analyze collected data
20
Outline
Introduction Terminology Software Monitors Hardware Monitors Monitoring Distributed Systems
21
Monitoring Distributed Systems
More difficult than single computer system
Monitor itself must be distributed
Easiest with layered view of monitors
May be zero+ components of each layer
Many-to-many relationship between layers
Management Console Interpretation Presentation Analysis Collection Observation
Distributed system: many hardware and software components working together separately and concurrently
Layered view of a distributed-system monitor
22
Layered View
Observation – gather raw data on individual components of the system; each component may have an observer designed specifically for it
Collection – collects data from various observers; may have more than one observer on large systems
Analysis – Analyzes data gathered at various collectors. May include various statistical routines to summarize the data characteristics
Presentation – Deals with human user interface (reports, displays, alarms)
Interpretation – Intelligent entity (human or expert system) that can make meaningful interpretations of the data (more sophisticated than simple threshold-based rules)
Console – Interface to control the system parameters and states (outside monitor)
Management – Entity that makes the decision to set or change system parameters or configuration (manager). Implements decisions suing consoles.
23
Components of a Distributed Systems Monitor
Subsystem1 Subsystem2Subsystem3
Observer1 Observer2 Observer3
Collector1 Collector 2
Analyzer1 Analyzer2
Presenter1 Presenter2
Interpreter1 Interpreter2
Console1 Console2
Manager1 Manger2HumanBeings
24
Observation (1 of 2)
Concerned with data gathering Implicit spying – promiscuously observing the
activity on the bus or network link Little impact on existing system Accompany with filters that can ignore some
events E.g.: tcpdump between two IP address
Explicit instrumentation – incorporating trace points, hooks, … Adds overhead, but can augment implicit data E.g.: may have application hooks
logging when data sent
25
Observation (2 of 2)
Probing – making “feeler” requests to see performance E.g.: packet pair techniques to gauge capacity (a
special packet sent to a given destination and looped back may provide info about queuing at the source, intermediate bridges, the destination, and back
There is overlap between the three techniques, but they are not totally redundant -- often one shows a part of the system that others cannot
26
Collection
Data gathering component, perhaps from several observers E.g.: I/O and network observer on one host could
go to one collector for the system May have different collectors share same
observers Collectors can poll observers for data Or observers can advertise when they have data
Clock synchronization can be an issue Usually aggregate over a large interval
to account for skew
27
Analysis
More sophisticated than collector Division of labor unclear, but usually, if fast,
infrequent in observer, but if takes more processing time, put in analyzer
Or, if it requires aggregate data, put in analyzer Ex: if successful transaction rate depends upon
disk error rate and network error rate then analyzer needs data from multiple observers
General philosophy, simplify observers and push complexity to analyzers
28
Presentation (1 of 2)
User interface, closely tied with monitor function Three key functions 1) Performance monitoring – helps quantify if service
provided is correct Throughput, response time, utilization of different
components Summary statistics Time stamped traces
29
Presentation (2 of 2)
2) Error monitoring – incorrect performance Error statistics, counts or traces Maybe sort to help determine what part of system is
unreliable 3) Configuration monitoring – non-performance of the
system components Tell which are up Show initial configurations May show only incremental configurations Scope to allow zoom or whole system
30
Interpretation and Console
Interpreter – uses set of rules to make judgments about state of system Often need expert system to warn about faults before
they occur May suggest configuration changes
Console functions – allow system manager to change system, bring up and down, allow remote diagnostics Ideally, one console can get feedback and apply
configuration, but some parts may be vendor specific
32
Performance Tuning
Performance tuning steps 1) Define the performance problem 2) Identify the bottlenecks using
monitoring and measurement tools 3) Remove bottlenecks by applying
a tuning methodology 4) Repeat steps 2 and 3 until
you find a satisfactory resolution
33
Measuring Execution Time
No changes to the program date time
Added to the program code directly clock gettimeofday
Program profilers gprof
34
Using the date Command
Read ~/docs/ performance.measurement.txt
* To learn more about the date command type in man date.
sr4 $ date && dsize 12 && dateThu Jan 11 16:04:58 CST 2007-1473822656 TOT_INS: 490005749 Thu Jan 11 16:04:59 CST 2007
sr4 $ date && dsize 24 && date
Thu Jan 11 16:08:16 CST 2007
1529910656
TOT_INS: 946006155
Thu Jan 11 16:08:18 CST 2007
sr4 $ date && dsize 36 && date
Thu Jan 11 16:07:39 CST 2007
1604971008
TOT_INS: 1402006388
Thu Jan 11 16:07:42 CST 2007
35
Using the time Command
Read ~/docs/ performance.measurement.txt
* To learn more about the date command type in man time.
sr4 $ time dsize 12
-1473822656
TOT_INS: 490005733
real 0m1.217s
user 0m1.040s
sys 0m0.090ssr4 $ time dsize 24
1529910656
TOT_INS: 946006063
real 0m2.154s
user 0m1.980s
sys 0m0.070s
sr4 $ time dsize 36
1604971008
TOT_INS: 1402006545
real 0m3.084s
user 0m2.930s
sys 0m0.090s
36
Using the clock() Function
The clock() function allows you to measure the time spent in a section of a program
* To learn more about the clock() function type in man clock
* A typical program template for using the clock() function
#include <time.h>
....
int main(void) {
clock_t start_time, finish_time;
...
// determine overhead
start_time = clock();
finish_time = clock();
double delay_time = (double) (finish_time - start_time);
...
start_time = clock();
...// code you want to determine the execution time for
finish_time = clock();
double elapsed_time = finish_time - stat_time - delay_time;
double elapsed_time_sec = elapsed_time/CLOCKS_PER_SEC;
...
}
37
Using the gettimeofday() function
* To learn more about this function type in man gettimeofday
The function gettimeofday returns two integers
The first one indicates the number of seconds from January 1, 1970
and the second returns the number of microseconds since the most recent second boundary.
* A sample program that uses gettimeofday().
#include <stdio.h>
#include <sys/time.h>
struct timeval start, finish ;
int msec;
int main ()
{
gettimeofday (&start, NULL);
sleep (200); /* wait ~ 100 seconds */
gettimeofday (&finish, NULL);
msec = finish.tv_sec * 1000 + finish.tv_usec / 1000;
msec -= start.tv_sec * 1000 + start.tv_usec / 1000;
printf("Time: %d milliseconds\n", msec);
}
38
Program Profiling
Profilers are utility programs used to determine execution profiles,in other words they tell us how much time is spent in each subroutine or function
10-90 rule of thumb states that 10% of your code is responsible for 90% of the program execution time
Tuning the most time-consuming subroutines that dominate execution time can be very rewarding (assuming that we do this right)
The profiler collects the data during the program's execution Typical steps in profiling are as follows:
enable it when compiling and linking programs a profiling data file are generated when the program is executed profiling data are analyzed using gprof
39
Example: gprof
An excerpt from testsort.report:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@....
granularity: each sample hit covers 4 byte(s) for 0.05% of 21.18 seconds
% cumulative self self total time seconds seconds calls ms/call ms/call name 47.2 9.99 9.99 internal_mcount [5] 36.0 17.61 7.62 5894908 0.00 0.00 partition [4] 11.7 20.08 2.47 70536890 0.00 0.00 swap [6] 2.1 20.52 0.44 1 440.00 10530.00 quicksort [3] 1.6 20.86 0.34 10000000 0.00 0.00 rand [8] 0.8 21.02 0.16 1 160.00 500.00 fillArray [7] 0.8 21.18 0.16 _mcount (665) 0.0 21.18 0.00 24 0.00 0.00 _return_zero [329] 0.0 21.18 0.00 12 0.00 0.00 _mutex_unlock [330] 0.0 21.18 0.00 12 0.00 0.00 mutex_lock [9] 0.0 21.18 0.00 3 0.00 0.00 atexit [10] 0.0 21.18 0.00 3 0.00 0.00 get_mem [11] 0.0 21.18 0.00 2 0.00 0.00 free_mem [12] 0.0 21.18 0.00 1 0.00 0.00 _atexit_init [331]
40
PAPI Interface
Read PAPI documentation athttp://www.ece.uah.edu/~milenka/cpe619-08S/docs/papi.README.ver2.s07.txt
41
Tuning Example
sample1.c – prints the prime numbers up to 50,000
Optimize it using gprof
#include <stdlib.h>#include <stdio.h> int prime (int num); int main() { int i; int colcnt = 0; for (i=2; i <= 50000; i++) if (prime(i)) { colcnt++; if (colcnt%9 == 0) {
printf("%5d\n",i);colcnt = 0;
} else printf("%5d ", i); } putchar('\n'); return 0; }
int prime (int num) { /* check to see if the number is a prime? */ int i; for (i=2; i < num; i++) if (num %i == 0) return 0; return 1; }
42
Tuning Example (cont’d)
Compile it using –pg option
gprof –b ./sample1 Analyze output =>
almost all time is spent in the prime routine
Use gcov to look at the actual number of times each line of the program was executed (hot spots)
#include <stdlib.h>#include <stdio.h> int prime (int num);int main() { int i; int colcnt = 0; for (i=2; i <= 50000; i++) if (prime(i)) { colcnt++; if (colcnt%9 == 0) {
printf("%5d\n",i);colcnt = 0;
} else printf("%5d ", i); } putchar('\n'); return 0; }
int prime (int num) { /* check to see if the number is a prime? */ int i; for (i=2; i < num; i++) if (num %i == 0) return 0; return 1;}
43
Tuning Example (cont’d)
sample2.c – use sqrt to reduce the number of operations in the hot sport
Repeat steps, measure performance
#include <stdlib.h>#include <stdio.h>#include <math.h>
int prime (int num);int faster (int num);
int main() { int i; int colcnt = 0; for (i=2; i <= 50000; i++) if (prime(i)) { colcnt++; if (colcnt%9 == 0) {
printf("%5d\n",i);colcnt = 0; }
else printf("%5d ", i); } putchar('\n'); return 0;}int prime (int num) { /* check to see if the number is a prime? */ int i; for (i=2; i <= faster(num); i++) if (num %i == 0) return 0; return 1;}int faster (int num) { return (int) sqrt( (float) num);}
44
Homework #3
Read chapters 7 (and 8) Read documents in /docs directory
performance.measurements.txt papi.README.ver2.s07.txt
Write a program that prints first N prime number (N should be input from the command line)
Measure execution time using time command Measure execution time using clock() function Measure the number of clock cycles the program take using PAPI Profile the program using gcov and gprof
Due: Monday, February 4, 2008, 12:45 PM Submit by email to instructor with subject
“CPE619-HW3” Name file as: FirstName.SecondName.CPE619.HW3.doc