Using netstat for Performance Analysis

Zack Hoffmann

Network vs. Individual System Performance

SystemArrivals Completions

This system could contain a number of

networked subsystems!

System A

System B

System C

A switching or broadcasting device

Aa

Ab

Ac

Ca

Cb

Cc

CnAn

Now, in addition to worrying aboutindividual system performance, we have

to worry about network performance!

Here we have a basic system containing networked subsystems.

Tracking Network Traffic

• In individual systems we track the progress of jobs, from their arrival to their completion, to determine performance.

• In a network we can not do this. Entire jobs are not moved in and out of systems on a network.

• Instead, packets of information pass through those systems. We can monitor these packets to analyze performance.

Introducing netstat

About netstat

• Netstat has options for viewing your network data in many different formats.– Some used to measure throughput.– Others used to diagnose network problems.– Many may require a deeper level of network

hardware knowledge than is applicable to this course.

netstat -i 1

What we’re looking for hereare errors (errs) from either

output or input.

Shows incoming and outgoing packet counts and sizes every second.

You won’t have to worryabout collisions unless you’reusing a hub-based network, but if you are then they can

be significant.

netstat -i

Can be run without time interval for a more comprehensive history (since last system boot).

Useful benchmarks for analysis using netstat –i:

•(Coll+Ierrs+Oerrs)/(Ipkts+Opkts) > 2%

•This could indicate problems in your network hardware.

•(Coll/Opkts) > 10%

•Your hub network should be segmented with additional switches.

•(Ierrs/Ipkts) > 25%

•Your system or network may be overloaded.

netstat -a

Lists all active connections

But as you can imagine,our fang machine has a ton of

them, so I’ll specify –pTCPto limit it to TCP connections.

This indicates that thereceiving system can not getthese packets fast enough.This is a problem with the

network (either yours or theirs)and NOT your system.

(However, sending queues withsizes this small are typical.)

Large, ongoing spikes here would indicate abottleneck on your system, as this queue is

for packets to be used by a job on this machine.

Keep an eye here, too.If a connection gets hung upin a waiting state, it may be a

sign of problems in yourprotocol handling.

Deeper Analysis

• netstat -s– Historical statistics, by protocol.– Diagnose protocol handling/routing issues.

• spray– Separate tool, sends bursts of meaningless packets.– Great for determining levels of congestion.

• Send absurd amount of packets, see how many get dropped.

– Same technique can be used between two networked machines to determine if an issue is the product of congestion or due to a slow receiving machine.

Measuring Individual Throughput with netstat

• You could use netstat -i 1, pick an interval of time, and see how much data gets pushed out in that interval.

• However, packet input and output are not in a 1-to-1 correspondence like job acceptance and completion.

• If a remote system is uploading a large file to your system, your packets in will be vastly greater than your packets out!

Using “Goodput” Instead

• Goodput is simply throughput which takes in to account protocol overhead and retransmitted packets.

• It measures only bits useful on the application level.

• If you know how much data from each packet is NOT overhead, this is calculated the same as throughput by simply multiplying by the percentage of non-overhead bits afterwards.

• Must also remember to exclude retransmitted packets (netstat -s provides this information).

So What’s the Point?

• When you’re trying to estimate performance on a network level, you have to consider factors OUTSIDE of your system.

• netstat can tell you whether or not your poor performance is caused by network elements, and can give you some indication of how much it may be slowing you down.

System Performance System Performance using using

vmstatvmstat

Adithya A GuruswamyAdithya A Guruswamy

SynopsisSynopsis

Analyzing Performance .Analyzing Performance . Why Analyze Performance ?Why Analyze Performance ? Performance Analysis Commands.Performance Analysis Commands. Virtual Memory.Virtual Memory. Vmstat.Vmstat. Determining CPU usage with Determining CPU usage with vmstat.vmstat. Graphical Representation of Graphical Representation of vmstat.vmstat. Bibliography.Bibliography.

Analyzing PerformanceAnalyzing Performance

Performance Analysis is the Process of Performance Analysis is the Process of analyzing performance bottlenecks in the analyzing performance bottlenecks in the system.system.

It Involves a number of steps :It Involves a number of steps :1.1. Identify the problem.Identify the problem.2.2. Is the problem CPU oriented or I/O related ?Is the problem CPU oriented or I/O related ?3.3. CPU Problem - What is the Load Average ?CPU Problem - What is the Load Average ?4.4. I/O problem – is it paging or normal disk I/O?I/O problem – is it paging or normal disk I/O?

Why AnaWhy Anallyze yze PerformancePerformance??

To eliminate bottlenecks.To eliminate bottlenecks.To reduce the Resident Time.To reduce the Resident Time.To increase the Throughput.To increase the Throughput. Increase the overall efficiency of the Increase the overall efficiency of the

system.system.

Performance Analysis Commands Performance Analysis Commands

vmstatvmstat : : VVIRTUAL IRTUAL MMEMORY EMORY STATSTATISTICS.ISTICS.

netstatnetstat : : NETNETWORK WORK STATSTATISTICS.ISTICS.

iostatiostat : : IINPUT/NPUT/OOUTPUT UTPUT STATSTATISTICSISTICS

Virtual MemoryVirtual Memory

Memory, often as simulated on a hard Memory, often as simulated on a hard disk, that emulates RAM, allowing an disk, that emulates RAM, allowing an application to operate as though the application to operate as though the computer has more memory than it computer has more memory than it actually does.actually does.

Computer HierarchyComputer Hierarchy

                                                                                      

                                                                                                                                                          

vmstatvmstat

vmstatvmstat ( (VVirtual irtual MMemory emory STATSTATistics) is a istics) is a computer computer operating systemoperating system monitoring tool that monitoring tool that collects and displays summary information about collects and displays summary information about memory, processes, interrupts, paging memory, processes, interrupts, paging andand blockblock I/OI/O information. information.

Also Also vmstatvmstat command reports statistics about command reports statistics about kernel threads, virtual memory, disks, traps kernel threads, virtual memory, disks, traps andand CPU activity. CPU activity.

Reports generated by the Reports generated by the vmstatvmstat command can command can be used to balance system load activity.be used to balance system load activity.

vmstatvmstatSyntaxSyntax : : vmstatvmstat [ [-a-a] [] [-n-n] [] [delaydelay [ [ countcount]] ]] vmstatvmstat [ [-f-f] [] [-s-s] [] [-m-m] ]

vmstatvmstat [ [-S unit]-S unit] vmstatvmstat [ [-d]-d] vmstatvmstat [ [-p disk -p disk partition]partition] vmstatvmstat [ [-V-V] ]

-f -f Reports the number of Reports the number of forksforks since system startup.since system startup.

-i -i Displays the number of Displays the number of interruptsinterrupts taken by each taken by each device since system startup.device since system startup.

-s-s Display the contents of the Display the contents of the sumsum structure, structure, giving the total number of several kinds of giving the total number of several kinds of paging related events which have occurred paging related events which have occurred since system startup. since system startup.

vmstatvmstat

-a-a switch displays switch displays active/inactiveactive/inactive memory . memory . -m-m displays slab info. displays slab info. -n-n switch causes the header to be displayed only switch causes the header to be displayed only

once rather than periodically. once rather than periodically. -d-d reports reports diskdisk statistics . statistics . -p-p followed by some followed by some partitionpartition name for detailed name for detailed

statistics.statistics. -V-V switch results in displaying switch results in displaying versionversion

information. information.

DETERMINING CPU USAGE WITH VMSTATDETERMINING CPU USAGE WITH VMSTAT

vmstat vmstat output output

DescriptionDescription

Procs Procs r: The number of processes waiting for r: The number of processes waiting for run run time. time. b: The number of processes in b: The number of processes in uninterruptibleuninterruptible sleep. sleep. MemoryMemory swpd: swpd: The amount of virtual memory used.The amount of virtual memory used. free: free: The amount of The amount of idleidle memory. memory. buff: buff: The amount of memory used as The amount of memory used as buffersbuffers.. cache: cache: The amount of memory used as The amount of memory used as cachecache.. inact: inact: The amount of The amount of inactiveinactive memory. (-a option) memory. (-a option) active: active: The amount of The amount of activeactive memory. (-a option) memory. (-a option)

DescriptionDescription

SwapSwap si: Amount of memory si: Amount of memory swapped inswapped in from disk (/s). from disk (/s). so: Amount of memory so: Amount of memory swapped toswapped to disk (/s). disk (/s).

IOIO bi: Blocks received from a bi: Blocks received from a block block device (blocks/s). device (blocks/s). bo: Blocks sent to a bo: Blocks sent to a blockblock device (blocks/s). device (blocks/s).

SystemSystem in: The number of in: The number of interruptsinterrupts per second. per second.

cs: The number of cs: The number of context switchescontext switches per second. per second.

DescriptionDescription

CPU CPU These are percentages of total CPU time. These are percentages of total CPU time.

us: Time spent running us: Time spent running non-kernelnon-kernel code. code.

sy: Time spent running sy: Time spent running kernelkernel code. code.

id: Time spent id: Time spent idleidle. .

wa: Time spent wa: Time spent waitingwaiting for io. for io.

st: Time st: Time stolenstolen from a virtual machine. from a virtual machine.

Graphical RepresentationGraphical Representationvmstatvmstat – by day – by day

Graphical RepresentationGraphical Representation

vmstatvmstat – by day – by day

Graphical RepresentationGraphical Representationvmstatvmstat - by week - by week

BibliographyBibliography

References obtained from :References obtained from :http://en.wikipedia.org/wiki/Main_Pagehttp://en.wikipedia.org/wiki/Main_Pagehttp://www.redhat.comhttp://www.redhat.comhttp://www.freebsd.orghttp://www.freebsd.orghttp://www.linuxcommand.org/man_pages/http://www.linuxcommand.org/man_pages/

vmstat8.htmlvmstat8.htmlhttp://www.linuxjournal.comhttp://www.linuxjournal.com

THE ENDTHE END

THANK YOUTHANK YOU

PERFORMANCE ANALYSISusing

IOSTAT

Chaitali Barve

What is Performance Analysis?

• Whenever CPU is occupied by a process, it is unavailable for processing other requests. This becomes a bottleneck in the system.

• To troubleshoot CPU performance finding CPU utilization is one of the important tasks.

iostat

• ‘iostat’ reports CPU statistics and Input/Output statistics for devices and partitions.

• The reports generated can be used to change system configuration to better balance the input/output load between physical disks.

• It can be used to find system’s average CPU utilization since the last reboot.

Syntax for ‘iostat’• Syntax is iostat <options> interval count

• options - Lets you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t  or -tdc ). x options gives the extended statistics.

• Interval - It is time period in seconds between two samples. ‘iostat 4’ will give data at each 4 seconds interval.

• count  - It is the number of times the data is needed. ‘iostat 4 5’   will give data at 4 seconds interval 5 times.

Analyzing CPU

Percentage of CPU utilization

Executing at the user level

with nice priority.

Executing at the system level (kernel).

Percentage of time that the CPU or CPUs

were idle during whichthe system had

outstanding request

Percentage of time that the CPU or CPUs

were idle during whichthe system did not have

outstanding request

Report

• The report generated by the ‘iostat’ command is the CPU Utilization Report.

• The report consists of a ‘tty’ and CPU header row followed by a row of ‘tty’ and CPU statistics.

• Also, the I/O wait state is defined system-wide and not per processor.

Analyzing Disk

number of transfers

per second

amount of data read from the drive

amount of data written to the drive

total number of blocks read

total number of blocks written

Report

• The second report generated by the ‘iostat’ is the Disk Utilization Report.

• The device report provides statistics on a per physical device or partition basis.

Iostat Using Time

Print the time for each report displayed

Report

• The first report contains statistics for the time since system start up (boot).

• Each subsequent report contains statistics collected during the interval since the previous report.

• Command used to get 2 reports for disk utilization in interval of 2 seconds :-

iostat –d 2 2

Putting it Together

• ‘iostat’ is basically used for CPU statistics & input/output statistics for devices and partitions.

• ‘iostat’ command can be specifically used to get reports either for CPU or for Input/Output.

• The time at which a certain was generated can also be captured.

• Thus, ‘iostat’ helps us to find out which process is using how much CPU or Input/Output. Likewise system configuration steps can be ammended to make necessary changes for CPU and Input/Output usage