7 Vital Command Line Tools for Measuring UNIX System Performance

Published on Published onFebruary 18, 2017

Ravi Yogesh

 FollowFollow Ravi YogeshWeb Performance Engineer | Wells Fargo | BITS Pilani5 articles

Like65 Comment9 44If you can't measure it, you can't improve it. -

Peter DruckerUNIX in its different avatars like HP-UX11i, IBM AIX 6, Oracle Solaris etc. and the hugely popular Linux and Mac OS comes equipped with a variety of command line tools which provide highly detailed data on the system health. These tools are designed to be quite dynamic in nature w.r.t. the configurability of output and the breadth and frequency of sample collection. This article is intended to list 7 of the most important and useful command line tools/utilities available in the arsenal of a UNIX System Administrator, discuss their applications, advantages, syntax, arguments (to customize the output as per need) and how to make sense out of the various metrics reported in their output

screen. This understanding should help in a quick deep dive into the system for bottleneck analysis in case of a performance degradation.

1. SAR: The SAR (System Activity Reporter) utility covers a huge number of metrics, ranging from CPU statistics and I/O on hard disks to even CPU temperatures and fan speeds under its umbrella, although the number of metrics covered varies with different flavors of UNIX. SAR is uniquely dynamic in the sense that it can give information about system health in real time as well as from historical records and can be setup to do so for future too. Without any arguments, SAR outputs CPU activity averaged for the current day distributed over system, user, idle and wio (wait i/o), while specifying the delay (d) and limit (n)  outputs the same stats measured every (d) seconds , (n) times; after giving the command. The start and end time for probing can be specified by appending the timestamps with -s and -e arguments respectively. Also, -f argument allows to read the SAR metrics from a file already existing. The generated output can be also written to a separate file for future reference.

                                   {sar -d -f /var/log/sysstat/sa16 -s 13:00:00}

sar -a outputs all the metrics across network, disk, CPU, memory and i/o. 

sar -d yields the disk activity, primary metrics being %busy, avque (average depth of the queue used to serialize disk activity), avwait (average time in ms that a disk read or write operation waits before it is performed) and avserv (average time (in milliseconds) that a disk read or write operation takes to execute) where the last two directly impact end-user experience.

sar -r outputs free memory statistics, sar -S gives swap space statistics while sar -q gives the size of the run queue.

sar-b gives overall i/o stats (tps: transactions per second, breads/s: Bytes read per second etc) while sar -w yields the number of process/context switches. 

sar -n ALL outputs all network activity.

2. TOP: Arguably the most commonly used command which is primarily used to list down the current tasks being serviced by the kernel and to get a broad overview of system health. More specifically, it provides total number of running processes and breakdown by process state, system uptime, load average over the last 5, 10 and 15 minutes, CPU state percentages (split across user, system, iowait, hardware interrupts, software interrupts), RAM and Virtual Memory utilization (total, used, free) and details about each running process like process id (PID), %CPU used and %Memory used by the process, process priority (PR) and process status (S).

                                                        {top -d 5 -n 10}

By default, the output of top is refreshed every 5 seconds, which can be altered by -ddelay attribute and the maximum number of refreshes can be specified by -n limitattribute. HTOP and ATOP are more interactive and customized variants of TOPgiving deeper insights for performance bottleneck analysis.

3. PS: PS (Process Status) command provides a snapshot of the current processes along with detailed information like user id, cpu usage, memory usage and command and is used to monitor the current processes on the machine. It is worth noting that the output of ps is not updated in real time unlike TOP/ATOP command.

                                                     {ps -p -f 1356}

ps -ef gives a full list of all the processes.

ps -r lists only the running processes. ps -f  gives detailed information about the listed processes. ps -C followed by the search term (say xyz) allows us to search the

processes by its name (xyz). ps -u followed by the username (say abc) can be used to get the list

of processes owned by that user (abc). ps -p followed by the process id (pid) can be used to list the

process attached to that pid. ps -ef | grep abcd can be used to flexibly search for processes with

"abcd" in their name. ps -L lists all the threads associated with the processes. 

4. MPSTAT: This command is used to fetch processor related statistics and to diagnose CPU related performance issues. It outputs activities of each available processor, where the first line displays the average since the time of booting and subsequent lines display current values. Refresh rate and number of iterations can be specified in a similar fashion like TOP command. From the displayed metrics, %irq (percentage of time spent by the CPUs to service interrupts),  intr/s (total number of interrupts received per second by the CPUs) and %soft (percentage of time spent by the CPU to service software interrupts), in addition to the already known %user, %nice, %system, %iowait and %idle are worth a mention.Without any option, mpstat will display the Global average activities by all CPUs.

                                                  {mppstat -p ALL 1 4}

mpstat -I prints interrupt statistics per processor. mpstat -P ALL displays statistics about all CPUs one by one. mpstat -A displays processor stats and interrupt stats for all

processors one by one.

5. IOSTAT: This is in itself a part of the SAR utility but can be used independently to get information specific to CPU usage and i/o statistics. Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system. For i/o operations, the metrics reported here include TPS (number of Transfer per second), Blk_read/s (number of blocks read per second), Blk_wrtn/s (blocks write per second), Blk_read (total block reads) and Blk_wrtn (total block writes). Also, one can refer to this awesome take on making sense out of iostat output for bottleneck analysis.

iostat -c displays only the CPU usage statistics. iostat -d, displays only the disk I/O statistics. iostat -n, displays only network statistics. iostat -p sda limits the output to only one device named sda. iostat x y executes iostat command every x seconds, y times. iostat -m changes the default iostat units from blocks to MBs. 

6. VMSTAT: It reports information about processes, memory, paging, block IO, traps, and CPU activity. Like IOSTAT, this is also a part of SAR utility. By default , vmstatdisplays the memory usage (including swap) on the system which is also the most common use for this command.

                                                        {vmstat -t 1 100}

From the default VMSTAT output, the important metrics to monitor are:

Procs: r: The number of processes waiting for run time, b: The number of busy processes.

Memory: swpd: the amount of virtual memory used, free: the amount of idle memory, buff: the amount of memory used as buffers, cache: the amount of memory used as cache.

Swap: si: Memory swapped from disk (for every second), so: Memory swapped to disk (for every second)

IO: bi: Blocks in. i.e blocks received from device (for every second), bo: Blocks out. i.e blocks sent to the device (for every second).

System – in: Interrupts per second, System – cs: Context switches

We have the following arguments to modify the output of VMSTAT as per need:

vmstat -a displays active and inactive memory information.

vmstat -t displays timestamps with each row in output, useful for trcking memory usage with time.

vmstat -S m is used to convert the display unit to MB from kb. vmstat -d displays the disk statistics w.r.t. reads, writes, and I/O

statistics of the disk. vmstat -p sdb1 displays io stats for specific disk partition sdb1. vmstat -s displays the output in a more user-friendly table format.

7. NETSTAT:  Network Statistics is a tool that displays network connections, routing tables, and a number of network protocol statistics. It is used for finding problems in the network and to determine the amount of traffic on the network as a performance measurement. 

                                                       {netstat -an}

The possible values of socket State as listed in the output are as follow:

ESTABLISHED: The socket has an established connection. SYN_SENT: The socket is actively attempting to establish a

connection. SYN_RECV: A connection request has been received from the

network. FIN_WAIT1: The socket is closed, and the connection is shutting

down. FIN_WAIT2: Connection is closed, and the socket is waiting for a

shutdown from the remote end.

TIME_WAIT: The socket is waiting after close to handle packets still in the network.

CLOSE: The socket is not being used. CLOSE_WAIT: The remote end has shut down, waiting for the

socket to close. LAST_ACK: The remote end has shut down, and the socket is

closed. Waiting for acknowledgement. LISTEN: The socket is listening for incoming connections. CLOSING: Both sockets are shut down but we still don't have all

our data sent. UNKNOWN: The state of the socket is unknown. 

netstat takes the following useful arguments as input:

netstat: Displays generic statistics about the network activity of the local system.

netstat -a: Lists all ports , both listening and non-listening. netstat -c: To continuously run netstat command every few

seconds. netstat -an: Shows information about all active connections to the

server, including the source and destination IP addresses and ports. netstat -l: Lists sockets in listening mode. netstat -at/-au : Lists TCP/UDP sockets. netstat -rn: Displays the routing table for all IP addresses bound to

the server. netstat -natp: Displays statistics about active Internet connections. netstat -i: Displays a table of all network interfaces. netstat -s: Displays summary statistics for each protocol.

nestat -st: Displays summary statistics for TCP protocol. netstat -an | grep ':80': Displays which process is using a

particular port (say 80).

References:

http://superuser.com/questions/575202/understanding-top-command-in-unix

http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.htm

http://www.linuxcommand.org/man_pages/mpstat1.htm http://www.thegeekstuff.com/2011/07/iostat-vmstat-mpstat-

examples http://www.computerhope.com/unix/unetstat.html http://www.thegeekstuff.com/2010/03/netstat-command-examples/ http://www.computerhope.com/unix/unetstat.htm