linux monitoring and troubleshooting for dba's
TRANSCRIPT
![Page 1: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/1.jpg)
Linux Monitoring and Troubleshooting
By
MySQL Consulting Team
![Page 2: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/2.jpg)
About MyDBOPS
• MySQL Consulting• MySQL Support• Remote DBA support.• Expert MySQL solutions• MariaDB, Percona, Galera, TokuDB are supported too.
![Page 3: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/3.jpg)
Agenda
• Introduction• Linux Architecture• Devices we monitor• Important tools• Examples
![Page 4: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/4.jpg)
INTRODUCTION
Unix-like computer operating system. The monitoring and maintenance of the system is the most important task. There are many commands line utilities created for this purpose.
CPU l
Network Bandwidth
Memory usage
Disk usage
Syslogs & A
Running processes
etc.
![Page 5: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/5.jpg)
Linux Architecture
![Page 6: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/6.jpg)
Devices we monitor
users
![Page 7: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/7.jpg)
Important tools
netstat
free
Df
Du
ps
history
lsof
top
systat
Sar
iostat
vmstat
Pidstat
mpstat
ping
telnet
ifconfig
w & uptime
who
whoami
![Page 8: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/8.jpg)
Top
Small tool, pre-installed in many unix systems. Display all running and active real time process
in ordered list & updates it regularly. CPU usage, Memory usage, Swap Memory, Cache Size,
Buffer Size, Process PID, User, Commands and much
more shows high memory and cpu utilization of a running
processes.
![Page 9: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/9.jpg)
sysstat
powerful logging and monitoring tool for Linux/Unix systems.
contains utilities to monitor system performance and usage activity.
used to monitor system performance and troubleshoot problems.
sysstat is a go-to for power users and can log and track pretty much everything going on
within
your Linux box.
To install :• redhat based - yum install sysstat• Debian based - apt-get install sysstat
![Page 10: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/10.jpg)
sar
Sar(System Activity Report) is part of the sysstat package.
collects and displays ALL system activities statistics.
can monitor performance of various Linux subsystems (CPU, Memory, I/O..) in real time.
also collect all performance data on an on-going basis, store them, and do historical analysis to
identify bottlenecks.
collected information can be used with ksar to plot graphs.
![Page 11: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/11.jpg)
Sar examples
CPU Usage of ALL CPUs (sar -u) This gives the cumulative real-time CPU usage of all CPUs.
sar -u 1 3 Displays real time CPU usage every 1 second
for 3 times.
“-P ALL” indicates that it should displays statistics for ALL
the individual Cores.
%user - % of cpu utilz. that occurred while executing at user level
%nice – at user level with nice priority.
%system – at sytem level
%iowait - % of time that cpu were idle during which s/m had an
outstanding disk I/O request
%idle – idle & s/m did not have an oustanding disk i/o
%steal - % of time spent in involuntary wait by cpu/cpu’s
![Page 12: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/12.jpg)
Sar examples
• Memory Free and Used (sar -r) This reports the memory statistics.
“1 3 reports for every 1 seconds a total of 3 times. ″
focus on “kbmemfree” and “kbmemused” for free and used memory.
![Page 13: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/13.jpg)
Sar examples• Overall I/O Activities (sar -b)
This reports I/O statistics.
“1 3 reports for every 1 seconds a total of 3 times.″ tps – Transactions per second (this includes both read and write)
rtps – Read transactions per second
wtps – Write transactions per second
bread/s – Bytes read per second
bwrtn/s – Bytes written per second
![Page 14: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/14.jpg)
Sar examples
• Individual Block Device I/O Activities (sar -d)
To identify the activities by the individual block devices
(i.e a specific mount point, or LUN, or partition)
The device name (DEV column) can display the actual
device name, use “sar -d”
tps –no of transfer per second
rd_sec/s – no sector read from device
wr_sec/s – no of sectors write from
device
avgrq-sz – avg size in sectors of the
req that were issued to device.
avgqu-sz – avg queue length of
request
await – avg time for i/o req.
svctm – avg service time for i/o req
%util – % of cputime during which i/o
req where issued
![Page 15: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/15.jpg)
Sar examples
• Reports run queue and load average (sar -q)
This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.
“1 3 reports for every 1 seconds a total of 3 times.″
runq-sz – run queue length (no of task wtng for run time)
plist-sz – no of tasks in the task list
![Page 16: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/16.jpg)
Sar examples
• To view previous records(sar –f)
To view old day records
Followed by path of old files
Maintains one month records
sar -f /var/log/sa/sa15
sar -f /var/log/sa/sa15 -q 1 3
LC_ALL=C sar -f /var/log/sa/sa14 -A
![Page 17: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/17.jpg)
iostat simple tool that will collect and show system
input and output storage device statistics.
often used to trace storage device performance issues including devices, local disks, remote disks such as NFS.
%user - % of cpu utilz. that occurred while executing at user level
%nice – at user level with nice priority.
%system – at sytem level
%iowait - % of time that cpu were idle during which s/m had an
outstanding disk I/O request
%idle – idle & s/m did not have an oustanding disk i/o
%steal - % of time spent in involuntary wait by cpu/cpu’s
blk_read/s – amt of data read from device
blk_wrtn/s – amt of data write to device
Blk_read - total no of block reads
Blk_wrtn - total no of block writes
![Page 18: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/18.jpg)
vmstat
Summary information of Memory, Processes, Paging etc.
used to display statistics of virtual memory, kernerl threads, disks, system processes, I/O
blocks, interrupts, CPU activity and much more.
r: The no of processes waiting for run time. b - The no of processes in uninterruptable sle
bi: Blocks sent to a block device (blocks/s). bo-: Blocks received from a block device (blocks/s).
si – Amount of memory swapped in from disk (kB/s). so – Amount of memory swapped to disk (kB/s).
in: The no of interrupts/s including the clock. cs : The number of context switches per second.
us: user time sy: system time id: idle time
![Page 19: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/19.jpg)
pidstat
pidstat is a part of sysstat utility.
To monitor an individual process which is managed by kernel and generate report.
Can also be used for monitoring the child processes of selected tasks.
reports various statistics including CPU used by a process, disk usage statistics of a process,
statistics for threads associated with selected tasks and child processes.
Display statistics for all running process - pidstat -p ALL
Display statistics for a particular process - pidstat -p pid (process id of application)
Display statistics based on process name - pidstat -C “mysql”Display Statistics of Dependent Process in Tree Format using –t - pidstat -t -C "mysql"
![Page 20: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/20.jpg)
mpstat
mpstat is a part of sysstat utility.
To monitor CPU utilization in our system.
Will be more useful if our system has multiple processor.
Print CPU utilization per processors - mpstat -P 0 , mpstat -P 1
Print all CPU Utilization - mpstat -P ALL
Print CPU utilization using intervals - mpstat 3 4
Print mpstat version - mpstat -V
![Page 21: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/21.jpg)
netstat
monitoring incoming and outgoing network packets statistics as well as interface statistics.
very useful tool for every system administrator to monitor network performance and
troubleshoot network related problems..
![Page 22: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/22.jpg)
free
built-in command that displays the total amount of free and used physical memory on your
machine.
also displays the buffers used by the kernel at that given moment.
![Page 23: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/23.jpg)
df
df is an abbreviation for disk free.
pre-installed program in all unix systems used to display the amount of available disk space for
filesystem which the user have access to.
![Page 24: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/24.jpg)
du
Linux “du” (Disk Usage) is a standard Unix/Linux command.
used to check the information of disk usage of files and
directories on a machine.
has many parameter options that can be used to get the
results in many formats.
also displays the files and directory sizes in a recursively
manner.
![Page 25: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/25.jpg)
ps
basic commands for viewing the processes running on the system
It provides a snapshot of the current processes along with detailed information like user id, cpu
usage, memory usage, command name etc.
It does not display data in real time like top or htop commands.
![Page 26: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/26.jpg)
lsof
lsof meaning 'List Open Files' is used to find out which files are open by which process.
when a disk cannot be un-mounted as it says the files are being used.
![Page 27: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/27.jpg)
lsof some examples
List User Specific Opened Files
lsof -u mydbops
Exclude User with ‘^’ Character lsof -i -u^root
Kill all Activity of Particular User kill -9 `lsof -t -u mydbops`
List all Network Connections
lsof -i
Search by PID
lsof -p 1
![Page 28: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/28.jpg)
ping
used to find out whether the peer host/gateway is reachable.
how much time it takes for that data to be exchanged.
default ping waits for 1 second before sending the next packet. increase or decrease using
option -i.
ping 122.166.212.214
ping -i 5 122.166.212.214
ping -c 3 122.166.212.214
![Page 29: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/29.jpg)
telnet
used for interactive communication with another host using the TELNET protocol.
command mode.
Syntax : telnet host port
![Page 30: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/30.jpg)
w & uptime
w - displays information about the users currently on the machine, and their processes
uptime - Tell how long the system has been running.
![Page 31: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/31.jpg)
who & whoami
who - prints information about all users who are currently logged in.
Displays the username, line, and time of all currently logged-in sessions.
whoami - This command prints the username associated with the current effective user ID.
![Page 32: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/32.jpg)
history
The history command can be used to list Bash's log of the commands you have typed:
The history command performs one of several operations related to recently-executed
commands recorded in a history.
Syntax: history
Eg: history | less
Search the history using Control+R
![Page 33: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/33.jpg)
How MyDBOPS Can Help You
• MySQL Consulting
• MySQL Support
• Remote DBA
• 24 * 7 Support
Contact : [email protected]
![Page 34: Linux monitoring and Troubleshooting for DBA's](https://reader033.vdocument.in/reader033/viewer/2022051101/58e8ef5c1a28ab1f248b4d91/html5/thumbnails/34.jpg)
Thank You