CONTAINER MONITORING WITH

SYSDIGPresenter Name: Sreenivas Makam

Presented at: Docker Meetup BangalorePresentation Date: Feb 27, 2016

About me• Senior Engineering Manager at Cisco

Systems Data Center group• Personal blog can be found at

https://sreeninet.wordpress.com/ and my hacky code at https://github.com/smakam

• Author of “Mastering CoreOS” book, published on Feb 2016. (https://sreeninet.wordpress.com/2016/02/27/mastering-coreos-book-got-published/)

• You can reach me on LinkedIn at https://in.linkedin.com/in/sreenivasmakam

Linux Debug tools and Container monitoring

• Strace – trace system calls• Tcpdump, netstat, iftop – monitor network activity• Top, htop – track cpu, memory usage• Lsof – List open files• Iotop – track process ioSince Containers run in their own namespace, it is not straightforward to monitor Containers using these tools.

Container monitoring options1. Install monitoring tools inside Container.

– This defeats purpose of Container and it’s not scalable.2. Install monitoring tool inside the host machine where

Container runs.– Difficult to do this in Container optimized OS like CoreOS,

RancherOS, Atomic3. Install monitoring tool as a Container with system level

privileges. – Preferred option

Sysdig follows a combination of 2 and 3.

Native Container monitoring using Docker tools• Docker stats – cpu, memory, io• Docker top – processes in container• Docker logs – Container logs• Docker events – Container eventsWhat cannot be done using above approach?• Top network connections• Which Containers are talking to each other and which

Containers are talking externally?• Top files being used• System calls made

Sysdig Overview• Sysdig is a monitoring software for bare metal, VM as well as

Containers. • Sysdig documentation calls sysdig as “strace + tcpdump + htop + iftop

+ lsof + ...awesome sauce”• Sysdig monitors kernel system calls to get monitoring visibility• Sysdig integrates with Docker, LXC and Rkt for Container monitoring• Sysdig integrates with Kubernetes and Mesos for visibility into

Container orchestration• Post-monitoring can be done using “.scap” files similar to “.pcap” files

with Wireshark.• Sysdig works mainly in Linux systems. Sysdig for windows can analyze

trace files but not do monitoring.

Sysdig Architecture

• Sysdig-probe is installed as kernel module.• Sysdig does monitoring with minimal kernel and CPU overhead. Reference:https://sysdig.com/interpreting-sysdig-output/

Sysdig Container Architecture

• Sysdig can be installed as a Container or as a binary in the host Linux system

Reference:https://sysdig.com/let-light-sysdig-adds-container-visibility/

Sysdig software

• Sysdig CLI – Open source CLI tool.• csysdig - Open source Text based ncurses

interface on top of Sysdig.• Sysdig cloud – Commercial product – Available for 14 day free trial.– Combines Sysdig output from multiple hosts to a

central Sysdig cloud server– Can be installed on-premise

Sysdig format• Incremental event number• Event timestamp – customize this with the -t command line flag (more info)• CPU ID• Command name• Thread ID• Event direction – ‘>’ means ‘process input’, while ‘<’ means ‘process output’• Event type• Event argumentsEg:90772 21:19:18.249796600 0 nginx (3212) < accept fd=3(<4t>172.19.0.4:35831->172.19.0.2:http) tuple=172.19.0.4:35831->172.19.0.2:http queuepct=0 queuelen=0 queuemax=12890780 21:19:18.249846551 0 nginx (3212) < open fd=11(<f>/usr/share/nginx/html/index.html) name=/usr/share/nginx/html/index.html flags=65(O_NONBLOCK|O_RDONLY) mode=0

Sysdig examples• sysdig -pc -c topprocs_cpu – List top processes by CPU usage• sysdig -pc -c topprocs_net - List top processes by network usage• sysdig -pc -c topprocs_file - List top processes by io usage• sysdig -pc -c spy_users – List all commands executed by user• sysdig -qw dumpfile.scap – Dump all system transactions into dumpfile.scap

tracefile • sysdig -r dumpfile.scap -c echo_fds container.name=haproxy – Read trace file

and filter output by file io and Container name• sysdig -pc -A -c echo_fds container.name=haproxy – List all file activity by

Container “haproxy” in ascii format• sysdig -l -> list filters• sysdig -cl -> list chisels• Csysdig –pc -> Start csysdig with Container visibility

Sysdig Kubernetes Integration• By integrating with Kubernetes, Sysdig becomes aware of

Kubernetes constructs like Namespaces, Replication controllers, Pods and Services.

• Sysdig becomes aware of Kubernetes constructs by getting details from Kubernetes API server.

• By grouping monitoring data at Kubernetes construct, user gets better visibility into the resource usage as a collection.

• Sysdig cloud has better integration with Kubernetes than Sysdig since monitoring data at cluster level is possible only with Sysdig cloud.

Demo-1(Video - https://www.youtube.com/watch?v=otiHinxObE4)

NetworkFE

NetworkBE

Ubuntu haproxy Nginx1 nginx2 nginx3

docker network create bedocker network create fe

docker run --name nginx1 --net be -v ~/haproxy/nginx1.html:/usr/share/nginx/html/index.html -d nginxdocker run --name nginx2 --net be -v ~/haproxy/nginx2.html:/usr/share/nginx/html/index.html -d nginxdocker run --name nginx3 --net be -v ~/haproxy/nginx3.html:/usr/share/nginx/html/index.html -d nginx

docker run -d --name haproxy --net be -v ~/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy

docker network connect fe haproxy

docker run -it --rm --net fe --name ubuntu myubuntu bash

Configurations present at: https://github.com/smakam/docker/tree/master/haproxy

Demo – 2(Guestbook Video - https://www.youtube.com/watch?v=oQw_2ZNpMd0)

Front end RC Redis master RC Redis slave RC

P P P P P P

Frontend service

RedisM service

RedisS service

Service

ReplicationController

Pods

./cluster/kubectl.sh create -f examples/guestbook/redis-master-controller.yaml

./cluster/kubectl.sh create -f examples/guestbook/redis-master-service.yaml

./cluster/kubectl.sh create -f examples/guestbook/redis-slave-controller.yaml

./cluster/kubectl.sh create -f examples/guestbook/redis-slave-service.yaml

./cluster/kubectl.sh create -f examples/guestbook/frontend-controller.yaml

./cluster/kubectl.sh create -f examples/guestbook/frontend-service.yaml

References• Sysdig install (http://www.sysdig.org/install/)• Interpreting sysdig (https://sysdig.com/interpreting-sysdig-output/)• Sysdig Internals (

https://sysdig.com/sysdig-vs-dtrace-vs-strace-a-technical-discussion/) • Sysdig for Containers (https://sysdig.com/let-light-sysdig-adds-container-visibility/) • csysdig manpage (http://man7.org/linux/man-pages/man8/csysdig.8.html)• Sysdig with Kubernetes (https://sysdig.com/digging-into-kubernetes-with-sysdig/) • Sysdig with Mesos, Marathon (http://

support.sysdigcloud.com/hc/en-us/articles/207886103-Sysdig-Cloud-Agent-Mesos-Marathon )

• Sysdig with Rkt (https://sysdig.com/monitoring-rkt-sysdig/ )• Sysdig with CoreOS (

https://sysdig.com/coreos-sysdig-part-1-digging-into-coreos-environments/)

QUESTIONS?

Setting up Sysdig cloud

• To try it out, I got a 14 day free trial account from Sysdig website.

• Install Sysdig cloud agent on each node by using the command specified in Sysdig cloud settings tab.

• For Kubernetes integration, Sysdig cloud needs to be installed in both Kubernetes master and slave nodes.

Setting up Kubernetes clusterFor installing Kubernetes cluster on AWS, I followed these steps after downloading Kubernetes.export KUBERNETES_PROVIDER=awsexport NUM_MINIONS=2export MASTER_SIZE=t2.microexport MINION_SIZE=t2.microexport KUBE_OS_DISTRIBUTION=trusty./cluster/kube-up.sh

Note:• I hit this issue with Kubernetes 1.1.7(http://

stackoverflow.com/questions/34993716/failed-to-run-install-fedora-deps-when-starting-up-local-kubernetes-cluster). I solved it by using the workaround mentioned in the link.

• To access Guestbook application externally, I used “Nodeport” based load balancer and opened up the specified port in AWS Security group on the slave nodes.

• To login to Kubernetes AWS nodes, ssh as “ubuntu” user with public key under .ssh/kube_aws_rsa

• Setting up Kubernetes cluster on Vagrant has a problem with Sysdig cloud since Sysdig cloud seems to get confused with multiple nodes residing behind a firewall and it shows up as a single node.