poznań jug: listening to the sounds of your application

Listen to the sounds of your application

!2

Maciej Biłas @maciejb softwaremill.com jbison.com

Originally authored by: Maciej Biłas

Krzysztof Ciesielski

Presented atJ-Day

Lbn.sc

Agenda

• Monitoring, huh?

• Introducing Graphite

• Log analysis – the whys

• Logstash architecture & use cases

• Exploring logs with Kibana

Monitoring, huh?

source: codeascraft.com

Types of measurements

Network

Machine

Application

source: codeascraft.com

measurement > prediction

measurement >> prediction

Our stack

Server

App

Server

App

…

Graphite Logstash

Yammer Metrics

http://graphite/render?target=server.web1.load&height=800&width=600

Graphite

echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003

successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297

successful.login.attempt = 1

Yammer metrics

private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }

MetricsTypes:

• Gauges

• Counters

• Meters

• Histograms

• Timers

• Health Checks

Reporters:

• STDOUT

• CSV

• SLF4J

• JMX

• Graphite

• Ganglia

Counters, meters… vs.

Gauges

Aggregationapp00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime

…

source: codeascraft.com

You can also check out• collectd https://collectd.org

• StatsD https://github.com/etsy/statsd/

• Riemman http://riemann.io/

• Twitter’s Ostrich https://github.com/twitter/ostrich

• Ganglia http://ganglia.sourceforge.net/

• Dashboards: http://shopify.github.io/dashing/https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk

Log aggregation?

Event Sourcing

Tracing and storing all the events

Current app state: replaying the stream

Great for data mining and analysis

Log aggregation

• Less invasive way to build an “event stream”

• Logs are data with plenty of value

Log aggregation

Log source

Log source

Log source

Central storage

Query engine

Web view

Logstash

• Open source

• Written in JRuby

• Gathers logs from various inputs

• Parses and extracts metadata

• Writes to various outputs

Architecture

Inputscollectd

drupal_dblog

elasticsearch

eventlog

exec

file

ganglia

gelf

gemfire

generator

graphite

heroku

imap

irc

log4j

lumberjack

pipe

rabbitmq

redis

relp

s3

snmptrap

sqlite

sqs

stdin

stomp

syslog

tcp

twitter

udp

unix

varnishlog

websocket

wmi

xmpp

zenoss

zeromq

Grok

time 2013-11-10 19:41:25.321

thread main

loglevel INFO

source o.a.camel.impl.DefaultCamelContext

message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

Thin layer on top of a regular expression

Grok

grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

Outputsboundary

circonus

cloudwatch

csv

datadog

datadog_metrics

elasticsearch

elasticsearch_http

elasticsearch_river

email

exec

file

ganglia

gelf

gemfire

google_bigquery

google_cloud_storage

graphite

graphtastic

hipchat

http

irc

jira

juggernaut

librato

loggly

lumberjack

metriccatcher

mongodb

nagios

nagios_nsca

null

opentsdb

pagerduty

pipe

rabbitmq

redis

riak

riemann

s3

sns

solr_http

sqs

statsd

stdout

stomp

syslog

tcp

udp

websocket

xmpp

zabbix

zeromq

Kibana

Checking app activity

Spotting anomalies

Regular events

How often is a feature used?

Plotting multiple event types

Other tools

• Splunk

• Graylog2

• Fluentd

• log.io

References• Graphite

http://graphite.readthedocs.org/en/latest/

• Yammer Metricshttp://metrics.codahale.com/

• Logstashhttp://logstash.net/

• Surfing the event stream by Sam Newman at Geeconhttp://www.slideshare.net/spnewman/surfing-the-event-stream

• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack

• Code as Craft http://codeascraft.com/

Thank you!Questions?