poznań jug: listening to the sounds of your application
TRANSCRIPT
![Page 1: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/1.jpg)
Listen to the sounds of your application
![Page 3: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/3.jpg)
Originally authored by: Maciej Biłas
Krzysztof Ciesielski
Presented atJ-Day
Lbn.sc
![Page 4: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/4.jpg)
Agenda
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
![Page 5: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/5.jpg)
Monitoring, huh?
![Page 7: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/7.jpg)
Types of measurements
Network
Machine
Application
![Page 9: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/9.jpg)
measurement > prediction
![Page 10: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/10.jpg)
![Page 11: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/11.jpg)
measurement >> prediction
![Page 12: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/12.jpg)
Our stack
Server
App
Server
App
…
Graphite Logstash
Yammer Metrics
![Page 13: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/13.jpg)
![Page 14: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/14.jpg)
http://graphite/render?target=server.web1.load&height=800&width=600
![Page 15: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/15.jpg)
Graphite
![Page 16: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/16.jpg)
echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003
![Page 17: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/17.jpg)
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
successful.login.attempt = 1
![Page 18: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/18.jpg)
Yammer metrics
private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
![Page 19: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/19.jpg)
MetricsTypes:
• Gauges
• Counters
• Meters
• Histograms
• Timers
• Health Checks
Reporters:
• STDOUT
• CSV
• SLF4J
• JMX
• Graphite
• Ganglia
![Page 20: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/20.jpg)
Counters, meters… vs.
Gauges
![Page 21: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/21.jpg)
Aggregationapp00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime
…
![Page 23: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/23.jpg)
You can also check out• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards: http://shopify.github.io/dashing/https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk
![Page 24: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/24.jpg)
Log aggregation?
![Page 25: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/25.jpg)
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
![Page 26: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/26.jpg)
Log aggregation
• Less invasive way to build an “event stream”
• Logs are data with plenty of value
![Page 27: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/27.jpg)
Log aggregation
Log source
Log source
Log source
Central storage
Query engine
Web view
![Page 28: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/28.jpg)
Logstash
• Open source
• Written in JRuby
• Gathers logs from various inputs
• Parses and extracts metadata
• Writes to various outputs
![Page 29: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/29.jpg)
Architecture
![Page 30: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/30.jpg)
Inputscollectd
drupal_dblog
elasticsearch
eventlog
exec
file
ganglia
gelf
gemfire
generator
graphite
heroku
imap
irc
log4j
lumberjack
pipe
rabbitmq
redis
relp
s3
snmptrap
sqlite
sqs
stdin
stomp
syslog
tcp
udp
unix
varnishlog
websocket
wmi
xmpp
zenoss
zeromq
![Page 31: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/31.jpg)
Grok
time 2013-11-10 19:41:25.321
thread main
loglevel INFO
source o.a.camel.impl.DefaultCamelContext
message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
Thin layer on top of a regular expression
![Page 32: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/32.jpg)
Grok
grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
![Page 33: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/33.jpg)
Outputsboundary
circonus
cloudwatch
csv
datadog
datadog_metrics
elasticsearch
elasticsearch_http
elasticsearch_river
exec
file
ganglia
gelf
gemfire
google_bigquery
google_cloud_storage
graphite
graphtastic
hipchat
http
irc
jira
juggernaut
librato
loggly
lumberjack
metriccatcher
mongodb
nagios
nagios_nsca
null
opentsdb
pagerduty
pipe
rabbitmq
redis
riak
riemann
s3
sns
solr_http
sqs
statsd
stdout
stomp
syslog
tcp
udp
websocket
xmpp
zabbix
zeromq
![Page 34: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/34.jpg)
Kibana
![Page 35: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/35.jpg)
Checking app activity
![Page 36: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/36.jpg)
Spotting anomalies
![Page 37: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/37.jpg)
Regular events
![Page 38: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/38.jpg)
How often is a feature used?
![Page 39: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/39.jpg)
Plotting multiple event types
![Page 40: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/40.jpg)
Other tools
• Splunk
• Graylog2
• Fluentd
• log.io
![Page 41: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/41.jpg)
References• Graphite
http://graphite.readthedocs.org/en/latest/
• Yammer Metricshttp://metrics.codahale.com/
• Logstashhttp://logstash.net/
• Surfing the event stream by Sam Newman at Geeconhttp://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
![Page 42: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/42.jpg)
![Page 43: Poznań JUG: Listening to the sounds of your application](https://reader033.vdocument.in/reader033/viewer/2022060107/55496176b4c905f74e8b5ad1/html5/thumbnails/43.jpg)
Thank you!Questions?