listen to the sounds of your application
DESCRIPTION
There are lots of valuable signals in our systems that we can capture and measure. Aside from monitoring physical resources and latencies, we can bring visibility to business data that we care about. We will present how to create a monitoring and alerting stack which gives a uniform view of those signals. Given such setup we can detect anomalies or system failures or examine how our application usage changes and plan our capacity in advance. During this 30 minute talk we will try to convince you – the Java developer – why you should care about applications metrics and logging from early on in development.TRANSCRIPT
![Page 1: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/1.jpg)
Listen to the sounds of your application
Listen to the sounds of your application
![Page 2: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/2.jpg)
!2
Krzysztof Ciesielski
@kpciesielski
softwaremill.com
Lublin Software Craftsmen
![Page 4: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/4.jpg)
!4
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
Agenda
![Page 5: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/5.jpg)
!5
Monitoring, huh?
![Page 7: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/7.jpg)
!7
Types of measurements
![Page 8: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/8.jpg)
!7
Types of measurements
Network
![Page 9: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/9.jpg)
!7
Types of measurements
Network
Machine
![Page 10: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/10.jpg)
!7
Types of measurements
Network
Machine
Application
![Page 12: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/12.jpg)
!9
measurement > prediction
![Page 13: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/13.jpg)
!10
![Page 14: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/14.jpg)
!11
measurement >> prediction
![Page 15: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/15.jpg)
!12
Our stack
![Page 16: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/16.jpg)
!12
Our stack
Server
App
Server
App
…
![Page 17: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/17.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
![Page 18: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/18.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
![Page 19: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/19.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
![Page 20: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/20.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
![Page 21: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/21.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
![Page 22: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/22.jpg)
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
Yammer Metrics
![Page 23: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/23.jpg)
!13
Introducing Graphite
![Page 24: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/24.jpg)
!14
http://graphite/render?target=server.web1.load&height=800&width=600
![Page 25: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/25.jpg)
!15
Graphite
![Page 26: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/26.jpg)
!16
PORT=2003 SERVER=graphite.your.org echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};
![Page 27: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/27.jpg)
!17
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
![Page 28: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/28.jpg)
!17
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
=> successful.login.attempt = 1
![Page 29: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/29.jpg)
!18
Yammer Metrics
![Page 30: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/30.jpg)
!18
Yammer Metrics
private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
![Page 31: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/31.jpg)
!19
Metrics
![Page 32: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/32.jpg)
!19
MetricsTypes:
![Page 33: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/33.jpg)
!19
MetricsTypes:
•Gauges
![Page 34: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/34.jpg)
!19
MetricsTypes:
•Gauges
•Counters
![Page 35: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/35.jpg)
!19
MetricsTypes:
•Gauges
•Counters
•Meters
![Page 36: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/36.jpg)
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
![Page 37: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/37.jpg)
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
![Page 38: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/38.jpg)
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
![Page 39: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/39.jpg)
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
Reporters:
•STDOUT
•CSV
•SLF4J
•JMX
•Graphite
•Ganglia
![Page 40: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/40.jpg)
!20
Counters, meters… vs.
Gauges
![Page 41: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/41.jpg)
!21
Aggregation
![Page 43: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/43.jpg)
!23
You can also check out• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards: http://shopify.github.io/dashing/ https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk
![Page 44: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/44.jpg)
!24
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
![Page 45: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/45.jpg)
!25
Log aggregation
Less invasive way to build an “event stream”
Logs are data with plenty of value
![Page 46: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/46.jpg)
!26
Log aggregation
Log source
Log source
Log source
Central storage
Query engine
Web view
![Page 47: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/47.jpg)
!27
Logstash
Open Source (written in JRuby)
![Page 48: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/48.jpg)
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
![Page 49: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/49.jpg)
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
![Page 50: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/50.jpg)
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
![Page 51: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/51.jpg)
!27
Logstash
Similar tools: !Splunk, Graylog, Fluentd, log.io
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
![Page 52: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/52.jpg)
!28
Logstash centralized architecture
![Page 53: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/53.jpg)
!29
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
![Page 54: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/54.jpg)
!29
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
time 2013-11-10 19:41:25.321
thread main
loglevel INFO
source o.a.camel.impl.DefaultCamelContext
message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
![Page 55: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/55.jpg)
!30
Logstash simple use cases
Output: e-mail alert
ExceptionsSecurity issuesRare and business-meaningful events
Grok:
![Page 56: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/56.jpg)
!31
Kibana
![Page 57: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/57.jpg)
!32
Kibana checking general app activity
![Page 58: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/58.jpg)
!33
Kibana Spotting anomalies
![Page 59: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/59.jpg)
!34
Kibana Checking regular events
![Page 60: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/60.jpg)
!35
Kibana How often is this feature used?
![Page 61: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/61.jpg)
!36
Kibana Displaying multiple query results
Query: exception Type: applog
Query: index Type: mongolog
![Page 62: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/62.jpg)
!37
References• Graphite http://graphite.readthedocs.org/en/latest/
• Yammer Metrics http://metrics.codahale.com/
• Logstash http://logstash.net/
• Surfing the event stream by Sam Newman at Geecon http://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
![Page 63: Listen to the sounds of your application](https://reader036.vdocument.in/reader036/viewer/2022070303/54b70f7e4a7959a8588b465f/html5/thumbnails/63.jpg)
!38