![Page 1: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/1.jpg)
ELKTake care of your logs with
@MattKetmo – IPC Berlin 2015
![Page 2: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/2.jpg)
Matthieu Moquet@MattKetmo
web engineer at
![Page 3: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/3.jpg)
Logs
![Page 4: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/4.jpg)
Why logs?Debug
Understand code Execution workflow
Metrics Monitoring
![Page 5: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/5.jpg)
Which logs?
![Page 6: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/6.jpg)
Syslog
![Page 7: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/7.jpg)
Access Logs
![Page 8: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/8.jpg)
Application Logs
![Page 9: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/9.jpg)
How to view logs?
![Page 10: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/10.jpg)
ssh example.org tail -‐f /var/log/some.log
![Page 11: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/11.jpg)
multitail colortail rainbow
grc …
![Page 12: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/12.jpg)
You should stop doing this!
![Page 13: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/13.jpg)
! Not easily readable ! No aggregations ! Bad performances on concurrent processes
![Page 14: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/14.jpg)
Dataviz > Log messages
![Page 15: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/15.jpg)
Send your logs
AWAY
![Page 16: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/16.jpg)
Open Source
![Page 17: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/17.jpg)
ELK
![Page 18: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/18.jpg)
Elasticsearch Logstash Kibana
![Page 19: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/19.jpg)
3 independent softwaresFlexible
Performant Scalable
![Page 20: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/20.jpg)
« Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents »
— Wikipedia
elasticsearch
![Page 21: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/21.jpg)
elasticsearch
Aggregations
Real-Time Data
Distributed
RESTful API
Schema-Free
High Availability
![Page 22: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/22.jpg)
elasticsearch
Download at https://www.elastic.co/downloads/elasticsearch
bin/elasticsearch
![Page 23: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/23.jpg)
elasticsearch
See it running…
curl -‐X GET http://localhost:9200/
![Page 24: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/24.jpg)
elasticsearch
Index a document
curl -‐XPUT http://localhost:9200/users/user/mattketmo -‐d '{ "name": "Matthieu Moquet", "age": 26}'
![Page 25: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/25.jpg)
elasticsearch
Get a document
curl -‐XGET http://localhost:9200/users/user/mattketmo
![Page 26: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/26.jpg)
elasticsearchUse aggregations
curl -‐XPOST "http://localhost:9200/users/user/_search" -‐d' { "size": 0, "aggregations": { "age_avg": { "avg": { "field": "age" } } } }'
![Page 27: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/27.jpg)
Logstash
Log Processing / ETL
Input
Output
Filters
![Page 28: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/28.jpg)
echo "[ALERT] Some error message" | sed -‐e "s/^\[\(.*\)\] \(.*\)/\1,\2/" > output.csv !
cat output.csv ALERT,Some error message
Logstash
![Page 29: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/29.jpg)
Logstash
gelf syslog
nginx logs varnish logs
udp …
date dns
geoip grok
urldecode …
elasticsearch redis
graphite nagios zabbix
…
Inputs Filters Outputs
![Page 30: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/30.jpg)
LogstashDownload at https://www.elastic.co/downloads/logstash
bin/logstash -‐e ' input { stdin {} } output { stdout {} }'
![Page 31: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/31.jpg)
LogstashDownload at https://www.elastic.co/downloads/logstash
bin/logstash -‐f /path/to/logstash.conf
![Page 32: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/32.jpg)
Logstashinput { ... } filter { ... } output { elasticsearch_http { host => "elasticsearch.tld" port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }
![Page 33: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/33.jpg)
Curator
https://github.com/elastic/curator
Allow you to remove old indexes (via a CRON)
curator -‐-‐host 10.0.0.2 delete indices \ -‐-‐older-‐than 30 -‐-‐time-‐unit days \ -‐-‐timestring '%Y.%m.%d'
![Page 34: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/34.jpg)
Heka
« Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing. »
http://hekad.readthedocs.org
![Page 35: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/35.jpg)
«!Fluentd is an open source data collector for unified logging layer!»
![Page 36: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/36.jpg)
Kibana
! Data visualization web app !Many graphes (histogram, pie chart, geo map, …) ! Built in HTML / CSS / Javascript
![Page 37: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/37.jpg)
KibanaDownload at https://www.elastic.co/downloads/kibana
bin/kibana
![Page 38: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/38.jpg)
upstream es_backend { server 127.0.0.1:9200; keepalive 64; } !server { listen 80; server_name kibana.tld; root /var/www/kibana; try_files $uri $uri/ index.html @elasticsearch; ! location @elasticsearch { proxy_pass http://es_backend; proxy_read_timeout 90; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass_header Access-‐Control-‐Allow-‐Origin; proxy_pass_header Access-‐Control-‐Allow-‐Methods; proxy_hide_header Access-‐Control-‐Allow-‐Headers; add_header Access-‐Control-‐Allow-‐Headers 'X-‐Requested-‐With, Content-‐Type'; add_header Access-‐Control-‐Allow-‐Credentials true; } }
Kibana 3 & nginx
![Page 39: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/39.jpg)
![Page 40: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/40.jpg)
Basic pipeline
Indexer
![Page 41: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/41.jpg)
Our current pipeline
Shipper + Indexer
Broker
![Page 42: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/42.jpg)
Kafka pipeline
1M+ messages per second
Logstash >= 1.5.0
![Page 43: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/43.jpg)
How to forward logs?
![Page 44: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/44.jpg)
Forwarding Syslog
![Page 45: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/45.jpg)
RSYSLOGThe rocket-fast system for log processing
*.* @127.0.0.1:514;RYSLOG_ForwardFormat
![Page 46: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/46.jpg)
Logstash input
input { udp { port => 514 type => syslog } }
![Page 47: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/47.jpg)
Logstash filter
filter { if [type] == "syslog" { grok { match => ["message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREENDYDATA:syslog_message}"] add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] add_tag => ["rsyslog"] } } }
![Page 48: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/48.jpg)
Forwarding Access Log
![Page 49: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/49.jpg)
logstash-forwarderA tool to collect logs locally in preparation for processing elsewhere
https://github.com/elastic/logstash-forwarder
logstash-‐forwarder -‐config conf.json
![Page 50: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/50.jpg)
logstash-forwarder... { "paths": [ "/var/log/nginx/access.log" ], "fields": { "type": "nginx-‐access" } } ...
![Page 51: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/51.jpg)
logtailSmall Perl script to read HTTP access logs
and send it directly to Redis !
https://github.com/shtouff/logtail
![Page 52: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/52.jpg)
Forwarding Application Logs
![Page 53: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/53.jpg)
Monologhttps://github.com/Seldaek/monolog
![Page 54: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/54.jpg)
PSR-3php-fig.org/psr/psr-3
github.com/php-fig/log
![Page 55: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/55.jpg)
<?php !namespace Psr\Log; !interface LoggerInterface { public function log($level, $message, array $context = array()); ! // Shortcuts public function emergency($message, array $context = array()); public function alert($message, array $context = array()); public function critical($message, array $context = array()); public function error($message, array $context = array()); public function warning($message, array $context = array()); public function notice($message, array $context = array()); public function info($message, array $context = array()); public function debug($message, array $context = array()); }
![Page 56: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/56.jpg)
<?php !namespace Psr\Log; !class LogLevel { const EMERGENCY = 'emergency'; const ALERT = 'alert'; const CRITICAL = 'critical'; const ERROR = 'error'; const WARNING = 'warning'; const NOTICE = 'notice'; const INFO = 'info'; const DEBUG = 'debug'; }
![Page 57: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/57.jpg)
Usage: PHPps4,4.!-(%.B6#6&6EUP6EE%+d.-(%.B6#6&6EUm'#>&%+UI)+%'<m'#>&%+d.!$$.j+%')%.'.&6E.*,'##%&.h,'#>&%+.\.#%g.I)+%'<m'#>&%+V?4'),$)6$b6-+W&6E?D.P6EE%+55nOQqkq1Yd.h&6E.\.#%g.P6EE%+V?#'<%?Yd.h&6E/[4-(,m'#>&%+Vh,'#>&%+Yd.!$$.O>>.+%*6+>(.)6.),%.&6E.h&6E/[g'+#"#EV?l66?Yd.h&6E/[%++6+V?u'+?Yd
![Page 58: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/58.jpg)
Usage: Symfonymonolog: handlers: main: type: fingers_crossed action_level: warning handler: file file: type: stream level: debug path: /var/log/symfony.log syslog: type: syslog level: error
![Page 59: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/59.jpg)
Usage: Symfony Controllerps4,4.!#'<%(4'*%.O*<%Uu-#>&%UO44u-#>&%Uj6#)+6&&%+d.!-(%.Ib<M6#bUu-#>&%Ul+'<%g6+=u-#>&%Uj6#)+6&&%+Uj6#)+6&&%+d.!*&'((.l66j6#)+6&&%+.%])%#>(.j6#)+6&&%[email protected]!&"*.M-#*)"6#.!'+O*)"6#[email protected]),"(/[E%)V?&6EE%+?Y/["#M6V?2]%*-)"#E.M66.!'+?Yd.!........+%)-+#.h),"(/[+%#>%+V?iO44u-#>&%$l66$!'+W,)<&W)g"E?Yd.....G.G
![Page 60: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/60.jpg)
Usage: Symfony Service
-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%[email protected]+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)Vl66.hM66D.P6EE%+k#)%+M'*%.h&6EE%+.\.#-&&[email protected]),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.h&6EE%+.s5.#%g.q-&&P6EE%+VYd.....G.G
Avoid "M.V#-&&.v\\.h&6EE%+Y
![Page 61: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/61.jpg)
Usage: Symfony Service-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%[email protected]+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)[email protected]),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.#%g.q-&&P6EE%+VYd.....G.!....4-!&"*.M-#*)"6#.(%)P6EE%+VP6EE%+k#)%+M'*%.h&6EE%[email protected]),"(/[&6EE%+.\.h&6EE%+d.....G.G
![Page 62: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/62.jpg)
40+ HandlersAmqpHandler
ErrorLogHandler FirePHPHandler
GelfHandler MailHandler
NewRelicHandler NullHandler
RedisHandler StreamHandler
SyslogUdpHandler ...
![Page 63: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/63.jpg)
SyslogHandler
<6#6&6E5...,'#>&%+(5......(b(&6E5.........)b4%5.(b(&6E.........&%L%&5.g'+#"#E
Ops problem now!
![Page 64: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/64.jpg)
GelfHandlermonolog: handlers: main: type: fingers_crossed action_level: warning handler: file gelf: type: gelf level: notice publisher: hostname: %logstash_host%
![Page 65: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/65.jpg)
Logstash input
input { gelf { port => 12201 type => gelf } }
![Page 66: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/66.jpg)
Logstash filter
filter {}
![Page 67: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/67.jpg)
Logstash output
output { elasticsearch { host => ["127.0.0.1"] port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }
![Page 68: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/68.jpg)
Log
{ "message": "Lorem ipsum", "level": "200", "level_name": "info", "@timestamp": "1432825193000" }
![Page 69: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/69.jpg)
![Page 70: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/70.jpg)
Wait! There is more…
![Page 71: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/71.jpg)
ChannelsUseful to group logs by category
Make it easy to filter Use different rules / handlers
![Page 72: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/72.jpg)
Channels
- assetic - doctrine - event - php - (php_error) - profiler - request - …
Symfony is shipped with many default channels
![Page 73: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/73.jpg)
Usage: Symfony
monolog: channels: -‐ api -‐ business_domain -‐ super_feature -‐ rabbitmq
Create as many as you want
![Page 74: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/74.jpg)
Usage: Symfony
monolog: handlers: main: type: stream path: /var/log/symfony.log channels: !doctrine doctrine: type: stream path: /var/log/doctrine.log channels: doctrine
![Page 75: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/75.jpg)
Usage: Symfony
monolog: handlers: default_notice: type: gelf level: NOTICE channels: [request, security] default_info: type: gelf level: INFO channels: [!request, !security]
![Page 76: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/76.jpg)
Usage: Symfony
services: my_service: class: Acme\Class\Name arguments: ["@logger"] tags: -‐ name: monolog.logger channel: acme
![Page 77: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/77.jpg)
Channels
Visualize proportions Easy filtering
![Page 78: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/78.jpg)
ContextMetadata for your logs
![Page 79: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/79.jpg)
Context
interface LoggerInterface { public function log( $level, $message, array $context = array() ); }
Use it! Abuse it!
![Page 80: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/80.jpg)
Context
h&6EE%+/["#M6V?;(%+.we8.,'(.&6E6-)?Yd
h&6EE%+/["#M6V?;(%+..&6E6-)?D.N?-(%+J">?.\[.e8RY
Instead of building dynamic log messages
Use static strings and add data into the context
![Page 81: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/81.jpg)
@.....A)%<4&')%A5.A&6E()'(,/XAD.....A(%))"#E(A.5.AWWWAD.....A<'44"#E([email protected](b(&6EA5.AWWWAD.........A,))4A5.AWWWAD.........AE%&[email protected]>b#'<"*J)%<4&')%([email protected]*)])[email protected]<')*,A5.A*)])JXAD.....................A<')*,J<'44"#EJ)b4%A5.A()+"#EAD.....................A<'44"#[email protected])b4%A5.A()+"#EAD.........................A"#>%]A5.A#6)J'#'&bK%>A.....................G.................G.............GRD.............A4+64%+)"%([email protected])"<%()'<[email protected])b4%A.5.A>')%AD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A<%(('E%[email protected])b4%A.5.A()+"#EAD.A"#>%]A.5.A'#'&bK%>A.GD.................AM'*"&")[email protected])b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A)b4%[email protected])b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A*)])J-(%+J">[email protected])b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.G.............G.........G.....G.G
![Page 82: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/82.jpg)
ProcessorAdd metadata for ALL your logs
![Page 83: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/83.jpg)
ProcessorsGitProcessor
IntrospectionProcessor MemoryPeakUsageProcessor
MemoryProcessor MemoryUsageProcessor
ProcessIdProcessor PsrLogMessageProcessor
TagProcessor UidProcessor
WebProcessor
![Page 84: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/84.jpg)
ProcessorCurrent user ID
User-Agent Locale
Country Code IP
Current route Application name / type
Environment (dev / staging / prod) Request UUID
API client …
![Page 85: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/85.jpg)
4-!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>[email protected]%#L.....\.h),"(/[*6#)'"#%+/[E%)V?=%+#%&?Y/[E%)2#L"+6#<%#)VYd.....h*6#)%]).\.h),"(/[*6#)'"#%+/[E%)V?*6#)%])?Yd.
....h+%*6+>N?%])+'?RN?%#L?R.....\.h%#Ld.
....h+%*6+>N?%])+'?RN?&6*'&%?R..\.h*6#)%])/[E%)P6*'&%VYd.
....h+%*6+>N?%])+'?RN?<%>"'?R...\.h*6#)%])/[E%)B%>"'VYd.
....h+%*6+>N?%])+'?RN?*&"?R.....\.h*6#)%])/["(j&"VY.s.Z.5.9d.
"M.Vh),"(/[*6#)'"#%+/["(I*64%O*)"L%V?+%C-%()[email protected]+%C-%().\.h),"(/[*6#)'"#%+/[E%)V?+%C-%()J()'*=?Y/[E%)B'()%+Q%C-%()VYd.
"M.Vh+%C-%()/[,%'>%+(/[,'(V?0/Q%C-%()/k>[email protected]+%C-%()k>.\.h+%C-%()/[,%'>%+(/[E%)V?0/Q%C-%()/k>?Yd.............h+%*6+>N?%])+'?RN?+%C-%()J">?R.\.h+%C-%()k>d.........G.
........h+%*6+>N?%])+'?RN?+6-)%?R.\.h+%C-%()/[E%)V?J+6-)%?Yd.
........h+%*6+>N?%])+'?RN?*&"%#)J"4?R.\.h+%C-%()/[E%)j&"%#)k4VYd.
....G.
"M.V#-&&.v\\.h-(%+.\.h-(%+B'#'E%+/[E%)j-++%#);(%[email protected]+%*6+>N?%])+'?RN?-(%+J">?R.\.h-(%+/[E%)k>VYd.....G.
$$.WWW.
+%)-+#.h+%*6+>d.G
![Page 86: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/86.jpg)
![Page 87: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/87.jpg)
Context + Processor
h4+6t%*).\.#%g.:+6t%*)Ve8D.?M66!'+?Yd.!h&6EE%+/["#M6V?:+6t%*).*+%')%>?D.N.....?4+6t%*)?.\[.h4+6t%*).RYd.
Use processor to pretty format the context
![Page 88: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/88.jpg)
Context + Processor*&'((.:+6t%*):+6*%(([email protected]!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>(Y.....@........."M.Vv"((%)Vh+%*6+>(N?*6#)%])?RN?4+6t%*)?RYY.@.............+%)-+#d.........G.!........h4+6t%*).\.h+%*6+>(N?*6#)%])?RN?4+6t%*)?Rd........."M.Vvh4+6t%*)."#()'#*%6M.:+6t%*)Y.@.............+%)-+#d.........G.!........h+%*6+>(N?*6#)%])?RN?4+6t%*)?R.\.N.............?">?...\[.h4+6t%*)/[E%)k>VYD.............?#'<%?.\[.h4+6t%*)/[E%)q'<%VYD.........R.!........+%)-+#.h+%*6+>(d.....G.G
![Page 89: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/89.jpg)
ELK is very powerful & flexible
![Page 90: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/90.jpg)
Other usages
![Page 91: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/91.jpg)
Business Metrics
![Page 92: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/92.jpg)
Business Metrics
http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/
![Page 93: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/93.jpg)
PacketBeat: Performance Management
![Page 94: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/94.jpg)
Suricate: Security
https://home.regit.org/tag/suricata/
![Page 95: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/95.jpg)
Graphana: Monitoring
![Page 96: Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo – IPC Berlin 2015](https://reader033.vdocument.in/reader033/viewer/2022041422/5e1fafb92ddf4243cd4080df/html5/thumbnails/96.jpg)
Thank You
Slides available at
moquet.net/talks/ipc-2015-elk
Leave feedbacks at @MattKetmo