centralized + unified logging
TRANSCRIPT
Centralized + Unified Logging
Gabor Kozma / [email protected] / @kozmag82
Everybody wants to write logs!
✓ Application Logs (frontend / backend)➢ php, java, ruby, python, bash
✓ Access Logs➢ apache, nginx, tomcat, jetty
✓ System Logs➢ syslog, hardware error log
✓ Database Logs➢ history, transaction
Centralized Logging
Central Logging Architecture
✓ Collection➢ file, syslog, database
✓ Transport➢ chukwa, heka, syslog, logstash, flume, fluentd,
kafka, nsq, nxlog, other custom solution.Typical: syslog-ng, rsyslog
✓ Storage / Store➢ Amazon S3, Glacier, NAS ...
Central Logging Architecture
✓ Analysis (You need a way to analyze them!)
➢ Apache Hadoop + HDFS + Map-Reduce jobs■ Hive, Pig, HBase, Impala....
➢ Elasticsearch + Graylog2 / Kibana➢ MongoDB + Map-Reduce/Aggregation Framework➢ Graphite, Statsd + Dashboards
✓ Alerting (Errors almost always indicate a problem!)
➢ Airbreak/Errbit, Sentry, Honeybadger, Nagios, Zabbix, Open/PagerDuty
Unified Logging Layer
Unified Logging Layer
✓ Ubiquity➢ Various format problem➢ Various source and destination➢ You must be optimize most of use case!
✓ Rigidity vs. Flexibility➢ Apache Thrift , Apache Avro, Protocol Buffer , JSON /
BSON, MessagePack
Unified Logging Layer
✓ Reliability and Scalability➢ Scalable➢ Support retryable data transfer➢ Sync / Async data transfer➢ Push / Pull base system
✓ Extensibility➢ Support new input / output
■ You don’t have to modify anything else.
Fluentd - Pluggable architecture
✓ Input, Output, Buffer, Parser, Formatter300+ plugins
Fluentd - Minimum res. require
✓ Combination of C language and Ruby✓ 1 node✓ 30/40 Mbyte RAM✓ 1 CPU core
13.000 event / sec
Fluentd - Built-in Reliability
✓ Buffer➢ file or memory
✓ Retrying✓ Error handling
➢ transaction, failover, secondary node support (heartbeat)
Fluentd - Event structure (log)
✓ Time➢ Second unit➢ From data source or adding parsed time
✓ Tag➢ for message routing
✓ Record➢ JSON format
■ MessagePack internally :)■ none structured
Fluentd - Useful plugins
✓ Output➢ stdout, file, forest, graphite, mongo, mysql,
elasticsearch, splunk, null, s3, geoip, webhdfs✓ Input
➢ syslog, tail, http, udp, tcp, scribe✓ Buffer
➢ memory, file✓ Formatter and/or Parser
➢ lstv, json, multiline
Examples
Fluentd - Examples<source>
type tail
format /^(?<host>[^ ]*)\:(?<port>[^ ]*) (?<ip>[^ ]*) (?<user>[^ ]*) (?<remotelog>[^ ]*) \
[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)
(?: "(?<referer>[^\"]*)" "\\"(?<agent>[^\\"]*)\\"")?(?: "(?<referer>[^\"]*)" "(?<agent>
[^\"]*)")?$/
path /var/log/apache2/other_vhosts_access.log.*
pos_file /var/log/fluent/apache2.other_vhosts_access.log.pos
time_format %d/%b/%Y:%H:%M:%S %z
tag apache2.access.raw
read_from_head true
</source>
Fluentd - Examples<match apache2.*.raw>
type record_reformer
enable_ruby false
renew_record false
remove_keys remotelog
tag ${tag_prefix[-2]}.reformed
<record>
hostname ${hostname}
</record>
</match>
<match apache2.*.reformed>
type geoip
geoip_lookup_key ip
geoip_database /usr/share/GeoIP/GeoIPCity.dat
<record>
geo_city ${city['ip']}
...
geo_region ${region['ip']}
</record>
add_tag_suffix .geoip
flush_interval 5s
</match>
<match apache2.access.reformed.geoip>
type forward
flush_interval 5s
buffer_type file
buffer_queue_limit 512
buffer_chunk_limit 100M
buffer_path /opt/fluent/buffer/apache2/
<server>
name hostname
host xxx.xxx.xxx.xxx
weight 10
</server>
Fluentd - Examples...
<server>
name hostname
host xxx.xxx.xxx.xxx
standby
</server>
<secondary>
type file
path /var/log/fluent/forward-
failed/apache2/
</secondary>
</match>
Fluentd - Examples<match apache2.access.**>
type copy
<store>
type file
path /opt/fluent/apache2/access
time_format %Y%m%dT%H%M%S%z
flush_interval 60s
append true
compress gzip
utc
num_threads 4
...
...
</store>
<store>
type datacounter
...
</store>
<store>
type graphite
...
</store>
</match>
Fluentd - Testimonials
Questions?http://www.fluentd.org
http://docs.fluentd.org/
http://fluentular.herokuapp.com/
https://rubygems.org/search?query=fluent-plugin-
http://msgpack.org/