fluentd at hkoscon
TRANSCRIPT
Fluentd: Unified logging layerJune 25, 2016 / HKOScon Masahiro Nakagawa
Who am I• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data, Inc. • Senior Software Engineer • Fluentd / td-agent developer
• Living at OSS :) • D language - Phobos, a.k.a standard library, committer • Fluentd - Main maintainer • MessagePack / RPC • etc…
Background
Before
✓ duplicated code for error handling...✓ messy code for retrying mechanism...
After
Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/
Concept /
Design
What’s Fluentd?• Data collector for unified logging layer
• Streaming data transfer based on JSON / MessagePack
• Robust core and plugins written in Ruby
• Rubygems based various plugins by the community
• www.fluentd.org/plugins • Apache License, Version 2.0
• https://github.com/fluent/fluentd
Reliable data transfer
error retry
error retry retry
retryBatch
Stream
Other stream
(micro-batch)
Core
Common concerns Use case specific
PLUGINSCORE• Read Data• Parse Data• Buffer Data• Write Data• Format Data
• Divide & Conquer• Buffering & Retries• Error Handling• Message Routing• Parallelism
> logged time
Event structure(log message)✓ Time
> for message routing
✓ Tag> Actual content
> JSON / MessagePack
✓ Record
127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...
...
2012-12-11 07:26:27apache.log
{ "host": "127.0.0.1", "method": "GET", ...}
Apache log Fluentd event
convert
timetag
record
Architecture
Data processing pipeline
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
Input plugins
File tail (in_tail)Syslog (in_syslog)HTTP (in_http)RDBMS (in_sql)...
✓ Receive logs
✓ Or pull logs from data sources
✓ non-blocking
InputInput
Parser plugins
JSONRegexpApache/NginxSyslogCSV/TSVetc.
✓ Parse into JSON
✓ Common formats out of the box
✓ Some inputs plugin depends on
Parser plugin
ParseParser
Filter plugins
greprecord_transformersuppress…
✓ Filter / Mutate record
✓ Record level and Stream level
✓ v0.12 and above
Filter
Buffer plugins
✓ Improve performance
✓ Provide reliability
✓ Provide thread-safetyMemory (buf_memory)File (buf_file)
Buffer
Buffer internal
✓ Chunk = adjustable unit of data
✓ Buffer = Queue of chunks
chunk
chunk
chunk output
Input
Formatter plugins
✓ Format output
✓ Convert json into other format
✓ Some plugins depends on
Formatter plugins
JSONCSV/TSV“single value”msgpack
Formatter
Output plugins
✓ Write to external systems
✓ Buffered & Non-buffered
✓ 300+ pluginsFile (out_file)Amazon S3 (out_s3)Kafka (out_kafka)...
Output
Use cases
Simple Forwarding
# logs from a file<source> @type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag backend.apache</source>
# logs from client libraries<source> @type forward port 24224</source>
# store logs to MongoDB<match backend.*> @type mongo database fluent collection test</match>
Less Simple Forwarding
- At-most-once / At-least-once - HA (failover)- Load-balancing
Container Logging
Treasure DataFrontend
Job Queue
WorkerHadoop
Presto
Fluentd
Applications push metrics to Fluentd (via local Fluentd)
Datadogfor realtime monitoring / alerting
Treasure Datafor historical analysis
Fluentd sums up data minutes(partial aggregation)
Slideshare
http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/
Roadmap
History / Roadmap• v0.10.0 (In Oct 2011)
• v0.12.0 (In Dec 2014) -> Current stable
• v0.14.0 (In May 2016)
• v0.14.x (some versions in 2016)
• v1 (4Q in 2016 / 1Q in 2017)
v0.10 (old stable)
> First stable release> Mainly for log forwarding
> Only Input and Output> No complex event handling support
> Only At-most-once semantics in forwarding> Treasure Data provides td-agent 1 for v0.10
v0.12 (current stable)> Event handling improvement
> Filter, Label, Error Stream> New configuration format> Add at-least-once semantics in forwarding
> Use require_ack_response parameter> HTTP RPC based process management> Treasure Data provides td-agent 2 for v0.12
• New Plugin APIs, Plugin Helpers & Plugin Storage
• Supervisor using ServerEngine
• Time with Nanosecond precision
• Windows support
v0.14 (Next stable)
Time with nanosecond• Fluent::EventTime
• behaves as Integer (used as time in v0.12) • has methods to get sub-second precision • be serialized into msgpack using Ext type
• Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-
system (e.g., fluent-logger, Docker logging driver)
Windows support• Fluentd and core plugin work on Windows
• several companies have already used v0.14.0.pre version on production
• We will send a patch to popular plugins ifit doesn’t work on Windows
• Use HTTP RPC instead of signals
• Treasure Data will provide td-agent msi package
v0.14.x - v1
• v0.14.x (some versions in 2016) • Symmetric multi processing model • Counter API • TLS/authentication/authorization support
(merging secure-forward into core)
• v1 (4Q in 2016 / 1Q in 2017) • Stable version for new APIs/features • fully compatible with v0.12.x and v0.14.x
Symmetric multi processing model
• 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and
one (or some) output plugin • No more fluent-plugin-multiprocess
Worker
Supervisor
Worker Worker
Worker
Supervisor
Worker Worker
Supervisor Supervisor
Using fluent-plugin-multiprocess
v0.14
TLS/Authn/Authz support for forward plugin
• secure-forward will be merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/ non-SSL
forwarding
• Authentication and Authorization providers • Who can connect to input plugins?
What tags are permitted for clients? • New plugin types (3rd party authors can write it) • Mainly for in/out forward, but available from others
Ecosystem
Treasure Agent (td-agent)> Treasure Data distribution of Fluentd
> include Ruby runtime, fluentd and popular plugins> deb, rpm, dmg are supported
> Treasure Agent 2 is current stable> fluentd v0.12 and Ruby 2.1
> Treasure Agent 3 will be released at fall> fluentd v0.14 and Ruby 2.3> msi will be supported for Windows
fluentd-ui> Manage Fluentd instance via Web UI
> https://github.com/fluent/fluentd-ui
fluent-bit
> http://fluentbit.io/> Log collector mainly for embedded systems
> Written in C> Can communicate with Fluentd
> There are several plugins> Treasure Data, Elasticsearch, MQTT, etc…
> Bulk Loader version of Fluentd> Pluggable architecture
> JRuby, JVM languages> High throughput by parallel processing
> Invented by Treasure Data> Share your script as a plugin> https://github.com/embulkSlide: Fighting Against Chaotically Separated Values with Embulk