Muga NishizawaTreasure Data, Inc.
the missing log collector
Muga Nishizawa (@muga_nishizawa)Chief Software Architect, Treasure Data
3
Treasure Data Overview Founded to deliver big data analytics in days not months without
specialist IT resources for one-tenth the cost of other alternatives Service based subscription business model World class open source team
• Founded world’s largest Hadoop User Group• Developed Fluentd and MessagePack• Contributed to Memcached, Hibernate, etc.
Treasure Data is in production• 60+ customers incl. Fortune 500 companies• 400+ billion records stored
Processing 40,000 messages per second
=Fluentd
syslogd+
many
=Fluentd
syslogd+
many
✓ Plugins
✓ JSON
> Open sourced log collector written in Ruby
> Using rubygems ecosystem for plugins
In short
It’s like syslogd, butuses JSON for log messages
Make log collection easyusing Fluentd
Reporting & Monitoring
Reporting & Monitoring
Collect Store Process Visualize
Collect Store Process Visualize
easier & shorter time
Hadoop / Hive
MongoDBTreasure Data
Tableau
Excel
RReporting & Monitoring
Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDBTreasure Data
Tableau
Excel
R
Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDBTreasure Data
Tableau
Excel
R
Before Fluentd
Application
・・・
Server2
Application
・・・
Server3
Application
・・・
Server1
FluentLog ServerHigh Latency!must wait for a day...
After Fluentd
Application
・・・
Server2
Application
・・・
Server3
Application
・・・
Server1
In streaming!
Fluentd Fluentd Fluentd
Fluentd Fluentd
Many Users
Many Meetups
Growth by Community
Why did we develop Fluentd?
Apache
App
App
Other data sources
td-agent RDBMS
Treasure Data columnar data
warehouse
Query Processing Cluster
Query API
HIVE, PIG (to be supported)
JDBC, REST
MAPREDUCE JOBS
User
td-command
BI apps
Treasure Data Service Architecture
Apache
App
App
Other data sources
td-agent RDBMS
Treasure Data columnar data
warehouse
Query Processing Cluster
Query API
HIVE, PIG (to be supported)
JDBC, REST
MAPREDUCE JOBS
User
td-command
BI apps
Treasure Data Service ArchitectureOpen Sourced
writes logs to text files
Rails app
GoogleSpreadsheet
MySQL
MySQL
MySQL
MySQL
writes logs to text files
Nightly
INSERT
hundreds of app servers
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
writes logs to text files
Rails app
- Limited scalability- Fixed schema- Not realtime- Unexpected INSERT latency
Example Use Case – MySQL to TD
hundreds of app servers
sends event logs
sends event logs
sends event logs
Rails app td-agent
td-agent
td-agent
GoogleSpreadsheet
Treasure Data
MySQL
Logs are available
after several mins.
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
Rails app
✓ Unlimited scalability✓ Flexible schema✓ Realtime✓ Less performance impact
Example Use Case – MySQL to TD
td-agent
> Open sourced distribution package of fluentd
> ETL part of Treasure Data
> Including useful components> ruby, jemalloc, fluentd> 3rd party gems: td, mongo, webhdfs, etc...
td plugin is for TD
> http://packages.treasure-data.com/
How Fluentd works?
=Fluentd
syslogd+
many
✓ Plugins
✓ JSON
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Input Plugins Output Plugins
Buffer Plugins(Filter Plugins)
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Architecture
Buffer OutputInput
> Forward> HTTP> File tail> dstat> ...
> Forward> File> Amazon S3> MongoDB> ...
> Memory> File
Pluggable Pluggable Pluggable
Architecture
Buffer OutputInput
> Forward> HTTP> File tail> dstat> ...
> Forward> File> Amazon S3> MongoDB> ...
> Memory> File
Pluggable Pluggable Pluggable
117 plugins!Contributions by Community
Input Plugins Output Plugins
2012-02-04 01:33:51myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing”}
timetag
record
JSON
log
> second unit
> from data source oradding parsed time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> JSON format
> MessagePackinternally
> non-unstructured
✓ Record
in_tail: reads file and parses lines
fluentdapache
access.log
✓ read a log file✓ custom regexp✓ custom parser in Ruby
in_tail
out_mongo: writes buffered chunks
fluentdapache
access.log buffer
in_tail
failure handling & retrying
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
in_tail
out_s3
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
Amazon S3
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
out_hdfs
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
HDFS
✓ custom text formater
routing / copying
fluentdapache
access.log buffer
✓ routing based on tags✓ copy to multiple storages
in_tail
Amazon S3
Hadoop
Fluentd
# RubyFluent.open(“myapp”)Fluent.event(“login”, {“user” => 38})#=> 2012-12-11 07:56:01 myapp.login {“user”:38}
> Ruby> Java> Perl> PHP> Python> D> Scala> ...
Application
Time:Tag:Record
Client libraries
# logs from a file<source> type tail path /var/log/httpd.log format apache2 tag web.access</source>
# logs from client libraries<source> type forward port 24224</source>
# store logs to MongoDB and S3<match **> type copy
<match> type mongo host mongo.example.com capped capped_size 200m </match>
<match> type s3 path archive/ </match></match>
Fluentd
out_forward
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
fluentd
fluentd
fluentd
✓ automatic fail-over✓ load balancing
forwarding
fluentd
fluentd
fluentd
fluentd
fluentd
fluentdfluentd
send / ackFluentd
Fluentd - plugin distribution platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo
Use cases
hundreds of app servers
sends event logs
sends event logs
sends event logs
Rails app td-agent
td-agent
td-agent
GoogleSpreadsheet
Treasure Data
MySQL
Logs are available
after several mins.
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
Rails app
✓ Unlimited scalability✓ Flexible schema✓ Realtime✓ Less performance impact
Cookpad
✓ Over 100 RoR servers (2012/2/4)
http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013
NHN Japan
by @tagomoris
✓ 16 nodes✓ 120,000+ lines/sec✓ 400Mbps at peak✓ 1.5+ TB/day (raw)
Web Servers Fluentd
Cluster
ArchiveStorage(scribed)
FluentdWatchers
GraphTools
Notifications(IRC)
Hadoop ClusterCDH4
(HDFS, YARN)
webhdfs
HuahinManager
hiveserver
STREAM
Shib ShibUI
BATCH SCHEDULEDBATCH
Treasure Data
FrontendJob Queue
WorkerHadoop
Hadoop
Fluentd
Applications push metrics to Fluentd(via local Fluentd)
Librato Metricsfor realtime analysis
Treasure Data
for historical analysis
Fluentd sums up data minutes(partial aggregation)
Key to Fluentd’s growth is...
=
Fluentd
syslogd+
many+
Community
✓ Plugins
✓ JSON
Muga NishizawaTreasure Data, Inc.
the missing log collector