unifying events & logs into the cloud i/o . logging. logging matters pros ... type tail type...
TRANSCRIPT
![Page 1: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/1.jpg)
Eduardo [email protected]
@edsiper
Unifying Events & Logs into the Cloud
August 17, 2015CloudOpen/LinuxCon, Seattle
![Page 2: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/2.jpg)
About MeEduardo Silva
● Github & Twitter @edsiper● Personal Blog http://edsiper.linuxchile.cl
Treasure Data
● Open Source Engineer● Fluentd / Fluent Bit http://github.com/fluent
Projects● Monkey HTTP Server http://monkey-project.com● Duda I/O http://duda.io
![Page 3: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/3.jpg)
Logging
![Page 4: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/4.jpg)
Logging MattersPros
● Application status
● Debugging
● General information about anomalies: errors
● Troubleshooting / Support
● Local or Remote (network)
![Page 5: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/5.jpg)
Logging MattersFrom a business point of view
● Input data Analytics→
● User interaction / behaviors
● Improvements
![Page 6: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/6.jpg)
Assumptions
![Page 7: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/7.jpg)
Logging MattersAssumptions
● I have enough disk space
● I/O operations will not block
● Log messages are human readable
● My logging mechanism scale
![Page 8: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/8.jpg)
Logging MattersAssumptions
Basically, yeah.. it should work.
![Page 9: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/9.jpg)
Concerns
![Page 10: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/10.jpg)
Logging MattersConcerns
● Logs increase = data increase
● Message format get more complex
● Did the Kernel flush the buffers ? (sync(2))
● Multi-thread application ?, locking ?
● Multiple Applications = Multiple Logs
![Page 11: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/11.jpg)
Logging MattersConcerns
If Multiple Applications = Multiple logs
Multiple Hosts x Multiple Applications = ???
![Page 12: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/12.jpg)
OK, so:
1. Logging matters2. It's really beneficial3. but...
![Page 13: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/13.jpg)
It needs to be done right.
![Page 14: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/14.jpg)
LoggingCommon sources & inputs
● Application Logs
● Apache● NginX● Syslog (-ng)
● Custom applications / Languages
● C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.
![Page 15: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/15.jpg)
In a galaxy not so far away...
![Page 16: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/16.jpg)
![Page 17: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/17.jpg)
How to parse/storemultiple data sources ?
note: performance matters!
![Page 18: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/18.jpg)
![Page 19: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/19.jpg)
Fluentd is an open source data collector
It let's you unify the data collection for a better use and understanding of data.
![Page 20: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/20.jpg)
before
![Page 21: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/21.jpg)
after
![Page 22: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/22.jpg)
FluentdHighlights
● High Performance
● Built-in Reliability
● Structured Logs
● Pluggable Architecture
● More than 300 plugins! (input/filtering/output)
![Page 23: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/23.jpg)
FluentdArchitecture
![Page 24: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/24.jpg)
FluentdInternals simplified
![Page 25: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/25.jpg)
FluentdInput plugins
![Page 26: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/26.jpg)
FluentdOutput plugins
![Page 27: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/27.jpg)
FluentdBuffer plugins
![Page 28: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/28.jpg)
FluentdBuffer plugins
![Page 29: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/29.jpg)
![Page 30: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/30.jpg)
M x N M + N→
![Page 31: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/31.jpg)
FluentdSimple Forwarding
![Page 32: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/32.jpg)
FluentdSimple Forwarding: configuration
# logs from a file # store logs to MongoDB<source> <match backend.*> type tail type mongo path /var/log/httpd.log database fluent format apache2 collection test tag backend.apache </match></source>
# logs from client libraries<source> type forward port 24224</source>
![Page 33: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/33.jpg)
FluentdLess Simple Forwarding
![Page 34: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/34.jpg)
FluentdLambda Architecture
![Page 35: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/35.jpg)
Fluentd# logs from a file # store logs to MongoDB<source> <match *.*> type tail type copy path /var/log/httpd.log <store> format apache2 type elasticsearch tag backend.apache logstash_format true</source> </store> # logs from client libraries <store><source> type webhdfs type forward host 192.x.y.z port 24224 port 50070</source> path /path/to/hdfs </store> </match>
![Page 36: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/36.jpg)
Who uses Fluentd in production ?
![Page 37: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/37.jpg)
![Page 38: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/38.jpg)
![Page 39: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/39.jpg)
We collect800k events per second !
![Page 40: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/40.jpg)
Internet of Things
![Page 41: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/41.jpg)
Internet of ThingsFacts
● IoT will grow to many billions of devices over the next decade.
● Now it's about device to device connectivity.
● Different frameworks and protocols are emerging.
● It needs Logging.
![Page 42: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/42.jpg)
Internet of ThingsAlliances
Vendors formed alliances to join forces and develop generic software layers for their products:
![Page 43: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/43.jpg)
Internet of ThingsSolutions provided
Alliance Framework
→
→
![Page 44: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/44.jpg)
IoT & Big DataAnalytics
IoT requires a generic solution to collect events and data from different sources for further analysis.
Data can come from a specific framework, radio device, sensor or other. How do we collect and unify data properly ?
![Page 45: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/45.jpg)
@fluentbit
![Page 46: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/46.jpg)
Fluent Bit is an open sourcedata collector
It let's you collect data from IoT/Embedded devices and transport It to third party services.
![Page 47: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/47.jpg)
Fluent BitTargets
● Services
● Sensors / Signals / Radios
● Operating System information
● Automotive / Telematics
![Page 48: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/48.jpg)
Fluent BitRequirements
IoT and Embedded environment requires special handling,specifically on performance and resource utilization:
● Lightweight
● Written in C Language
● Customizable, pluggable architecture
● Full integration with Fluentd
![Page 49: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/49.jpg)
Fluent BitIntegration
![Page 50: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/50.jpg)
Fluent BitDirect Output
![Page 51: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/51.jpg)
Containers
![Page 52: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/52.jpg)
![Page 53: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/53.jpg)
DockerLogging driver
● Docker v1.6 released the concept of logging drivers
● Route container output
● Fluentd ?
![Page 54: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/54.jpg)
Docker
![Page 55: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/55.jpg)
Docker v1.8Fluentd Logging driver!
![Page 56: Unifying Events & Logs into the Cloud I/O . Logging. Logging Matters Pros ... type tail type mongo path /var/log/httpd.log database fluent format](https://reader031.vdocument.in/reader031/viewer/2022030507/5ab6d3f97f8b9ab47e8e4569/html5/thumbnails/56.jpg)
We Love Data!
Thank you!
● http://fluentd.org
● http://fluentbit.io
● https://docs.docker.com/reference/logging/fluentd/
● http://github.com/fluent/fluentd