logging application behavior to mongodb
DESCRIPTION
Presentation given at Mongo SV conference in Mountain View on December 3, 2010. Covers reasons for logging to MongoDB, logging library basics and library options for Java, Python, Ruby, PHP and C#. Updated 1/1/2012 with more info on logging in Ruby and tailable cursors.TRANSCRIPT
![Page 1: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/1.jpg)
Logging Application Behavior to MongoDB
Robert Stewart @wombatnation
1/1/2012
![Page 2: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/2.jpg)
TOC
• Why Log to MongoDB?
• Logging Library Basics
• Logging by Language
• Other Ways to Distribute Your Logging
![Page 3: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/3.jpg)
Why Log to MongoDB?
• Centralized application logging
• Easy remote access, compared to files
• Log events good fit for document model
• Flexible schema
• Indexes for frequent queries
• Capped collections are very efficient
• Analyze data in-place with JavaScript MapReduce
![Page 4: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/4.jpg)
Event Storage
• Create a database for your app
• Create a capped collection to store log events
• Time-based eviction more natural, but less efficient
• Capped collection reduces chance of filling up disk
![Page 5: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/5.jpg)
Log Analysis
• Compared to files, easier to analyze app behavior across servers and days
• Log values to custom keys where possible
• JavaScript MapReduce
![Page 6: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/6.jpg)
Miscellaneous Free Advice
• Take advantage of separating storage from presentation
• Be aware of which classes your Mongo driver knows how to BSONify
• If logging from multiple languages, sticking to Log4Lang family can help
![Page 7: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/7.jpg)
Logging Library Basics
• Some provide API with pluggable implementation
• Configuration
• Programmatic
• Declarative
• Dynamic
![Page 8: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/8.jpg)
Logging Concepts
• Named, Hierarchical Logger Objects
• Identify a class or an area of an app
• Log Level, e.g., error, warn, info, debug
• Thresholds determine if event is logged
• Logging Event
• The message
• Metadata, e.g., time, level, class, method, line
![Page 9: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/9.jpg)
Logging Concepts
• Handler - determines event destination
• File, console, socket, email, syslog, MongoDB, etc.
• Handler level threshold
• Async vs. sync
• Filter - for logger and/or handler
• Formatter - event and message
• Pattern with placeholder tokens
![Page 10: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/10.jpg)
App calls logger.info(“freak out”)
Common Flow for Logging
Is logger info
enabled?
Passes filter, if any?
For each handler:
No
No
Yes
Yes
Is handler info
enabled?
Passes filter, if any?
No
Yes
Yes
Format and store event
No
![Page 11: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/11.jpg)
Polyglotism
• Good news - lots of client language bindings and logging libraries for MongoDB
• Mostly good news - Log4J very influential
• Sort of bad news - default log event formats can be very different
![Page 12: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/12.jpg)
Logging by Language
• Java
• Python
• Ruby
• PHP
• C#
![Page 13: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/13.jpg)
Java
• APIs
• Apache Commons Logging
• SLF4J
• java.util.logging, a.k.a., JUL
• Limited feature set
• Destinations - console, file, socket, memory
• JVM-level config
![Page 14: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/14.jpg)
Log4J • Apache Log4J is dominant logging library
• Logback is potential successor
• Logger - usually, represents class logging the event
• LoggingEvent - message and metadata
• Appender - sends LoggingEvent to destination
• Layout - formats the event
• Converter - convert placeholder to value
![Page 15: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/15.jpg)
Log4mongo-java
• Set of Log4J Appenders for MongoDB
• log4mongo.org
• Open source project on GitHub
• I’m a committer with Peter Monks and Jozef Ševčík
• I used it at Voxify with speech reco applications and infrastructure services
• Peter has used it at Alfresco
![Page 16: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/16.jpg)
MongoDbAppender
• Stores a Log4J LoggingEvent object as a document in a collection
• Format is very similar to structure of LoggingEvent Java class
• Where possible, data is stored as appropriate BSON objects, rather than just strings
![Page 17: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/17.jpg)
MongoDbPatternLayoutAppender
• Stores log event based on user-specified pattern
• Standard Log4J pattern layout, parser and converter functionality
• Very flexible formatting at cost of additional config
• Values stored as strings or arrays
• Custom values determined at time event is logged
![Page 18: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/18.jpg)
BsonAppender
• Custom log format by extending BsonAppender
• Can use it to log BSON objects to other data stores
• Includes code for BSONifying Java exceptions
![Page 19: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/19.jpg)
Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!
• Specify host (optional port, username and password). For replica set, specify all active hosts.
log4j.appender.MongoDB.hostname = localhost
• Specify database and collection log4j.appender.MongoDB.databaseName = app_name
log4j.appender.MongoDB.collectionName = log
• Add named appender to rootLogger list log4j.rootLogger = INFO, file, MongoDB
![Page 20: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/20.jpg)
Using Pattern Layout Appender • Specify pattern layout appender
log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender
• Specify your custom Pa2ernLayout class log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout
• Specify a conversion pattern
• Must be a valid JSON document
• Can have sub-documents
• Values can be strings or arrays
log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}", "level":"%p","class":"%c{1}","message":"%m"}
![Page 21: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/21.jpg)
Shell and Tail
![Page 22: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/22.jpg)
In the Shell
> use app_name switched to db app_name > db.log.findOne() {
"_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags”
}
![Page 23: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/23.jpg)
Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js
> function last(count) {
if (count==null) count = 1;
return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count);
}
> last(2)
{ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "uninformative message" }
{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "some info kind of message" }
![Page 24: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/24.jpg)
Tailing Logs
• You’ll really miss ability to tail logfiles
• Or, ... will you?
• MongoDB offers tailable cursors
• Specify cursor, e.g., from find, as tailable
• While cursor alive, print new docs then sleep
![Page 25: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/25.jpg)
Python Tailing Example from pymongo import Connection import time db = Connection().my_db coll = db.my_collection cursor = coll.find(tailable=True) while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)
![Page 26: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/26.jpg)
Python
• Feature-rich logging module in Python 2.3
• Influenced by Log4J
![Page 27: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/27.jpg)
Components • Logger - represents area of library or project
• LogRecord
• Handler
• Filter
• Formatter
• LoggerAdapter (2.6) - add contextual info
![Page 28: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/28.jpg)
mongodb-log
• Python logging handler to log to MongoDB
• Stores dictionary directly as BSON doc
• https://github.com/andreisavu/mongodb-log
![Page 29: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/29.jpg)
log4mongo-python
• Similar design to log4mongo PHP and .NET appenders
• Example programmatic handler config and usage: import logging from log4mongo.handlers import MongoHandler
logger = logging.getLogger('test') logger.addHandler(MongoHandler(host='localhost')) logger.warning('test')
• http://log4mongo.org/display/PUB/Log4mongo+for+Python
• https://github.com/log4mongo/log4mongo-python
![Page 30: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/30.jpg)
Ruby
• Built-in Logger library
• Limited feature set
• Console and file outputs
• Message format is fixed
• Datetime format is configurable
![Page 31: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/31.jpg)
Log4r • Influenced by Log4J
• Logger
• Outputter
• Formatter
• Configurator
• http://log4r.rubyforge.org
• Code for using Log4r with MongoMapper
• http://gist.github.com/207347
![Page 32: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/32.jpg)
Logging
• Similar to Log4r with a few enhancements
• Supports JSON as output format
• http://logging.rubyforge.org/
![Page 33: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/33.jpg)
mongo_db_logger • Rails plugin which uses Rails Logger, which can use
Ruby Logger, Log4r, etc.
• Logs a single document per request
• IP address, time, etc.
• All messages logged during request
• Custom request data
• Configure via database.yml
• http://github.com/peburrows/mongo_db_logger
![Page 34: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/34.jpg)
Central Logger
• Forked from mongo_db_logger
• Added Rails 3 support and gemified
• Web UI for searching, filtering and analyzing logs
• Configure via database.yml (or mongoid.yml or central_logger.yml)
• http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb
• http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf
• https://github.com/customink/central_logger
![Page 35: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/35.jpg)
MongodbLogger
• Based on central_logger
• Different web UI for searching, filtering and analyzing logs
• Configure via database.yml (or mongoid.yml or mongodb_logger.yml)
• http://mongodb-logger.catware.org/
![Page 36: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/36.jpg)
PHP
• error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog)
• Log4PHP - heavily influenced by Log4J
• PEAR Log - PHP-specific logging library
• Zend_Log - Logging for Zend Framework
![Page 37: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/37.jpg)
Log4mongo-php
• Log4PHP appender
• Programmatic & declarative config
• Timestamp stored as Mongo Date
• Exception stored as array - msg, code, stacktrace
• Inner exception stored if available
• https://github.com/log4mongo/log4mongo-php
![Page 38: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/38.jpg)
PEAR Log
• PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB
• Programmatic config
• Specify handler when creating logger object
• Composite handler for multiple destinations
• http://pear.php.net/package/Log/
![Page 39: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/39.jpg)
Log Writer for Zend���
• Project -> Recordshelf_Log_Writer_MongoDb
• Described as a “prototypesque implementation” • Standard Zend tools to config and init
• Zend log event is associative array
• Project has code for accessing & filtering events • http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html
![Page 40: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/40.jpg)
C#/.NET
• Log4net
• Heavily influenced by Log4J
• http://logging.apache.org/log4net/
![Page 41: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/41.jpg)
Log4mongo-net
• Uses community developed C# driver, but may move to official driver
• Tested with .NET 3.5+ and Mono 2.8
• Standard App.config/Web.config or programmatic
• http://log4mongo.org/display/PUB/Log4mongo+for+.NET
![Page 42: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/42.jpg)
Other Options • Destinations include MongoDB
• Flume, logstash, Fluentd
• Primary Destination is HDFS
• Chukwa and Scribe
• Commercial Hosted
• Loggly
• Commercial On-Premise
• Splunk
![Page 43: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/43.jpg)
Flume • If you can’t change the source of logs ...
• Can aggregate files, syslog, etc.
• Agent --> Collector --> Storage
• Storage plugins for HDFS, MongoDB, etc.
• Decorators transform data & extract metadata
• https://github.com/cloudera/flume
• https://github.com/mongodb/mongo-hadoop
• Chukwa, Honu and Scribe more HDFS-centric
![Page 44: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/44.jpg)
logstash
• Log event storage, search and graphing
• Read from file, syslog, AMQP, tcp, etc.
• Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc.
• http://logstash.net/
• http://code.google.com/p/logstash
![Page 45: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/45.jpg)
Fluentd • Ruby-based open source log collector
• Input, output and buffer plug-ins
• For example, plug-ins for tailing Apache log files and storing log events to MongoDB
• http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with-fluentd-and-mongodb
• http://fluentd.org/doc/
• https://github.com/fluent/fluentd
![Page 46: Logging Application Behavior to MongoDB](https://reader034.vdocument.in/reader034/viewer/2022042508/554be996b4c9055a368b4c9c/html5/thumbnails/46.jpg)
Questions?
• Photo/Image Credits • Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702
• Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/
• Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/
• Advice - www.flickr.com/photos/wurzle/659315/
• Shell - http://www.flickr.com/photos/geishaboy500/305555652/
• Tail - http://www.flickr.com/photos/autumnsonata/2632194442/
• Log flume - www.flickr.com/photos/halloweenjack/1330806722/
• Cufflinks - www.flickr.com/photos/oberazzi/318947873/