state of the art logging
TRANSCRIPT
Copyright 2013 BalaBit IT Security Ltd.
State of the art loggingSyslog-ng, journal, CEE/Lumberjack and ELSA
Péter Czanikcommunity manager
Copyright 2013 BalaBit IT Security Ltd.
Topics
• No, it is not about cutting trees :-)• What is syslog? And syslog-ng?• Free-form messages against name-value pairs• The new buzzword: journal• Standardization efforts: CEE/Lumberjack• Name-value pairs at work: ELSA
Copyright 2013 BalaBit IT Security Ltd.
What is syslog?
• Logging: recording events
• Syslog:- Application: collecting events- Protocol: forwarding events
Copyright 2013 BalaBit IT Security Ltd.
What is syslog-ng?
• “Next Generation” syslog server• “Swiss army knife” of logging
• More input sources (files, sockets, and so on)• Better filtering (not only priority, facility)• Processing (rewrite, normalize, correlate, and so
on)• More destinations (databases, encrypted network,
and so on)
Copyright 2013 BalaBit IT Security Ltd.
What is new since 2.0
• 2.0 is best known, but EOL• Most important new features since 2.0:
- PatternDB and CSV message parsing- Correlation- SQL and MongoDB destinations- JSON formatting- Modularization- Multi-threading
• Next: 3.4- JSON parsing- More flexible configuration
Copyright 2013 BalaBit IT Security Ltd.
Free form log messages
• Most logs are in /var/log• Most are from syslog (but also wtmp, apache, and
so on)• Most are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2
• Text = English sentence with some variable parts• Easy to read
Copyright 2013 BalaBit IT Security Ltd.
Why it does not scale?
• Few logs (workstation) → easy to find information• Many logs (server) → difficult to find information• Relevant information is presented differently by
each application• Difficult to process them with scripts
• Answer: structured logging- Events represented as name value pairs
Copyright 2013 BalaBit IT Security Ltd.
Solution from syslog-ng: PatternDB
• Most messages are static texts with some variable parts embedded
• PatternDB parser:- Can extract useful information into name-value pairs- Add status fields based on message text
• Example:- user=root- action=login- status=failure
• It requires patterns• syslog-ng: name-value pairs inside
Copyright 2013 BalaBit IT Security Ltd.
Journal
• The logging component of systemd• Name-value pairs inside:
- Message- Trusted properties- Any additional name-value pairs
• Native support for name-value pair storage
Copyright 2013 BalaBit IT Security Ltd.
Journal: the enemy?
• FAQ: Q: is journal the enemy? A: No!• Journal is limited to Linux/systemd (syslog-ng: all
Linux/BSD/UNIX)• Journal is local only (syslog-ng: client – server)• Journal does not filter or process log messages
• Journal + syslog-ng complement each other• Logs forwarded to syslog-ng through:
/run/systemd/journal/syslog
• syslog-ng can filter, process and forward logs to many different destinations (one day also to journal)
Copyright 2013 BalaBit IT Security Ltd.
CEE
• Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name-value pairs
• All use different field names• Standardization is a must: CEE → Common Event
Expression• Events: name-value pairs instead of free-form text
- Taxonomy: name-value pairs to describe events (example: status)- Dictionary: name-value pairs for event parameters (example: user)
• PatternDB can turn free-form messages into CEE
Copyright 2013 BalaBit IT Security Ltd.
Lumberjack
• Make CEE happen → implementation• Coordinated by RedHat
- CEE (Mitre), syslog-ng, rsyslog, and so on- Open, with high traffic mailing list- https://fedorahosted.org/lumberjack/
• API(s) to make structured logging easier• Work on dictionary, taxonomy, transport issues
Copyright 2013 BalaBit IT Security Ltd.
Name-value pairs in action: ELSA
• ELSA: Enterprise Log Search and Archive• Based on syslog-ng, PatternDB and MySQL• Simple and powerful web GUI• Extreme scalability• Patterns focused on network security (Cisco,
Snort, HTTP, Bro, and so on)
Copyright 2013 BalaBit IT Security Ltd.
Some logs
Copyright 2013 BalaBit IT Security Ltd.
Diagram
Copyright 2013 BalaBit IT Security Ltd.
A few extras
Copyright 2013 BalaBit IT Security Ltd.
Questions?
• Questions?