eko10 workshop opensource database auditing
DESCRIPTION
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled. In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products. Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.TRANSCRIPT
OPEN SOURCE DATABASE MONITORING AT SCALE
#eko10
What is this talk about?● Auditing tools available for databases, commercial and otherwise
● What problems they have
● How we can make them scale as much as we might need
About us
● Juan Bernero @89berner / [email protected]
o Hobbies = [‘Movies/Series','Reading','Programming']
o Mostly Blue Team
o http://secureandscalable.wordpress.com/
● Pablo Garbossao @pgarbossa / [email protected]
o Fully Blue Team
About MercadoLibre● Devops culture (everyone and their mothers can access the boxes)
● Different DBs technologies
● Hybrid Cloud
● Database servers > 1K && Servers > 15K
● More than 100000 qps
Commercial products● Expensive
● Lots of functionalities you might not need
● Don’t scale so well
● Will make you choose what to log
Audit options● Inline / TAP / sensors or agents
● Plugin based
● Sniffers
● Client loggers
Mysql Audit Options● Commercial products
● Mysql General Log
● MySQL Enterprise Audit Log Plugin
● Mysql audit plugins
● Mysql sniffer
Mysql General Log● Easy to activate, by default in mysql
● Can be customized by modifying the log table to a degree
● As of 5.1 can be activated on the fly
● Less freedom than audit plugins
MySQL Enterprise Audit
Log Plugin● Available for Mysql Enterprise
● Uses the open MySQL Audit API
● Does not log triggers or prepared statements
● Allows asynchronous or synchronous logging
Mysql Audit Plugin (1)● Works using API created by Mysql to replace the general log
● Available in Github
● Flexibility to choose objects to inspect, types of queries to log or users to
whitelist
● Similar restrictions as Mysql Enterprise Plugin
Mysql Audit Plugin (2)● Steps to audit
o Download the plugin from github
o Move the library to /usr/lib/mysql/plugin/
o Enable with INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so';
o Either send it to a file or to a socket
Mysql Audit Plugin (3)● The log now looks like:
"msg-type":"activity","date":"1414531661274","thread-id":"72","query-
id":"1600563","user":"workshop","priv_user":"workshop","host":"ip-172-31-32-202.us-west-
2.compute.internal","ip":"172.31.32.202","cmd":"show_fields","query":"show_fields"
Mysql Audit Plugin (4)We can parse it with logstash into:
{
….
"@timestamp": "2014-10-29T04:10:37.000Z",
"type": "mysqlplugin",
"host": "0.0.0.0",
"path": "/var/log/mysqlplugin-2014-10-29.log",
"date": "Oct 29 04:10:37",
"agent": "54.200.106.239",
"user": "workshop",
"priv_user": "workshop",
"srcip": "54.69.169.73",
"command": "show_fields\",\"",
"query": "show_fields"
}
}
Mysql Audit Plugin DEMO
Problems with the Plugin● Generates overhead on the host
● Can’t log all events (audit api limitations)
● Not available in sniffing only situations
Sniffing options● Span port which sends you the traffic
● Sniffing and parsing from the server
● Forwarding the traffic from the agent (ie: iptables)
● Using agents to sniff traffic and forward it to repeaters (which repeat locally
the traffic with the original address)
Mysql Sniffer● Client / Server architecture
● Sniffs for common queries (select/insert/update/delete)
● Beta phase
● Has to keep up with protocol changes
Mysql Sniffer Agent● It will sniff traffic on Mysql port 3306 and send it elsewhere
● Small use of resources
● Must be tweaked to work in high load situations (ie: increase buffer for
packets to be processed)
./agent eth0 3306 DESTINATION 9200 1000 5000 5
Mysql Sniffer Repeater● Application that listens at a tcp port for connections
● Receives packets and does a local replay of them
● Packets are seen as coming from the original client
./repeater 9200
Mysql Sniffer Parser● Listens to traffic on the interface for the mysql port
● Parsers queries and keeps track of connections
● Writes output to logfile:
Wed Oct 29 00:20:24,54.69.169.73,55981,172.31.32.202,workshop,test,
select,"select * from test"
Mysql Sniffer DEMO
Mysql Sniffer Problems● Not reliable
● Depends on the protocol not to change or something weird not to happen
● Only a limit subset of types of queries which represent most queries
● Shouldn’t be used on databases with small activity
You can use a combo● Mysql sniffer to audit common queries without giving overhead to the
mysql server
● Mysql Audit Plugin to audit all other queries or specific objects with more
reliability
● Be creative
MongoDB● Document oriented database
● Great scaling capabilities
● Bson Data Store
● Most popular NoSQL (according to wikipedia)
MongoDB Operations● Insert: db.scores.save({a:99})
● Delete: db.scores.remove({server: 999});
● Update: db.scores.update({a: 5}, {server:999});
● Query: db.scores.find();
MongoDB auditing options● Server log
● MongoDB Enterprise Auditing
● Query to the oplog
● Mongosniff
MongoDB Mongosniff● Gives you detailed output of operations in MongoDB
● Does not come in the default package, you need to compile it
● Uses the mongo libraries to parse the commands
● Sample output: 111.22.33.44:6612 <<– 22.33.44.55:42947 262 bytes id:6a89eb 6982123 –
308293
reply n:4 cursorId: 0
{ _id: “db”, partitioned: false, primary: “Segmon_RS1″ }
MongoDB Mongosniff
(Modified)● Some pcap tweaks to reduce dropped packets
● Minor bug fixes
● Different output format:
172.31.36.172:56228,54.68.230.224:6612,test.$cmd,,query,{ authenticate: 1, nonce:
"745ad1e4a6075a25", user: "workshop", key: "869c8d69703e2d1bb9394ddf4c116dcb" }
ntoreturn: 1 ntoskip: 0AAAAAAA
MongoDB Mongosniff
Wrapper● Ruby wrapper
● Handles extra functions we would need without modifying mongosniff
● Output format:
Oct 29 03:43:11,workshop,54.68.230.224,workshop,test,172.31.36.172:56231, 54.68.230.224:6612,
test.cmd,,query,{ isMaster: 1.0, forShell: 1.0 } ntoreturn: -1 ntoskip: 0
MongoDB Mongosniff
Architecture
Mongo Sniffer DEMO
MongoDB Sniffer Problems● No support for packet fragmentation
● Not 100% reliable
● Not it’s intended use
References
● https://github.com/89berner/MysqlAudit
● https://github.com/mcafee/mysql-audit
thank you!
we’re hiring ;)