Download - Eko10 Workshop Opensource Database Auditing
OPEN SOURCE DATABASE MONITORING AT SCALE
#eko10
What is this talk about?● Auditing tools available for databases, commercial and otherwise
● What problems they have
● How we can make them scale as much as we might need
About us
● Juan Bernero @89berner / [email protected]
o Hobbies = [‘Movies/Series','Reading','Programming']
o Mostly Blue Team
o http://secureandscalable.wordpress.com/
● Pablo Garbossao @pgarbossa / [email protected]
o Fully Blue Team
About MercadoLibre● Devops culture (everyone and their mothers can access the boxes)
● Different DBs technologies
● Hybrid Cloud
● Database servers > 1K && Servers > 15K
● More than 100000 qps
Commercial products● Expensive
● Lots of functionalities you might not need
● Don’t scale so well
● Will make you choose what to log
Audit options● Inline / TAP / sensors or agents
● Plugin based
● Sniffers
● Client loggers
Mysql Audit Options● Commercial products
● Mysql General Log
● MySQL Enterprise Audit Log Plugin
● Mysql audit plugins
● Mysql sniffer
Mysql General Log● Easy to activate, by default in mysql
● Can be customized by modifying the log table to a degree
● As of 5.1 can be activated on the fly
● Less freedom than audit plugins
MySQL Enterprise Audit
Log Plugin● Available for Mysql Enterprise
● Uses the open MySQL Audit API
● Does not log triggers or prepared statements
● Allows asynchronous or synchronous logging
Mysql Audit Plugin (1)● Works using API created by Mysql to replace the general log
● Available in Github
● Flexibility to choose objects to inspect, types of queries to log or users to
whitelist
● Similar restrictions as Mysql Enterprise Plugin
Mysql Audit Plugin (2)● Steps to audit
o Download the plugin from github
o Move the library to /usr/lib/mysql/plugin/
o Enable with INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so';
o Either send it to a file or to a socket
Mysql Audit Plugin (3)● The log now looks like:
"msg-type":"activity","date":"1414531661274","thread-id":"72","query-
id":"1600563","user":"workshop","priv_user":"workshop","host":"ip-172-31-32-202.us-west-
2.compute.internal","ip":"172.31.32.202","cmd":"show_fields","query":"show_fields"
Mysql Audit Plugin (4)We can parse it with logstash into:
{
….
"@timestamp": "2014-10-29T04:10:37.000Z",
"type": "mysqlplugin",
"host": "0.0.0.0",
"path": "/var/log/mysqlplugin-2014-10-29.log",
"date": "Oct 29 04:10:37",
"agent": "54.200.106.239",
"user": "workshop",
"priv_user": "workshop",
"srcip": "54.69.169.73",
"command": "show_fields\",\"",
"query": "show_fields"
}
}
Mysql Audit Plugin DEMO
Problems with the Plugin● Generates overhead on the host
● Can’t log all events (audit api limitations)
● Not available in sniffing only situations
Sniffing options● Span port which sends you the traffic
● Sniffing and parsing from the server
● Forwarding the traffic from the agent (ie: iptables)
● Using agents to sniff traffic and forward it to repeaters (which repeat locally
the traffic with the original address)
Mysql Sniffer● Client / Server architecture
● Sniffs for common queries (select/insert/update/delete)
● Beta phase
● Has to keep up with protocol changes
Mysql Sniffer Agent● It will sniff traffic on Mysql port 3306 and send it elsewhere
● Small use of resources
● Must be tweaked to work in high load situations (ie: increase buffer for
packets to be processed)
./agent eth0 3306 DESTINATION 9200 1000 5000 5
Mysql Sniffer Repeater● Application that listens at a tcp port for connections
● Receives packets and does a local replay of them
● Packets are seen as coming from the original client
./repeater 9200
Mysql Sniffer Parser● Listens to traffic on the interface for the mysql port
● Parsers queries and keeps track of connections
● Writes output to logfile:
Wed Oct 29 00:20:24,54.69.169.73,55981,172.31.32.202,workshop,test,
select,"select * from test"
Mysql Sniffer DEMO
Mysql Sniffer Problems● Not reliable
● Depends on the protocol not to change or something weird not to happen
● Only a limit subset of types of queries which represent most queries
● Shouldn’t be used on databases with small activity
You can use a combo● Mysql sniffer to audit common queries without giving overhead to the
mysql server
● Mysql Audit Plugin to audit all other queries or specific objects with more
reliability
● Be creative
MongoDB● Document oriented database
● Great scaling capabilities
● Bson Data Store
● Most popular NoSQL (according to wikipedia)
MongoDB Operations● Insert: db.scores.save({a:99})
● Delete: db.scores.remove({server: 999});
● Update: db.scores.update({a: 5}, {server:999});
● Query: db.scores.find();
MongoDB auditing options● Server log
● MongoDB Enterprise Auditing
● Query to the oplog
● Mongosniff
MongoDB Mongosniff● Gives you detailed output of operations in MongoDB
● Does not come in the default package, you need to compile it
● Uses the mongo libraries to parse the commands
● Sample output: 111.22.33.44:6612 <<– 22.33.44.55:42947 262 bytes id:6a89eb 6982123 –
308293
reply n:4 cursorId: 0
{ _id: “db”, partitioned: false, primary: “Segmon_RS1″ }
MongoDB Mongosniff
(Modified)● Some pcap tweaks to reduce dropped packets
● Minor bug fixes
● Different output format:
172.31.36.172:56228,54.68.230.224:6612,test.$cmd,,query,{ authenticate: 1, nonce:
"745ad1e4a6075a25", user: "workshop", key: "869c8d69703e2d1bb9394ddf4c116dcb" }
ntoreturn: 1 ntoskip: 0AAAAAAA
MongoDB Mongosniff
Wrapper● Ruby wrapper
● Handles extra functions we would need without modifying mongosniff
● Output format:
Oct 29 03:43:11,workshop,54.68.230.224,workshop,test,172.31.36.172:56231, 54.68.230.224:6612,
test.cmd,,query,{ isMaster: 1.0, forShell: 1.0 } ntoreturn: -1 ntoskip: 0
MongoDB Mongosniff
Architecture
Mongo Sniffer DEMO
MongoDB Sniffer Problems● No support for packet fragmentation
● Not 100% reliable
● Not it’s intended use
References
● https://github.com/89berner/MysqlAudit
● https://github.com/mcafee/mysql-audit
thank you!
we’re hiring ;)
