eko10 workshop opensource database auditing

OPEN SOURCE DATABASE MONITORING AT SCALE

#eko10

What is this talk about?● Auditing tools available for databases, commercial and otherwise

● What problems they have

● How we can make them scale as much as we might need

About us

● Juan Bernero @89berner / 89berner@gmail.com

o Hobbies = [‘Movies/Series','Reading','Programming']

o Mostly Blue Team

o http://secureandscalable.wordpress.com/

● Pablo Garbossao @pgarbossa / pablo.garbossa@gmail.com

o Fully Blue Team

About MercadoLibre● Devops culture (everyone and their mothers can access the boxes)

● Different DBs technologies

● Hybrid Cloud

● Database servers > 1K && Servers > 15K

● More than 100000 qps

Commercial products● Expensive

● Lots of functionalities you might not need

● Don’t scale so well

● Will make you choose what to log

Audit options● Inline / TAP / sensors or agents

● Plugin based

● Sniffers

● Client loggers

Mysql Audit Options● Commercial products

● Mysql General Log

● MySQL Enterprise Audit Log Plugin

● Mysql audit plugins

● Mysql sniffer

Mysql General Log● Easy to activate, by default in mysql

● Can be customized by modifying the log table to a degree

● As of 5.1 can be activated on the fly

● Less freedom than audit plugins

MySQL Enterprise Audit

Log Plugin● Available for Mysql Enterprise

● Uses the open MySQL Audit API

● Does not log triggers or prepared statements

● Allows asynchronous or synchronous logging

Mysql Audit Plugin (1)● Works using API created by Mysql to replace the general log

● Available in Github

● Flexibility to choose objects to inspect, types of queries to log or users to

whitelist

● Similar restrictions as Mysql Enterprise Plugin

Mysql Audit Plugin (2)● Steps to audit

o Download the plugin from github

o Move the library to /usr/lib/mysql/plugin/

o Enable with INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so';

o Either send it to a file or to a socket

Mysql Audit Plugin (3)● The log now looks like:

"msg-type":"activity","date":"1414531661274","thread-id":"72","query-

id":"1600563","user":"workshop","priv_user":"workshop","host":"ip-172-31-32-202.us-west-

2.compute.internal","ip":"172.31.32.202","cmd":"show_fields","query":"show_fields"

Mysql Audit Plugin (4)We can parse it with logstash into:

{

….

"@timestamp": "2014-10-29T04:10:37.000Z",

"type": "mysqlplugin",

"host": "0.0.0.0",

"path": "/var/log/mysqlplugin-2014-10-29.log",

"date": "Oct 29 04:10:37",

"agent": "54.200.106.239",

"user": "workshop",

"priv_user": "workshop",

"srcip": "54.69.169.73",

"command": "show_fields\",\"",

"query": "show_fields"

}

}

Mysql Audit Plugin DEMO

Problems with the Plugin● Generates overhead on the host

● Can’t log all events (audit api limitations)

● Not available in sniffing only situations

Sniffing options● Span port which sends you the traffic

● Sniffing and parsing from the server

● Forwarding the traffic from the agent (ie: iptables)

● Using agents to sniff traffic and forward it to repeaters (which repeat locally

the traffic with the original address)

Mysql Sniffer● Client / Server architecture

● Sniffs for common queries (select/insert/update/delete)

● Beta phase

● Has to keep up with protocol changes

Mysql Sniffer Agent● It will sniff traffic on Mysql port 3306 and send it elsewhere

● Small use of resources

● Must be tweaked to work in high load situations (ie: increase buffer for

packets to be processed)

./agent eth0 3306 DESTINATION 9200 1000 5000 5

Mysql Sniffer Repeater● Application that listens at a tcp port for connections

● Receives packets and does a local replay of them

● Packets are seen as coming from the original client

./repeater 9200

Mysql Sniffer Parser● Listens to traffic on the interface for the mysql port

● Parsers queries and keeps track of connections

● Writes output to logfile:

Wed Oct 29 00:20:24,54.69.169.73,55981,172.31.32.202,workshop,test,

select,"select * from test"

Mysql Sniffer DEMO

Mysql Sniffer Problems● Not reliable

● Depends on the protocol not to change or something weird not to happen

● Only a limit subset of types of queries which represent most queries

● Shouldn’t be used on databases with small activity

You can use a combo● Mysql sniffer to audit common queries without giving overhead to the

mysql server

● Mysql Audit Plugin to audit all other queries or specific objects with more

reliability

● Be creative

MongoDB● Document oriented database

● Great scaling capabilities

● Bson Data Store

● Most popular NoSQL (according to wikipedia)

MongoDB Operations● Insert: db.scores.save({a:99})

● Delete: db.scores.remove({server: 999});

● Update: db.scores.update({a: 5}, {server:999});

● Query: db.scores.find();

MongoDB auditing options● Server log

● MongoDB Enterprise Auditing

● Query to the oplog

● Mongosniff

MongoDB Mongosniff● Gives you detailed output of operations in MongoDB

● Does not come in the default package, you need to compile it

● Uses the mongo libraries to parse the commands

● Sample output: 111.22.33.44:6612 <<– 22.33.44.55:42947 262 bytes id:6a89eb 6982123 –

308293

reply n:4 cursorId: 0

{ _id: “db”, partitioned: false, primary: “Segmon_RS1″ }

MongoDB Mongosniff

(Modified)● Some pcap tweaks to reduce dropped packets

● Minor bug fixes

● Different output format:

172.31.36.172:56228,54.68.230.224:6612,test.$cmd,,query,{ authenticate: 1, nonce:

"745ad1e4a6075a25", user: "workshop", key: "869c8d69703e2d1bb9394ddf4c116dcb" }

ntoreturn: 1 ntoskip: 0AAAAAAA

MongoDB Mongosniff

Wrapper● Ruby wrapper

● Handles extra functions we would need without modifying mongosniff

● Output format:

Oct 29 03:43:11,workshop,54.68.230.224,workshop,test,172.31.36.172:56231, 54.68.230.224:6612,

test.cmd,,query,{ isMaster: 1.0, forShell: 1.0 } ntoreturn: -1 ntoskip: 0

MongoDB Mongosniff

Architecture

Mongo Sniffer DEMO

MongoDB Sniffer Problems● No support for packet fragmentation

● Not 100% reliable

● Not it’s intended use

References

● https://github.com/89berner/MysqlAudit

● https://github.com/mcafee/mysql-audit

thank you!

we’re hiring ;)