Logging with Netfilter and ulogd2

Éric Leblond

Stamus Networks

June 23, 2015

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 1 / 17

Eric Leblond a.k.a Regit

FrenchNetwork security expertFree Software enthousiastNuFW project creator (Now ufwi), EdenWall co-founderNetfilter developer:

Maintainer of ulogd2: Netfilter logging daemonMisc contributions:

NFQUEUE library and associatesPort of some features iptables to nftables

Currently:co-founder of Stamus Networks, a company providing Suricatabased network probe appliances.Suricata IDS/IPS funded developer

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 2 / 17

Packet logging

Syslog loggingFlat packet loggingOne line per packetUse printk kernel facility

Syntax

i p t a b l e s −A INPUT −p tcp −−dpor t 25 −−syn \− j LOG −−log−p r e f i x "SMTP access "

Not sexyINPUT DROP IN=eth0 OUT= MAC=00:1a:92:05:ee:68:00:b0:8e:83:3b:f0:08:00 \

SRC=62.212.121.211 DST=91.121.73.151 LEN=60 TOS=0x00 PREC=0x00 \TTL=58 ID=35342 DF PROTO=TCP SPT=59261 DPT=113 WINDOW=5440 RES=0x00 SYN URGP=0

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 3 / 17

Defensive security

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 4 / 17

ULOG

Socket base messagingNetlink based communicationDifferent groupsBatching systemIPv4 only

Syntax

i p t a b l e s −A INPUT −p tcp −−dpor t 25 −−syn \− j ULOG −−ulog−p r e f i x "SMTP access " \−−ulog−nlgroup 2−−ulog−qthresho ld 10

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 5 / 17

ulogd daemon

A logging daemonListen to eventStore event in various formats

Flat fileDatabases

Ulogd outputsLOGEMUOPRINTMySQLPostgresqlsqlite3pcap

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 6 / 17

ulogd: first interfaces

Using SQL backendProviding a dashboardNulog released 14 Apr2000

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 7 / 17

2.6.14: the nfnetlink revolution

NfnetlinkFirst major evolution of Netfilter (Linux 2.6.14, 2005)Netfilter dedicated configuration and message passingmechanism

New interactionsNFLOG: enhanced logging systemNFQUEUE: improved userspace decision systemNFCT: get information and update connection tracking entries

Based on Netlinkdatagram-oriented messaging systempassing messages from kernel to user-space and vice-versa

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 8 / 17

ulogd2

Ulogd reloadedInteract with the post 2.6.14 librariesFirst release on 01 Feb 2006Multiple output and input through the use of stacks

Stack examplestack=log2:NFLOG,mark1:MARK,base1:BASE,ifi1:IFINDEX,ip2bin1:IP2BIN,\

mac2str1:HWHDR,mysql1:MYSQLstack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,\

mac2str1:HWHDR,pgsql1:PGSQL

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 9 / 17

ulogd2: packet logging

Nothing really newOne ulogd2 can handle multiple logging inputMultiple output is also supported

But improved databasesMagical schema discoveryBetter schemaInsertion via SQL procedure

It is possible to create custom logging in SQLNo need to know C

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 10 / 17

ulogd2: connection logging

InterestsLog volume of exchange dataLog NAT transformation

Ulogd2 supportFile and database outputstack=ct2:NFCT,ip2str1:IP2STR,pgsql2:PGSQL

More info https://home.regit.org/2014/02/logging-connection-tracking-event-with-ulogd/

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 11 / 17

nfacct

nfacctEfficient accounting systemAppeared in 2012

In ulogdDump nfacct counter at regular intervalRealize storage

XMLPostgresqlGraphite

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 12 / 17

ulogd2: graphite

GraphiteScalable Realtime GraphingBased on rrdtoolsAllow to combine datahttp://graphite.wikidot.com/start

AlternateFrontend: GrafanaBackend: Influxdb

Ulogd2 configurationstack=acct1:NFACCT,\

graphite1:GRAPHITE

[acct1]pollinterval = 2

[graphite1]host="127.0.0.1"port="2003"

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 13 / 17

ulogd2: graphite

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 14 / 17

JSON output

JSON formatFormatted messageSchema lessEasy to use in code and toolsIntegration with Splunk or Elasticsearch

JSON pluginUse ulogd key, value systemTranslation to text of key is enoughUsable for all input plugins

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 15 / 17

Demo: Ulogd + Kibana

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 16 / 17

Conclusion

Ulogd2 brings complete logging to NetfilterPackets loggingConnection tracking loggingAccounting

More informationNetfilter: http://www.netfilter.org/My blog: https://home.regit.org/Stamus Networks: https://www.stamus-networks.com/

Éric Leblond (Stamus Networks) Logging with Netfilter and ulogd2 June 23, 2015 17 / 17