log management with open source tools
TRANSCRIPT
-
8/10/2019 Log Management With Open Source Tools
1/21
Log Management with Open-Source Tools
Risto Vaarandirvaarandi 4T Y4H00 0T !0M
-
8/10/2019 Log Management With Open Source Tools
2/21
-
8/10/2019 Log Management With Open Source Tools
3/21
"h# collect logs %rom #our &Ts#stem and networ'$
O(servation ) logs contain in%ormation which is o%tennot availa(le %rom other sources
Real-time monitoring ) anal#*e logs in real-time +ornear-real-time, %ashion in order to discover importantchanges in the state o% the &T s#stem
.ost-%actum incident anal#sis ) leverage collecteddata %or discovering un'nown past incidents andgetting detailed insights into them
-
8/10/2019 Log Management With Open Source Tools
4/21
"h# use open source tools %or logmanagement$
!ommercial S&/M and log management %ramewor's
man# %ramewor's are consultant-oriented ) have comple1 design andinsu%%icient documentation
prohi(itive deplo#ment and licensing costs
man# %ramewor's repeat a num(er o% design mista'es o% networ'management solutions +made almost two decades ago2,
.ast e1perience with networ' management solutions
.hase 3 initial mar'eting h#pe %ollowed (# a num(er o% success storiesin the conte1t o% large and wealth# institutions
.hase disappointment among man# potential customers +%aileddeplo#ments prohi(itive pricing etc5, and search %or alternatives
.hase6 appearance o% well-designed open-source solutions which(ecome widel# used and ac'nowledged especiall# (# small- and mid-si*e enterprises
-
8/10/2019 Log Management With Open Source Tools
5/21
Traditional log collection protocols
The scene o% log collection protocols was relativel#sta(le %or two decades
7S s#slog ) the onl# cross-vendor protocol designedspeci%icall# %or logging
8. (ased plainte1t thus resource-e%%icient (utunrelia(le and not secure
Simple message la#out in the 8. %rame ) priorit#
simple timestamp host name program nameunstructured message te1t
-
8/10/2019 Log Management With Open Source Tools
6/21
9ew log collection protocols
&/T: s#slog +00;, ) support %or including structureddata in messages 8. and T!. (ased transportencr#ption and authentication detailed timestamps
!// +!ommon /vent /1pression, logging standard+03, ) use /L: S9M.trap messages etc5
-
8/10/2019 Log Management With Open Source Tools
7/21
/1amples
? Traditional 7S s#slog ) priorit# value @ encapsulates %acilit# value 6? +daemon, and severit# value 4 +warning, 6A@ B 4 C @
D@E9ov 3F 366G;m#hostids3;;I port scan from 192.168.1.102
? &/T: s#slog ) note high granularit# timestamps with time*one in%ormation
? and two (loc's o% structured data
D@E3 03-33-3FT366G;56B000 m#hostids3;;-timeJualit# t*KnownC3 isS#ncedC3Iorigin ipC3053535Iport scan from 192.168.1.102
? !// message %ormat ) use standard 7S s#slog message %or transporting? structured data in
-
8/10/2019 Log Management With Open Source Tools
8/21
"h# pass structured data in logmessages$
8nstructured message %ields o%ten contain additionalin%ormation a(out event which needs to (e highlighted
&t is much easier to parse structured data +'e#word-value pairs, than unstructured %ree-%ormat strings
Some structured data can (e used without e1traparsing )
-
8/10/2019 Log Management With Open Source Tools
9/21
9
Log collection on Linu1 plat%orm
local
programs
s#slog
server
networ'port
=dev=log
incoming messages
messages to
remote s#slogservers
messages %rom other nodes
=etc=s#slog-server5con%
con%iguration
=proc='msg
'ernel
=var=log=555
to local log%ilesopenlog+6,s#slog+6,555
d( >8&
-
8/10/2019 Log Management With Open Source Tools
10/21
S#slog servers ) rs#slog
http==www5rs#slog5com
B %ast message processing e%%icient multithreading designed to handleat least 3G0-00K messages per second +see the paper QRs#slog goingup %rom 40K messages per second to G0K (# Rainer >erhards %romLinu1 Kongress 030,
B (ac'wards compati(le with 89& s#slogd con%iguration directives
B has a num(er o% uniue %eatures and advantages over competitors+dis' (ased (u%%ers support %or /lasticsearch data(ase etc5,
- documentation could (e (etter
- con%iguration language has a non-intuitive s#nta1
- %iltering conditions can not (e named which prevents their reuse
-
8/10/2019 Log Management With Open Source Tools
11/21
S#slog servers ) s#slog-ng
http==www5(ala(it5com=networ'-securit#=s#slog-ng=
B a %le1i(le and reada(le con%iguration language which allows %orspeci%#ing comple1 con%igurations
B single-threaded until the 65 version (ut multi-threading has (een
introduced into recent versions which considera(l# improves scala(ilit#and per%ormance
B well documented
- open-source edition does not support dis' (ased (u%%ers
- no support %or /lasticsearch +although could (e con%igured through a
sel%-developed output plugin,
-
8/10/2019 Log Management With Open Source Tools
12/21
S#slog servers ) n1log
http==n1log-ce5source%orge5net=
B native support %or "indows plat%orm and "indows /vent Log
B supports the use o% em(edded .erl constructs %or message processing
B supports a num(er o% input and output t#pes not supported (#
competitors +e5g5 accepting input events %rom SJL data(asesproducing output events in >/L: %ormat etc5,
- poor message %iltering per%ormance
-
8/10/2019 Log Management With Open Source Tools
13/21
/lasticsearch 7 %or logmanagement
http==www5elasticsearch5org=
Upache Lucene (ased noSJL data(ase technolog# that is %reuentl#used %or storing log data
native support %or distri(uted operations and (uilding clusters
allows %or splitting inde1es into parts +shards, and distri(uting shardsover several nodes +e5g5 split an inde1 into shards and distri(ute themover nodes turning dis's at individual nodes into a single logicalstorage space,
inde1es can (e con%igured to have one or more replicas which increases%ault tolerance +e5g5 split an inde1 into shards and con%igure the inde1to have 3 replica and distri(ute resulting 4 shards across 4 nodes,
(uiltin support %or data compression +important when storing largevolumes o% log data,
supported (# several log management tools +Ki(ana >ra#loglogstash rs#slog,
-
8/10/2019 Log Management With Open Source Tools
14/21
Log management tools ) Ki(ana
http=='i(ana5org=
Ki(ana is a >8& %or searching log data stored into /lasticsearch 7
Ki(ana is designed to wor' with logstash log preprocessing tool (ut canaccept data %rom an# other tool which is a(le to store it to /lasticsearch
in a recogni*a(le wa# +e5g5 rs#slog, Ki(ana is lightweight written in Ru(# accessi(le over HTT. and
contains onl# searching and reporting %unctionalit# +e5g5 userauthentication and SSL connectivit# has to (e accomplished withe1ternal tools li'e Upache reverse pro1#,
"hen (uilding a Ki(ana (ased log management solution #ou arecreating the s#stem %rom well-documented and well-esta(lished (uilding(loc's and thus having the opportunit# %or man# customi*ations duringinitial installation and later maintenance
-
8/10/2019 Log Management With Open Source Tools
15/21
Ki(ana we( inter%ace
-
8/10/2019 Log Management With Open Source Tools
16/21
Log management tools ) >ra#log
http==gra#log5org=
U %ull log management solution consisting o% a server %or log messagereception +s#slog >/L:, and a >8&
The >8& is user-%riendl# with (uiltin help and is intuitive to use
Man# con%iguration tas's +such as setting log data retention intervalsetc5, can (e accomplished through a we( inter%ace
>ra#log supports users with di%%erent roles and passwordauthentication
/arlier versions o% >ra#log emplo#ed single-server approach which
limited the s#stem scala(ilit# while most recent versions allow to runseveral servers in parallel
-
8/10/2019 Log Management With Open Source Tools
17/21
>ra#log we( inter%ace
-
8/10/2019 Log Management With Open Source Tools
18/21
Other log management tools
Logstash +http==www5logstash5net=, - has a we( inter%ace %orsearching logs stored to /lasticsearch data(ase (ut since itsupports large num(er o% input and output t#pes it is mostl#used as a log parsing and conversion tool
/LSU +http==code5google5com=p=enterprise-log-search-and-archive=, - a log management s#stem which is (uilt on top o%s#slog-ng M#SJL and Sphin1
-
8/10/2019 Log Management With Open Source Tools
19/21
9et%low protocol
.roposed (# !isco in 3;;0s nowada#s supported (#man# maor vendors
U 9et%low-ena(led networ' device +e5g5 router switchdedicated pro(e, collects networ' tra%%ic statistics ande1ports it to collector over 8.
Tra%%ic statistics consists o% %low records where eachrecord descri(es some network flow
9etwor' %low ) unidirectional seuence o% pac'etswhich share transport protocol source and destination&. source and destination port and %ew otherparameters +e5g5 t#pe o% service,
-
8/10/2019 Log Management With Open Source Tools
20/21
/1ample o% collected 9et%low data
The %ollowing two records represent a success%ull# negotiatedand completed T!. connection %rom client 30565353 port 4@@;to the HTT. service +port @0, running at the server 3055353
StartC 036-0-3@ 00040G5F66DurationC 05034
T!. 305653534@@; -E 3055353@0
TCPflagsC 5U.5S:PacketsC GBytesC G36
StartC 036-0-3@ 00040G5F64DurationC 05030
T!. 3055353@0 -E 305653534@@;
TCPflagsC 5U.5S:PacketsC 4BytesC 6FG
-
8/10/2019 Log Management With Open Source Tools
21/21
How to collect=use net%low data
/na(le 9et%low collection at #our networ' device oruse dedicated pro(es +e5g5 %pro(e,
Open-source so%tware pac'ages %or collecting 9et%low
9%Sen +http==n%sen5source%orge5net=, SiLK +http==tools5netsa5cert5org=sil'=,
:low-tools +http==www5splintered5net=sw=%low-tools=, - unmaintained
"hat #ou might (e interested in %inding in 9et%low data
:lows with unusual com(inations o% T!. %lags +e5g5 :&9 without U!K, :lows which represent connections to=%rom 'nown (ad &. addresses
8ne1pected spi'es in tra%%ic volumes +measured in num(er o% (#tespac'ets %lows, associated with certain sources +e5g5 %oreign &.addresses or (ad &. addresses,