experiences*in*traffic*logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_ku-log... ·...
TRANSCRIPT
![Page 1: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/1.jpg)
Surasak Sanguanpong
Department of Computer EngineeringFaculty of Engineering, Kasetsart University
Tech%Talk% Session,%WUNCA%33rd Chulalongkorn University,% July%14%2016
Experiences*in*Traffic*Logging*and*Visualization*with*ELK*and*D3.js
U-Bahn Station Candidplazt, Munich, Germany
![Page 2: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/2.jpg)
In This Talk
2
Real%Time%
Visualization
with%D3.js
Search%Platform%
with%ELKAbout%Traffic%Log Lessons%Learnt
![Page 3: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/3.jpg)
Log Monitoring
Collecting
Processing
Analysing
Visualising
3Image:%https://www.flickr.com/photos/sbeebe/4772418919
![Page 4: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/4.jpg)
At What Scale?
Hmm..Large..
4http://www.24hourcampfire.com/ubbthreads/ubbthreads.php/topics/5976731/all/That_s_a_load_of_logs
![Page 5: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/5.jpg)
Traffic Logging Solution
Splunk? Great, but..commercial, proprietary
Graylog?Excellence, but too automatic
Elasticsearch, Logtash, Kibana, D3That is!, a lot of fun to play
5
![Page 6: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/6.jpg)
Chapter I Log Architecture and
Raw Log Management: A Case Study
6
![Page 7: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/7.jpg)
Evolution of KU Traffic Logging Design
2008-2015 2015-
7
Raw Log
MySQL
Simple GUI
Raw Log
Elasticsearch
Kibana/D3
![Page 8: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/8.jpg)
Logging Architecture
8
Mirror packets
PacketLog
Web Log
Login Log
Network
Login/LogoutLogin
Search GUILogging Engine
![Page 9: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/9.jpg)
Login Log FormatDate Time Action IP UserName LogServer
Jul 1 10:04:57 login 158.108.X.X [email protected] 192.168.1.1Jul 1 10:04:58 logout 158.108.X.X [email protected] 192.168.1.2Jul 1 10:04:59 timeout 158.108.X.X [email protected] 192.168.1.2
9
![Page 10: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/10.jpg)
Web Log FormatUnixTime SrcIPv4 SrcIPv6 DstIPv4 DstIPv6 SrcPort DstPort URL Referer/HTTPS
20151103010000 192.55.X.X - 158.108.X.X - 17490 80 mirror1.ku.ac.th/fedora-epel/6/i386/jday-devel-2.4-5.el6.i686.rpm http://mirror1.ku.ac.th/fedora-epel/6/i386/
20151103010000 10.X.X.X - 203.104.175.X - 62635 80 sg-nvapis.line.me/ ping?&msgpad=1446487199964&md=9LMRXqv1Nb8P07aj0Vo%3D –
20151103010000 - 2406:3100:1018:1::XX - 2600:1417:a::174c:XX 61154 443 fbcdn-photos-g-a.akamaihd.net HTTPS
20151103010000 - 2406:3100:1018:1::XX - 2a03:2880:f002:105:fa:b0:0:YYXX 59960 443 edge-mqtt.facebook.com HTTPS
10
![Page 11: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/11.jpg)
Packet Log Format (Header Log)TimeStamp SrcIP DstIP SrcPort Proto Size DstPort SrcPort [Flag]2009-07-16 17:53:59.999206 208.117.8.X 158.108.234.X 1514 TCP 80 1371 0x10
2009-07-16 17:53:59.999209 158.108.2.X 202.143.136.X 90 UDP 123 123
TimeStamp SrcIP DstIP Proto Code2009-07-16 17:53:59.999210 158.108.184.X 218.164.54.X ICMP 168
11
![Page 12: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/12.jpg)
Example of Log Folder
Time basedHierarchical Folder
12
Minutes%FileHourDayMonthYear
2015
01
01
00
201501010000.txt201501010001.txt
:201501010059.txt01
::
23
201501012300.txt201501012301.txt
:201501012359.txt
02:
3002:
12
![Page 13: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/13.jpg)
Minutely HTTP Log
13
11"days"(11x"24x60="15,640"data"points)
![Page 14: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/14.jpg)
Request Rate and Log Sizing
14
![Page 15: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/15.jpg)
Accumulated Log Request and Size
15
#Files":"120
20M
2.04"GB
14.1B
2.57"TB
#Files":"172,800
28.03"TB
3.27T
#Files":"172,800
![Page 16: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/16.jpg)
Log Processing and Search Services
• On the fly Text based Log to MySQL converter
• Slow processing/ searching time
• Simple Search
16
![Page 17: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/17.jpg)
Chapter II ELK Stack Testbed
17
![Page 18: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/18.jpg)
What is the Elasticsearch?
18
Real\time
Search/Analytic
Engine%SW
Document\
Oriented
REST%API
&
JSON
Distributed ScalablePlugin
Architecture
JAVA/Lucenebased
Open"SourceApache"2"License
REST:%Representational%State%Transfer
JSON:%JavaScript%Object%Notation
![Page 19: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/19.jpg)
What does Elasticsearch offer?
19
Full%Text%Search Very%Fast Fault%Tolerance High%Availability
![Page 20: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/20.jpg)
How the world is using Elasticsearch?
20
Analytics solution on 40 million documents per day to deliver
real-time visibility
Providing search across GitHub's code
Full-text search to find related questions and answers
Full-text search with highlighted search snippets
![Page 21: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/21.jpg)
Elasticsearch and Big DataES-Hadoop: Connectivity of Hadoop's big data analytics and the real-time search of Elasticsearch.
21
https://www.elastic.co/products/hadoop
![Page 22: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/22.jpg)
ELK stack from Elastic
22
Elasticsearch: High-performance scalable search engine
Logtash: Log transport and processing daemon
ELK StackKibana: Visualisation dashboard
![Page 23: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/23.jpg)
Logtash
23
Log aggregator and parser
Transferring parsed data to Elasticsearch
Configuration file for specifying input, filtering
(parsing) and output
input%{%stdin {%}%}
filter% {%%
grok {%%%
match%=>% {%"message"% =>%"%{COMBINEDAPACHELOG}"% }%%
}%%
date%{%%%%match%=>% [%"timestamp"% ,"dd/MM/yyyy:HH:mm:ss"% ]%%
}
}
output%{%%Elasticsearch {%hosts%=>% ["localhost:9200"]%}%%
stdout {%codec%=>%rubydebug }}
![Page 24: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/24.jpg)
Kibana
24
General purpose query UI
Includes many widgets
Query Elasticsearch without coding
![Page 25: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/25.jpg)
Alternative Stack
25
ELK
EFK
![Page 26: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/26.jpg)
Elasticsearch Indexing Performance
26
35
36
37
38
39
40
41
42
43
44
45
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
THOUSANDS
MILLIONS
Daily**Performance*Indexing
#Records Records/s• Single Dell R220 • Xeon E3-1271v3 3.6
Ghz 4C/8T• 32 GB RAM• 2x6 TB NLSAS
• Elasticsearch2.3.2• 10 Shards/0 Replica• Hyper-threading off• Web Log Indexing
![Page 27: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/27.jpg)
Search PerformanceSearch keyword: “ face” against each daily log
Not yet Optimization
27
2.01
2.33
1.992.13
2.67
2.00
1.33
1.02
3.00
2.33
2.00
2.67
3.00
2.67
2.43
3.33
2.67
2.14
3.33
17,551
22,816
16,346
18,218
16,240
7,958
5,622
1,886
23,559
9,1278,221
12,343
28,259
25,405
22,092
33,528
17,683
12,951
18,054
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
SEARCH
"TIM
E"(M
S)
Search "Performance"and"Hits
Search%Time%(ms) Hits
![Page 28: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/28.jpg)
Kibana: Main Dashboard
28
![Page 29: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/29.jpg)
Kibana : per IP Log
29
![Page 30: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/30.jpg)
Kibana: Login Profile
30
![Page 31: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/31.jpg)
Kibana: Concurrent Login View
31
![Page 32: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/32.jpg)
Chapter III Playing with D3.js
32
![Page 33: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/33.jpg)
Real Time Visualization with D3.js
• Data-Driven Documents (D3)
• JavaScript library for manipulating documents based on data
• Developed by Mike Bostock
33
https://d3js.org/
![Page 34: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/34.jpg)
D3 Architecture! Input data to build
visualizations (JSON, CSV,…)
! Data manipulation of HTML elements dynamically with JavaScript
34
node.js
socket.io
![Page 35: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/35.jpg)
Sample Gallery
35
![Page 36: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/36.jpg)
Real-time makes impression
36
http://map.norsecorp.com/#/Norse%Live%Attack%Map%
![Page 37: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/37.jpg)
Concurrent Login
37
![Page 38: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/38.jpg)
IP Matrix Occupied
38
![Page 39: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/39.jpg)
Tree Map Web Access
39
![Page 40: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/40.jpg)
Traffic Connectivity
40
![Page 41: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/41.jpg)
Chapter IV New Log Design
41
![Page 42: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/42.jpg)
New Logging Architecture
42
Mirror packets
FlowLog
Web Log
Login Log
Network
Login/outeventLogin
Logging Engine
ElasticsearchReal time Indexing
Session"Tracking"&Accounting
DHCP,RADIUS
ElasticsearchGUI/
Analytics
![Page 43: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/43.jpg)
Logging Redesign
43
User"identificationLegal"Logging
Real^timeAccounting
UserSessionControl
TrafficAnalytics
SIEMSupports
PerformanceManagement
![Page 44: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/44.jpg)
New*Login*Log*Format• Real-time logging, one file per day• Fields
login_session_id user login_timestamplogout_timestamp mac_address ipv4 ipv6agent_ip agent_typevia_ip ipv4_byte_in ipv4_byte_outipv4_pkt_in ipv4_pkt_out ipv6_byte_in ipv6_byte_out ipv6_pkt_in ipv6_pkt_out
• Sample Log67686345 [email protected] 1467551484.163681 0 001122334455 192.0.2.1 2001:db8::1 203.0.113.5 login – 0 0 0 0 0 0 0 067686346 [email protected] 1467551490.524125 0 - 192.0.5.5 - 203.0.113.1 login – 0 0 0 0 0 0 0 067686345 [email protected] 1467551484.163681 1467551833.754636 001122334455 192.0.2.1 2001:db8::1 203.0.113.5 login – 234342 423442 5522 6622 233456 22334 445 665
![Page 45: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/45.jpg)
New*Web*Log*Format• Real-time logging, one file per minute• Fields
request_timestamp {flow link fields} {login link fields} {ip info fields}{tcp info fields} method host path referrer agent
• Sample Log554455 1467551484.180000 67686345 [email protected] 1467551484.1636814 192.0.2.1 198.51.100.1 tcp 5566 80 GET www.domain.com /index.html - “Linux”
![Page 46: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/46.jpg)
Traffic*Flow*log• Log commit periodically (Configurable 1 minute to 1 hour interval)• Fields
• flow_id flow_start_timestamp {segment info fields} {login link fields}• {ip info fields} {tcp info fields} {tcp additional info fields} {tcp stat fields}
• Sample Log554455 1467551484.180000 1467551484.180000 1467551492.954258 18 20 1628 25456 223344 f 67686345 [email protected] 1467551484.163681 4 192.0.2.1 198.51.100.1 tcp 5566 80 1 - -1428 1428 864 24522 3 17 2 2 0 30000 0 30000
![Page 47: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/47.jpg)
Chapter V Lessons Learned
47
![Page 48: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/48.jpg)
Lessons Learned
Elasticsearch offers a very fast full-text search services
Indexing size may 3x to 5x bigger than source data
Use Elasticsearch for search services, not for data archiving
48
![Page 49: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/49.jpg)
Lessons Learned
Logtash : A powerful tool to manipulate log
Kibana : Simple and useful for visualize data
49
![Page 50: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/50.jpg)
Lessons Learned
D3 prosFlexible, Facsinating Visualization
D3 consLow Level, Steep Learning Curve, CPU intensive
50
![Page 51: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/51.jpg)
Lessons Learned
Combination of Lawful Log,
Security information and event management (SIEM) and Accounting
51
![Page 52: Experiences*in*Traffic*Logging*and* …wunca.uni.net.th/wunca_regis/wunca33_doc/14/021_KU-Log... · 2016-07-21 · New*Web*Log*Format • Real-time logging, one file per minute •](https://reader034.vdocument.in/reader034/viewer/2022042221/5ec7f6e04759d0174b19ecba/html5/thumbnails/52.jpg)
Thank you for your attention
Q & A Time
Q&A…
52
Kasom Koth-Arsa
Core Log Design and DevelopmentJautuporn Chuchuay
Peerapol BoonthaganonWeb GUI Development
Sataporn TechaaramwongWeb/Elasticsearch Development
Peerapong Thongpubeth
Jiradech SirijantadilokKibana Development
Poomipat ThongudomNichapat Nattee
D3 DevelopmentSurachai ChitpinijyolProject Coordinator
Surasak SanguanpongProject Director
Special Thanks to Kasetsart Office of Computer Services for supporting traffic dataSunset at Narita Airport