real-time botnet detection using nonnegative tucker … · 2020-05-01 · real-time botnet...
TRANSCRIPT
![Page 1: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/1.jpg)
Real-Time Botnet Detection Using Nonnegative Tucker Decomposition
2019/4/8-12 SAC2019
Hideaki Kanehara1,2 Yuma Murakami1 Jumpei Shimamura3
Takeshi Takahashi2 Daisuke Inoue2 Noboru Murata1,2 1: Waseda University 2: National institute of information and communications technology 3: Clwit Inc.
![Page 2: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/2.jpg)
!2
1. Background
2. Methodology
Factorization-based method
Real-time tensor factorization
Botnet detection using NTD
3. Experiment
Experimental setting
Result
• NTD visualization
• Comparison with the actual traffic
• Related incident
Outline
![Page 3: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/3.jpg)
!3
1. Background
2. Methodology
Factorization-based method
Real-time tensor factorization
Botnet detection using NTD
3. Experiment
Experimental setting
Result
• NTD visualization
• Comparison with the actual traffic
• Related incident
Outline
![Page 4: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/4.jpg)
!4Background | Darknet
early detection of cyber attacks is essential
botnet: a group of infected devices that are remotely controlled by attackers
-> DDoS attacks are often performed by botnets
![Page 5: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/5.jpg)
unused IP address space
but in reality, a lot of malicious packets arrive
• a network scanning (for spreading malware infection)
tells us the malicious trend of the wide area of the Internet without having to monitor the overall hosts
!5Background | Darknet
darknet
early detection of cyber attacks is essential -> DDoS attacks are often performed by botnets
![Page 6: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/6.jpg)
Timestamp Src IP Src port Dst port …
12:34:56 12.125.x.x 37721 25 …
12:34:56 252.156.x.x 52521 23 …
12:34:57 123.35.x.x 25162 8888 …
12:34:58 156.33.x.x 12732 3218 …
… … … …
An example of the darknet traffic
Background | Darknet !6
Cooperative behavior (botnet) detectionPurpose
packet->
![Page 7: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/7.jpg)
An example of the darknet traffic
Background | Research purpose
Cooperative behavior (botnet) detection
!7
an activity of a host group using almost the same port numbers at almost the same time/frequency
Purpose
Timestamp Src IP Src port Dst port …
12:34:56 12.125.x.x 37721 25 …
12:34:56 252.156.x.x 52521 23 …
12:34:57 123.35.x.x 25162 8888 …
12:34:58 156.33.x.x 12732 3218 …
… … … …
![Page 8: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/8.jpg)
An example of the darknet traffic
Background | Research purpose
Cooperative behavior (botnet) detection
!8
an activity of a host group using specific port numbers
at almost the same time/frequency
Purpose
Timestamp Src IP Src port Dst port …
12:34:56 12.125.x.x 37721 25 …
12:34:56 252.156.x.x 52521 23 …
12:34:57 123.35.x.x 25162 8888 …
12:34:58 156.33.x.x 12732 3218 …
… … … …Also, we want to know … - Where are they from (src IP) - What is their aim (dst port)
![Page 9: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/9.jpg)
!9
1. Background
2. Methodology
Factorization-based method
Real-time tensor factorization
Botnet detection using NTD
3. Experiment
Experimental setting
Result
• NTD visualization
• Comparison with the actual traffic
• Related incident
Outline
![Page 10: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/10.jpg)
!10Overview of our proposed method
Timestamp Src IP Dst Port
12:34:56 12.125.x.x 2512:34:56 252.156.x.x 23
… … …
30min
Data input stage
Feature extraction stage
Group activity detection stage
‣ Preprocessing darknet traffic
‣ Extracting frequent patterns ‣ Real-time Tensor factorization • LRA-NTD • FSTD
‣ thresholding to raise alerts ‣ If group activities exist, then trying to identify their src IPs and dst ports
30 × 216 × 216
![Page 11: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/11.jpg)
Timestamp Src IP Dst port
12:34:56 12.125.x.x 25
12:34:56 252.156.x.x 23
12:34:57 123.35.x.x 8888
12:34:58 156.33.x.x 3218
… … …
!11Data input stage | Tensor
Input data can be represented as a tensor (multidimensional array) Timestamp・Src IP・Dst Port
30 × 216 × 216
Cooperative behavior (botnet) detectionPurpose
min IP port⇡ 1011
elements
![Page 12: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/12.jpg)
Timestamp Src IP Dst Port
12:34:56 12.125.x.x 25
12:34:56 252.156.x.x 23
12:34:57 123.35.x.x 8888
12:34:58 156.33.x.x 3218
… … …
!12Data input stage | Tensor
Cooperative behavior (botnet) detectionPurpose
Botnet A
Botnet B
Botnet C
Botnet D
Input data can be represented as a tensor (multidimensional array) Timestamp・Src IP・Dst Port
30 × 216 × 216min IP port
⇡ 1011elements
![Page 13: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/13.jpg)
Timestamp Src IP Dst Port
12:34:56 12.125.x.x 25
12:34:56 252.156.x.x 23
12:34:57 123.35.x.x 8888
12:34:58 156.33.x.x 3218
… … …
!13Why factorization?
Input data can be represented as a matrix Timestamp・Src IP・Dst Port
30 × 216 × 216min IP port
Simplify the problem:
Grouping similar hosts from src IP and dst Port
Source IP
![Page 14: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/14.jpg)
!14Why factorization?
Src IP Dst Port
12.125.x.x 22
12.125.x.x 23
252.156.x.x 80
252.156.x.x 8080… …
Simplify the problem:
Grouping similar hosts from src IP and dst Port
12.125.x.x 252.156.x.x
…
22 23 80 … dst Port
src IP
x x
x x
x x
x x
…
![Page 15: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/15.jpg)
!15Why factorization?
12.125.x.x 252.156.x.x
…
22 23 80 … dst Port
src IP
x x
x x
x x
x x
=…
x xx x
x x
x x
x x
x x
5.12.x.x 5.15.x.x
…
80 81 8080 …
pairwise comparison sorting
Simplify the problem:
Grouping similar hosts from src IP and dst Port
![Page 16: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/16.jpg)
!16Why factorization?
12.125.x.x 252.156.x.x
…
22 23 80 … dst Port
src IP
x x
x x
x x
x x
=…
Simplify the problem:
Grouping similar hosts from src IP and dst Port
-> One Solution: apply the matrix factorization
x
x
22 23 80 …
x xx xx
x
: #basis vectors = 2
rr
r
dst Port
extracting important patterns
![Page 17: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/17.jpg)
!17Why factorization?
12.125.x.x 252.156.x.x
12.126.x.x
5.188.x.x
dst Port
src IP
x x
x x
x x
x x
…
Simplify the problem:
Grouping similar hosts from src IP and dst Port
x
x
x xx x
x
x
22 23 80 8080
-> One Solution: apply the matrix factorization
![Page 18: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/18.jpg)
!18Why factorization?
dst Port
x x
x x
x x
x x
…
Simplify the problem:
Grouping similar hosts from src IP and dst Port
x
x
x xx x
x
x
group A
12.125.x.x 252.156.x.x
12.126.x.x
5.188.x.x
src IPa basis vector ≒ frequent pattern
22 23 80 8080
-> One Solution: apply the matrix factorization
![Page 19: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/19.jpg)
!19Why factorization?
dst Port
x x
x x
x x
x x
…
Simplify the problem:
Grouping similar hosts from src IP and dst Port
x
x
22 23 80 8080
x xx x
x
x
12.125.x.x 252.156.x.x
12.126.x.x
5.188.x.x
a basis vector ≒ frequent traffic patterngroup B
src IP
-> One Solution: apply the matrix factorization
group A
![Page 20: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/20.jpg)
!20Why factorization?
dst Port
x x
x x
x x
x x
…
Simplify the problem:
Grouping similar hosts from src IP and dst Port
x
x
22 23 80 8080
x xx x
x
x
12.125.x.x 252.156.x.x
12.126.x.x
5.188.x.x
a basis vector ≒ frequent traffic pattern
src IP
-> One Solution: apply the matrix factorization
![Page 21: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/21.jpg)
Tensor factorization
higher-order extension of matrix factorization
!21Tensor factorization
22 23 80 … dst Port
src IP
=12.125.x.x 252.156.x.x
12.126.x.x
5.188.x.x
?
…
time
x x
x x
x x
x x
…
![Page 22: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/22.jpg)
: a factor matrix, set of frequent patterns
: a core tensor
: #basis vectors of mode n The larger, the better <=> computational cost
!22Tucker decomposition
G
A(1)
A(2)
�A(3)
Y
Y ⇡ G ⇥1 A(1) ⇥2 A
(2) ⇥3 A(3)
⇥n : mode-n product
R1
R2
R3
G
A(n)
Rn
Nonnegativity constraint -> Nonnegative Tucker Decomposition (NTD)
mode 1
mode 2
mode 3
weights
![Page 23: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/23.jpg)
!23Real-time NTD implementation
Y
G
A(1)
A(2)
A(3)
NTD
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
NTD [Kim+,2007]
A(n) A(n) � ⌘A(n) ~ @D
@A(n)
G G � ⌘G ~ @D
@G
![Page 24: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/24.jpg)
!24Real-time NTD implementation
Y
G
A(1)
A(2)
A(3)
NTD
✕minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
NTD [Kim+,2007]
A(n) A(n) � ⌘A(n) ~ @D
@A(n)
G G � ⌘G ~ @D
@GY 2 R30⇥216⇥216
![Page 25: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/25.jpg)
!25Real-time NTD implementation
Y
G
A(1)A(2)
A(3)
G
A(1)
A(2)
A(3)
NTD
✕
FSTD [Caiafa+,2010]one of the fastest decomposition: sampling the important fibers
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
NTD [Kim+,2007]
Fiber Sampling Tensor Decomposition
![Page 26: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/26.jpg)
!26Real-time NTD implementation
Y
G
A(1)A(2)
A(3)G
A(1)
A(2)
A(3)
NTD
G
A(1)
A(2)
A(3)
NTD
✕
Y = G ⇥1 A(1) ⇥2 A(2) ⇥3 A(3)
LRA-NTD [Zhou+,2015]efficient NTD based on two-step algorithm
FSTD [Caiafa+,2010]one of the fastest decomposition: sampling the important fibers
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
NTD [Kim+,2007]
Low-Rank Approximation NTD
![Page 27: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/27.jpg)
!27Real-time NTD implementation
LRA-NTD [Zhou+,2015]efficient NTD based on two-step algorithm
FSTD [Caiafa+,2010]one of the fastest decomposition: sampling the important fibers
Y
G
A(1)A(2)
A(3)G
A(1)
A(2)
A(3)
NTD
G
A(1)
A(2)
A(3)
NTD
✕
Y = G ⇥1 A(1) ⇥2 A(2) ⇥3 A(3)
NTD [Kim+,2007]
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
minG,A
1
2||Y � G ⇥1 A
(1) ⇥2 A(2) ⇥3 A
(3)||2
Our NTD implementation
#basis vectors … NTD: FSTD: 25 CPU … Intel Xeon X5600 (2.8GHz)
R1 = R2 = R3 = 5
• computational time: 70-90s
• memory usage: 1.9-2.5G
![Page 28: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/28.jpg)
!28Feature extraction using NTD
⇡
weight of each pattern
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Index
Index
Index
IndexWlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
30 0.0.*.* 255.255.*.* 0 655351
![Page 29: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/29.jpg)
results can be visualized like bipartite graphs
edge: core tensor values / node: basis vectors
!29Botnet detection | NTD visualization
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Index
Index
Index
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
30 0.0.*.* 255.255.*.* 0 65535time
patternsIP
patternsport
patterns
1
![Page 30: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/30.jpg)
Identify src IPs/dst ports of coordinated group
!30Botnet detection | thresholding
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Index
Index
Index
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Index
Wlist[[d]][, f]
Wlist[[d]][, f]
30 0.0.*.* 255.255.*.* 0 65535
(B)
(A)
(B)multiple IPs exceed a threshold -> they might be
cooperative (botnet)
(A)(B)
1
corresponding port numbers that exceed a threshold -> their attack destination
time patterns
IP patterns
port patterns
botnet IPs
![Page 31: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/31.jpg)
!31
1. Background
2. Methodology
Factorization-based method
Real-time tensor factorization
Botnet detection using NTD
Experiment
Experimental setting
Result
• NTD visualization
• Comparison with the actual traffic
• Related incident
Outline
![Page 32: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/32.jpg)
!32Experimental settings
[1] NICTER, Inoue et al., 2008
Input: darknet traffic (TCP/UDP) in different countries [1] (#IP addresses ≒ 35k)
Output: src IPs, dst ports of coordinated groups
real-time detection: apply NTD every 30 min
We introduce one of the interesting result and evaluate qualitatively
![Page 33: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/33.jpg)
factorized result of 5/9 5:30-6:00 TCP traffic
!33Result | NTD visualization
time src IP dst port30 0.0.*.* 255.255.*.* 0 655351
![Page 34: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/34.jpg)
!34Result | NTD visualization
~10000/TCP
30 0.0.*.* 255.255.*.* 0 655351time src IP dst port
factorized result of 5/9 5:30-6:00 TCP traffic
identifying the botnet IPs and their dst ports
6379/TCP
![Page 35: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/35.jpg)
original darknet TCP traffic (5/9 5:30-6:00)
!35Result | Original traffic
![Page 36: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/36.jpg)
original darknet TCP traffic (5/9 5:30-6:00)
filtered by botnet IPs
!36
dst port
src IP
color: difference in the basis vectors of IP characteristics
size: #packets/min
Result | Original traffic
![Page 37: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/37.jpg)
malicious group A: probing attack
!37
A total of 6318 ports
Result | Qualitative evaluation
0.0.*.*255.255.*.*
dst port
src IPIP charasterics
![Page 38: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/38.jpg)
malicious group B: exploiting some vulnerability?
!38
6379, 6380, 7379/TCP
Result | Qualitative evaluation
0.0.*.*255.255.*.*
dst port
src IPIP charasterics
![Page 39: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/39.jpg)
malicious group B: exploiting some vulnerability?
!39
6379, 6380, 7379/TCP
Result | Qualitative evaluation
0.0.*.*255.255.*.*
dst port
src IPIP charasterics
What did they attempt to do?
![Page 40: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/40.jpg)
On March 8, the research blog announced the malware that abuse known vulnerabilities of Redis server (listens on the port 6379 by default)
try to find vulnerable Redis servers by Internet-wide scanning
!40Result | Related incidents
https://www.imperva.com/blog/2018/03/rediswannamine-new-redis-nsa-powered-cryptojacking-attack/
![Page 41: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/41.jpg)
!41Result | Related incidents
On March 8, the research blog announced the malware that abuse known vulnerabilities of Redis server (listens on the port 6379 by default)
try to find vulnerable Redis servers by Internet-wide scanning
the diameter and the color of the points: #botnet IPs
after publication, our method continuously
detected group activities
summarized alerts on 6379/TCP
![Page 42: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/42.jpg)
We proposed a novel botnet detection method from darknet traffic
Nonnegative Tucker decomposition (NTD):
a powerful model for extracting co-occurrence patterns
-> but requires too high computational cost
Efficient NTD implementation enough to run in real-time LRA-NTD
FSTD
Demonstrated effectiveness by reviewing incidents
!42Conclusion
Future work
Improve the NTD algorithm
Our approach is very fast, but loses much information
Quantitative evaluation
![Page 43: Real-Time Botnet Detection Using Nonnegative Tucker … · 2020-05-01 · Real-Time Botnet Detection Using Nonnegative Tucker Decomposition 2019/4/8-12 SAC2019 Hideaki Kanehara1,2](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4f07f9836cd62ff01655c5/html5/thumbnails/43.jpg)
APPENDIX