automatic analysis of malware behavior using machine learningcavazos/cisc850-spring... · automatic...

25
Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius, Carsten Willems, Thorsten Holz Peng Su CISC850 Cyber Analytics

Upload: others

Post on 24-Aug-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Automatic Analysis of Malware Behavior using Machine Learning

Konrad Rieck, Philipp Trinius, Carsten Willems, Thorsten Holz

Peng Su

CISC850

Cyber Analytics

Page 2: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Automatic Analysis of Malware Behavior

• Malware threaten the Internet

• Dynamic VS Static • binary packers, encryption, or self-modifying code, to obstruct

analysis.

• behavior of malicious software during run-time.

CISC850 Cyber Analytics

Page 3: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Automatic Analysis of Malware Behavior

CISC850 Cyber Analytics

Page 4: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Monitoring of Malware Behavior

• Malware Sandboxes --CWSandbox

• Malware Instruction Set

CISC850 Cyber Analytics

Page 5: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Malware Instruction Set

• MIST instruction keep the stable and discriminative patterns such as directory and mutex name at the beginning.

CISC850 Cyber Analytics

Page 6: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Embedding of Malware Behavior

• Embedding using Instruction Q-grams

• Comparing Embedding reports

CISC850 Cyber Analytics

Page 7: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Embedding using Instruction Q-grams

• For example, if report x=‘1|A 2|A 1|A 2|A’, A={1|A, 2|A }, the q for q-grams is 2.

CISC850 Cyber Analytics

Page 8: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Embedding using Instruction Q-grams

• Normalization

• Redundancy of behavior, considered alphabet, length of reports

CISC850 Cyber Analytics

Page 9: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Comparing Embedding reports

• Euclidean distance

CISC850 Cyber Analytics

Page 10: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Clustering and Classification

• Prototypes->Clustering-> Classification

CISC850 Cyber Analytics

Page 11: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Prototype Extraction

CISC850 Cyber Analytics

Page 12: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Clustering using Prototypes

CISC850 Cyber Analytics

Page 13: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Classification using Prototypes

CISC850 Cyber Analytics

Page 14: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Incremental Analysis

CISC850 Cyber Analytics

Page 15: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Experiments & Application

• Evaluation Data • Three parameters to decide

• Evaluation of Components

• How to select the best parameters dp, dc, dr

CISC850 Cyber Analytics

Page 16: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation Data

• A reference data set

• Evaluate and calibrate the framework

• An application data set

• See the performance on unknown malwares

CISC850 Cyber Analytics

Page 17: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Reference Data Set

CISC850 Cyber Analytics

Page 18: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Application Data Set

CISC850 Cyber Analytics

Page 19: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation of Components

• Precision and recall

CISC850 Cyber Analytics

Page 20: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation of Components

• F-measure

Page 21: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation of Components--dp

CISC850 Cyber Analytics

Page 22: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation of Components--dc

CISC850 Cyber Analytics

Page 23: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Evaluation of Components--dr

CISC850 Cyber Analytics

Page 24: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

Comparative Evaluation with State-of-the-Art

CISC850 Cyber Analytics

Page 25: Automatic Analysis of Malware Behavior using Machine Learningcavazos/cisc850-spring... · Automatic Analysis of Malware Behavior using Machine Learning Konrad Rieck, Philipp Trinius,

An Application Scenario

CISC850 Cyber Analytics