acing the ioc game: toward automatic discovery and ...dvotipka/misc/osuciu_020117.pdfacing the ioc...

Acing the IOC Game: Toward Automatic Discovery and Analysis of Open-Source Cyber Threat Intelligence

Liao, Xiaojing, Kan Yuan, XiaoFeng Wang, Zhou Li, Luyi Xing, and Raheem BeyahPresented by Octavian Suciu

Octavian Suciu:: Acing the IOC Game

What are Indicators of Compromise (IOCs)?• Forensic artifacts of intrusion– virus signatures– IPs/domains used by botnets– MD5s of malware

• Shared across the community– Threat Intelligence platforms

• Used as input to security products– IDS, AVs

The OpenIOC Format

Context = IOC Category [write registry key]

The OpenIOC Format

Content = Specific artifact [file bing modified]

How is security information disseminated?• Blogs• Forums• Social Networks• Blacklists & Databases• Papers & Technical reports [FeatureSmith!]• Underground markers

Problem

Natural Language vs Machine-readable Format

Problem (2)• Volume & Velocity harden manual conversion

Problem (3)• Information Extraction tools are ineffective– domain-specific– high false positive rate

The Trojan downloads a file ok.zip from the server

It’s available as a Free 30 day trial download.

iACE Solution• Automated IOC extractor from technical blogs• Key observation:– Discourse in technical blogs is consistent and stable

• iACE in a nutshell:– discover an IOC token (ok.zip)– identify context (downloads)– analyze their grammatical relation – classify relation based on similarity with others

Outline• System Design• Datasets• Evaluation• Security Findings• Discussion

iACE Architecture

Continuously download websites

iACE Architecture

Filter out non-technical pages (i.e. login pages)

iACE Architecture

Get sentences likely containing IOCs

iACE Architecture

Check if the extracted relations are IOCs

iACE Architecture

Generate the OpenIOC format

Blog Scraper (BS)• Download complete websites

• Monitor them for new posts

Blog Preprocessor (BP)• Remove template from webpages– Retain only user-generated content

• Convert content to text– OCR on images, PDF to text

• Filter pages based on topic– topic words– article length– density of dictionary words

Relevant Content Picker (RCP)• Split text into sentences

• Determine IOC tokens (ok.zip)– regex matching

• Identify context terms (downloads)– dictionary of relevant terms

Relation Checker (RC)

• Graph Mining – Similarity metric for directed graphs– Compute the number of identical random walks

occuring in both graphs

• Train classifier on relations from ground truth• Classify new relations based on their similarity to

ground truth

IOC Generator (IG)• Generate Definition and Header in OpenIOC

format– map context & IOC terms to XML

Datasets• DS-Labeled (used for training)– 450 articles– 1,500 true IOC sentences– 3,000 false IOC sentences

• DS-Unknown (used for testing)– 45 blogs– 71,000 articles

• Training sample size is small

Performance of iACE• Precision = fraction of identified IOCs that are

truly IOCs• Recall = fraction of IOCs that are identified

Precision Recall

Topic Classifier 98% 100%

iACE on DS-Labeled 98% 92%

iACE on DS-Unknown 95% 90%

Performance of Existing Systems• Precision = fraction of identified IOCs that are

truly IOCs• Recall = fraction of IOCs that are identified

Precision Recall

iACE 98% 93%

AlienVault OTX 72% 56%

Stanford NER 71% 47%

Discovered IOCs• 45 blogs

• 71,000 articles (DS-Unknown)

• 20,000 identified as containing IOCs

• 900,000 IOCs identified

How are IOCs related to each other?• Cluster articles on infrastructure-related IOCs– IPs, domains, email addresses– 527 clusters likely corresponding to campaigns

– Little cross-reference between articles in same cluster– This allowed the discovery of new campaigns

How do IOCs evolve over time?• Cluster articles on attack vector IOCs– malware hashes, CVEs– measure decay time = # of consecutive months while

an IOC was mentioned

– most attack vectors are short lived– long lasting attacks pointed to small set of C&C

servers that were not taken down

What is the impact of IOCs on defenses?• How fast are IOCs adopted by the industry?

– Measure the time difference between when IOCs are blogged about and when they are detected on VirusTotal

– 47% of IOCs were detected before being blogged about– AVs respond much slower to domains & IPs than to hashes

What is the quality of the 45 blogs?

• Timeliness = % time being the first to report on IOCs– 10 blogs report first on 60% of campaigns– a blog with 13% timeliness has 84% exclusive IOCs

• Completeness = % of IOCs reported and how diverse they are (different types)– 6 blogs reported 40% of IOCs– 9 blogs reported 50% of IOC types

• Robustness = % of robust IOCs that are reported (these that remain unchanged during campaigns)– C&C servers, registry email are robust during

campaign– one blog reports 87% of the robust IOCs

Discussion• iACE automates IOC generation and has good

performance• Allows an analysis of impact, evolution and

relations between IOCs from technical blogs

• Limitations– Errors due to natural language ambiguity:

• e.g. masking http as hxxp in URLs– Other intelligence sources are also valuable

• iACE assumptions might not hold

Thank you!

Octavian Suciuosuciu@umiacs.umd.edu

acing the ioc game: toward automatic discovery and ...dvotipka/misc/osuciu_020117.pdfacing the ioc...

Documents

toward a code of discovery for juvenile delinquency

toward automated discovery of artistic in uence - arxiv ·...

toward large-scale vulnerability discovery using machine...

acing the ioc game: toward automatic discovery and...

test_2_dossier ioc

ioc - philippines

di == ioc ?

toward future discovery in electroweak physics toward future...

ioc documents

ioc outlets

lanka ioc: buy - acuity · sampath bank plc/ capital trust...

toward knowledge discovery in databases attached to grids

20110715 ioc

open source drug discovery: efforts toward the synthesis ......

ioc presentation

reconsidering popular music in frbr: toward linked data...

ioc etapa3

ioc generico

lanka ioc · lanka ioc 1 lanka ioc at a glance lanka ioc...

toward automated discovery of artistic infuence