botnet detection and response - caida.org€¦ · botnet detection and response the network is the...

Motivation/OverviewTaxonomyDetection

Response

Botnet Detection and ResponseThe Network is the Infection

David Dagon

[email protected] Institute of Technology

College of Computing

OARC Workshop, 2005

David Dagon Botnet Detection and Response


Response

Outline

Georgia Tech Campus(Cross Sectional View)

based on joint work with:

UMass CS: Cliff Zou

GaTech CS: Sanjeev Dwivedi,Robert Edmonds, Wenke Lee,Richard Lipton, and Merrick Furst

GaTech ECE: Julian Grizzard



Response

Outline

1 Motivation/OverviewDefinitionsThe Network is the Infection

2 TaxonomyPropagationCommand and Control

3 DetectionThe Rallying ProblemDetection Opportunities

4 Response



Response

DefinitionsThe Network is the Infection

Definition: BotsHard to Define; Easy to Detect

Definitions, ExamplesDefinition: autonomous programs automatically performingtasks, absent a real user.Benign bots

countless examples at http://www.botknowledge.com/Gray-area bots

Blogbots, e.g., wikipedia, xanga Note:http://en.wikipedia.org/wiki/Wikipedia:BotsOther examples: xdcc, fserve bots for IRCTrainer bots (MMORPGs)

Malicious botsKey characteristics: process forking, with network and fileaccess, and propagation potential.



Response


Definition: Botnets

Botnets: Also hard to defineDefinition: networks of autonomous programs capable ofacting on instructions.Again, gray areas: FServe bot farms, spider farms, etc.Today, just a narrow definition:

organized network of malicious bot clients

Key Insights

The network is the infection.We must track botnets, not just bots



Response


Botnets as a Root Cause

Botnets are a Root ProblemSpam botsClick fraudLarge-scale identity theft; “vicpic” sitesProxynets (for launching other attacks)

Lightning AttacksThe short vulnerability-to-exploitation window

makes bots particularly dangerous.– Emerging Cybersecurity Issues Threaten Federal Information Systems, GAO-05-231



Response


Botnet vs Bot Detection

What’s the Difference?Why track both bots and botnets?

Bot Detection BenefitsRE → signature IDS (content)Partial victim identification

Response Policy: RBL, QuarantineHost vulnerability analysis



Response


Botnet vs Bot Detection

What’s the Difference?Why track both bots and botnets?

Botnet Detection BenefitsCritical Infrastructure Protection, prioritize on harmto network, not just victims.RE → signature IDS (flows)More Complete victim identification

Remediation Policies: Windows 2003 NetworkAccess Protection (NAP), ISP quarantines



Response

PropagationCommand and Control

Botnet Propagation I

emailRequires user interaction, social engineeringEasiest method; common.Interesting: pidgin English affects propagation.

instant message

Various: social eng., file xfer, vulnerabilities



Response


Botnet Propagation II

remote software vulnerabilityOften, no interaction neededPredator, Prey and Superpredator: worms vs. worms(dabber)

web pagePlain vanilla malware, or even Xanga ghetto botnets

“seed” botnetsBotnets create botnets.Used for upgrades.Most significant for detection



Response


Command and Control Taxonomy

Goals:Anticipate future botnet structuresTaxonomy of botnet controls

An “important and sensible goal for an attacktaxonomy ... should be to help the defender” – R.Maxion

Thus, create a taxonomy based on detection opportunities,instead of random bot/botnet characteristics.



Response



ResourcesPublic, privateBotmaster’s administrative control over a resource

Rallying Services1 Medium used for rallying2 E.g., HTTP, IRCd, DNS tunnel, etc.3 Reminder: public and private versions of the above



Response



Resources (cont’d)Public, privateBotmaster’s administrative control over a resource

Name Services1 hosts(5), e.g., corruptingWINDOWS/system32/drivers/etc/hosts

2 DNS, public and private3 DDNS, public/private4 Hit lists



Response


Command and Control Taxonomy I

RFC ComplianceThe degree of standards compliance.

E.g., non-responsive IRCdAd-hoc protocols.

P2Pport-knockingTunneling (NSTx, sinit, bobax)



Response


Command and Control Taxonomy II

Activity LevelThe degree to which bots are in constant contact withbotmaster.

Time division: periodic phone in, flow-based, sessionless,statelessProximity: delegation of contact; clique connections

InsightNote: other lists possible. Key: organize them into categories.Can we detect these categories?



Response

The Rallying ProblemDetection Opportunities

The Rallying Problem

Let’s focus on “rallying” to identify detection opportunities.

C&C used to rally victimsDetecting C&C ⇒ detecting botnetGoal: detect C&C during formation

Therefore, reason like an attackerAttacker design goals:

RobustnessMobilityStealth

Assumption: The attackers are always motived by thesethree goals.



Response


The Rallying Problem

Suppose we create virusDownload vx code; fiddle; compileUses email propagation/social engr.

We mail it...

V1

V2

V3

V4

V5

VX

Welcome to the 1980s. What if we want to use victimresources?



Response


Simple Rallying I

Naively, we could have victims contact us...Problems

VX must include author’s address (no stealth)Single rallying point (not robust)VX has hard-coded address (not mobile)

V1

V2

V3

V4

V5

VX



Response


Simple Rallying II

Or, the victims could contact a 3d party, e.g., post toUsenet

Some connections dropped, single point of failure (notrobust)Rival VXers and AVers obtain list (not stealthy)

Public, lasting record of victims (not stealthy)

V1

V2

V3

V4

V5

VR

VX



Response


Simple Rallying III

Or, the victims could contact a robust service, e.g., IRCdNo single point of failure (is robust)Rival VXers and AVers id list (not stealthy)

Addressed by adjusting protocol adherence or private natureof service.

Portability of IRCd DNS (is mobile)

1

V2

V3

V4

V5

VVX



Response


Detection In-Protocol

Numerous ad-hoc bot detection frameworks:IRCd, public (DDD, Gnuworld)IRCd, private (RWTH Aachen)E-mail (CipherTrust ZombieMeter; everyone else)AV/Managed network sensing (Sophos)Obvious detection (existing blackhole mining)

Problem:Largely post-attackLargely cannot detect structure (rain drop analogy)Expensive to monitor (requires spam filter banks, ordifficult IRCd manipulations)Trivially evaded



Response


Detection Strategies

What should we do instead of in-protocol sensing?

Better approach: find invariant observable by sensorsBot must always exhibit some behaviorsIf we can sense, we can perform detection

One idea: DNS-based detection



Response


Protocol Agnostic Detection: DNS

Intuitionwww.example.com/productswww.example.com/homebotnet1.example.orgbotnet2.example.org

class 1︷︸︸︷

3LD .SLD.TLD/ subdir1/subdir2︸︷︷︸

class 2

Incentives for Subdirectorieslower skills (dns updates vs mkdir)less risk (fewer $ transactions)lower cost (package 3LD deals)



Response


Detecting DDNS Bots

Canonical DNS Request Rate

CSLDi = RSLDi +

|SLDi |∑

j=1

R3LDj

This is analogous to summing the children for a tree rooted onSLDi .

Key AssumptionDNS server is not authoritative for many zones with high 3LDcount.→ Dyn DNS Providers!



Response


Detecting DDNS Bots

Canonical DNS Request Rate

CSLDi = RSLDi +

|SLDi |∑

j=1

R3LDj


Use Chebyshev’s inequality:

P(|X − µ| ≥ t) ≤ σ2

t(1)




Response


DDNS-Based Detection

For DDNS customers, botnets tend to use subdomains;legitimate directories use subdirectoriesWe can use SLD/3LD-ratios to identify botnet traffic

10

100

1000

10000

01/23 01/24 01/25 01/26 01/27 01/28 01/29 01/30 01/31 02/01 02/02

DN

S lo

okup

s/ho

ur

time

Normal traffic (Weighted and Unweighted)Unweighted bot traffic

Weighted bot traffic



Response


DDNS-Based Detection

For DDNS customers, botnets tend to use subdomains;legitimate directories use subdirectoriesWe can use SLD/3LD-ratios to identify botnet traffic

0

200

400

600

800

1000

1200

1400

1600

01/23 01/24 01/25 01/26 01/27 01/28 01/29 01/30 01/31 02/01 02/02

DN

S L

ooku

ps/h

our

Time (in days)

Bot traffic (Canonical SLD Form)Normal traffic (Canonical SLD Form)



Response


Detecting DDNS Bots

Does Chebyshev’s inequality always work?

0

50

100

150

200

250

300

350

400

01/23 01/24 01/25 01/26 01/27 01/28 01/29 01/30 01/31 02/01 02/02

DN

S L

ooku

ps/h

r

Time

Bot trafficNormal traffic



Response


Detecting DDNS Bots

DNS Density Comparison

d2(x , y ) = (x − y)′

C−1(x − y) (2)

variable vectors (features):x - new observationy - trained normal profile

C – inverse covariance matrix for each member of trainingdata



Response


Detecting DDNS Bots

Simplified Distance MeasureMahalanobis distance considers variance and averagerequest rate‘

Thus, good for outlier detectionWe can assume independence of each feature in normal

DNS requests more likely not correlatedThus, drop covariance matric CAlso done in Wang, Stolfo, etc.

d(x , y) =

n−1∑

i=0

( |xi − yi |σi

)

(3)



Response


Detecting DDNS Bots

0

50

100

150

200

250

300

350

400

0 5 10 15 20 25

DN

S lo

okup

s/hr

Rank (by decreasing number of lookups)

BotNormal Host

Figure: Comparison of Sorted DNS Rates



Response

Response Options

Response options include:DNS RemovalPassive Logging (blackhole)Passive Monitoring (sinkhole)

TCP-layer 4 timeout gamesApplication-layer delays

Interactive MonitoringProxynet/Man-in-middleFingerprinting hosts: clock skew, OSservices, IP, time, etc.Bot Application versioningRemoval interactions (Caution!)

For today: victim epidemiology, andsinkholing



Response

Victim Epidemiology: Total Population

0

50000

100000

150000

200000

250000

300000

350000

12/3000:00

12/3012:00

12/3100:00

12/3112:00

01/0100:00

01/0112:00

01/0200:00

01/0212:00

01/0300:00

01/0312:00

01/0400:00

01/0412:00

01/0500:00

01/0512:00

Num

ber o

f Vic

tims

Time

Total Bot Population Over Time

port 2100 port 2125 port 2187

port 54123 port 6564 port 8092



Response

Victim Epidemiology: Country of Origin



Response

Victim Epidemiology: All

53K botnet

28.6 %

Unknown

70.3 %Windows 0.9 % Misc



Response

Victim Epidemiology: Windows-Only

2000SP4,XPSP1AssortedWin95,98,3.11

2000SP2+,XPSP1

XP,2000SP2XPSP1,2000SP3

XP/2000

XPProSP1,2000SP3

XPSP1,2000SP4

35.8 %2.51 %

25.4 %

7.6 %

1.2 %10.8 %

15.9 %

0.7 %



Response

Population Estimates

How complete?Analysis of closed systems. Lincoln-Petersontwo independent samples, M, and C, for the mark andcapture sets.Second is merely random set in

(NC

).

Define: M – individuals marked by the first sample,C – individuals observed in the second,R – number in both.

With R conditioned on M and C, the distribution of R ishypergeometric:

f (R|M, C) =

(MR

)(N−MC−R

)

(NC

)



Response


If the mark and capture population samples are suitably largepercentages of the total population, i.e., M + C ≥ N, theestimate N is unbiased even for small sample sizes.

N =(M + 1)(C + 1)

R + 1+ 1 (4)

may not always yield sufficiently large mark and capturesamples to estimate N.With a normal distribution for N, we can further calculate a 95%confidence interval for this population as N ± 1.96

√v , where:

v =(M + 1)(C + 1)(M − R)(C − R)

(R + 1)2(R + 2)



Response

Policy Implications for Sinkhole Collection

Policy First; Data SecondLarge data collection efforts always have policy implications.Upfront, we consider:

Privacy issues (granularity of clock skew)Use of Census data

Census of Victim OS/Patch-levelPriority rank research into servicesPolicy implication of discontinued/pay patch systemsConcrete analysis of “Monoculture” concerns



Response


How to improve?Dymanic models needed (non-closed population)Pen tester trend: Interaction with victim services (139, 445)to probe patch level.Borrow Broido’s TTL workAdd p0f dbs for NATing routersAdd behavioral parameter:

estimate of cache-flushing behavior (cf., Wessels &Fomenkov’s “Wow” paper)purpose/use of botnet (e.g., spam, DDoS, click fraud)



Response

Summary

So far:The Network is the InfectionGoal: detect botnets, not just botsExisting botnet detection serendipitous, fragileTaxonomy can direct towards solutionDDNS-based detection feasible

Not discussed:Expand DNS monitoring (future talk: algos and hardware)Expanded RETraceback, LEO involvementThreat metrics (cumulative bw estimation, key crackingpotential, evasion potential)Graph theoretic detection (P2P, TOR-based botnets)



Response

Need Data/Malware?

I have source for hundreds of bots, terabytes of pcapsIf you’re a researcher, and need samples or data:

Let’s exchange PGP keysand check with our advisors, net admins, etc.


botnet detection and response - caida.org€¦ · botnet detection and response the network is the...

Documents