The Anatomy of a Leak: AS9121or

How We Learned to Start Worrying andHate the Maximum Prefix Limits

Alin C. Popescu, Brian J. Premore, Todd Underwood

{alin,bj,todd}@renesys.com

Renesys Corporation

The Anatomy of a Leak: AS9121 1 May 15, 2005

Christmas Eve Leak

• 24 Dec 2004: 100K+ routes leaked from AS9121 (TTnet ),

globally propagated

• Bad routes resulted in misdirected/lost traffic for tens of

thousands of networks: serious global vulnerability

• Best common practices were insufficient to prevent direct and

collateral damage

• Will examine the timeline, assess the damage, and what steps

operators may take for infrastructure integrity assurance

The Anatomy of a Leak: AS9121 2 May 15, 2005

A Full Table of ... Turkey

• AS9121(TTnet ) announces an (almost) full table to peers,

including AS6762 (Telecom Italia )

• AS6762 has one misconfigured session with no

maximum prefix set, so they accept 100K+ prefixes

• AS6762 propagates those prefixes to their peers, hitting

maximum prefix limits on all of those sessions

• “Bad” prefixes originated by AS9121 replace those originated

by the real owners

The Anatomy of a Leak: AS9121 3 May 15, 2005

Sample Organizations with Hijacked Routes

Blue Cross Blue Shield of Iowa

Thomson Financial Services

Citicorp Global Information Network

MetLife Capital Corp

Pitney Bowes Credit Corporation

Brown Brothers Harriman & Company

LaSalle Partners

Kuwait Fund for Arab Economic Development

The Anatomy of a Leak: AS9121 4 May 15, 2005

Two Events: Timeline #1

• 09:19:57 UTC 24 Dec 2004: AS9121 starts announcing 106K+

prefixes to peers

• 09:19:57: AS6762 starts carrying 106K+ prefixes originated by

AS9121

• 09:19:58: Renesys hears reports of “bad” paths from 13 peers

The Anatomy of a Leak: AS9121 5 May 15, 2005

Two Events: Timeline #1 (cont’d)

• 09:20:07: 1/3 of Renesys peers heard and believed “bad” paths

• 09:20:27: “Bad” paths spread across the Internet

• 09:36:10: Peak in announcement rate

• 10:03:00: First event ends, but AS9121 continues to announce

bad prefixes throughout the rest of the day

The Anatomy of a Leak: AS9121 6 May 15, 2005

Two Events: Timeline #2

• 19:47:06: AS9121 begins announcing bad prefixes at a high

rate

• 19:47:39: Peak in announcement rate

• 19:50:00: Second event ends, but AS9121 continues to

advertise bad prefixes for a long time

The Anatomy of a Leak: AS9121 7 May 15, 2005

Damage Extremely Widespread – Highlights

• AS6762 carried 106606 bad prefixes

• AS1299 had maximum prefix to AS9121 set relatively low, but

was not saved:

– Heard only 1849 bad prefixes directly from AS9121

– Carried a total of 10925 bad prefixes from other peers:

ASN 6762 1239 6453 701

Num Prefixes 4413 3997 2522 612

The Anatomy of a Leak: AS9121 8 May 15, 2005

Collection Infrastructure

• Renesys operates a peering setup with

– ≈100 peering sessions

– peering at NOTA and LINX, multi-hop from elsewhere

– peers on 6 continents

• “Full tables” from all peers

• Globally integrated view: rapid query of updates from all sources,

not just a single collection point

The Anatomy of a Leak: AS9121 9 May 15, 2005

Propagation ofBad Prefixes

AS646118802

AS329226920

AS693965631

AS1541228283

AS6762106606

AS129911438

AS356157173

AS70116604

AS17415487

AS330329012

AS325724689

AS349159279

AS645321635

AS12399357

AS41346303

AS35499868

AS29149741

AS33567384

AS63477895

AS7025795

AS70186282

AS2095409

AS2282211455

AS79114590

AS55115883

AS9121106793

The Anatomy of a Leak: AS9121 10 May 15, 2005

Who SpreadMost Bad Prefixes

AS646166547

AS329266057

AS693993261

AS6762843572

AS129953675

AS356192735

AS701149601

AS17462429

AS330329049

AS325734927

AS3491183081

AS645396005

AS1239117686

AS354945279

AS291442743

AS335648666

AS70213329

AS701814510

AS20917129

AS791112477

AS551110186

AS9121220539

The Anatomy of a Leak: AS9121 11 May 15, 2005

Distinct “Bad” Prefixes Over Time

0

10000

20000

30000

40000

50000

60000

19:2017:4016:0014:2012:4011:0009:2007:40

# pr

efix

es

time

Event #1

Event #2

The Anatomy of a Leak: AS9121 12 May 15, 2005

Distinct “Bad” Prefixes Over Time (log scale)100000

10000

1000

100

10

1

019:2017:4016:0014:2012:4011:0009:2007:40

# pr

efix

es

time

Event #1

Event #2

The Anatomy of a Leak: AS9121 13 May 15, 2005

Event #1 – Zoom in

0

10000

20000

30000

40000

50000

60000

10:0009:5509:5009:4509:4009:3509:3009:2509:20

# pr

efix

es

time

The Anatomy of a Leak: AS9121 14 May 15, 2005

Event #2 – Zoom in

0

1000

2000

3000

4000

19:5019:4919:4819:47

# pr

efix

es

time

The Anatomy of a Leak: AS9121 15 May 15, 2005

Rates of Advertisement – Event #1

0

10000

20000

30000

40000

50000

60000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 9121AS 6762

0

10000

20000

30000

40000

50000

60000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018

AS 702

0

10000

20000

30000

40000

50000

60000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 1299AS 22822

AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

0

10000

20000

30000

40000

50000

60000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 174AS 1239

AS 701AS 2914AS 3356AS 6461

AS 209AS 5511

The Anatomy of a Leak: AS9121 16 May 15, 2005

Rates of Advertisement – Event #1

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018AS 702

The Anatomy of a Leak: AS9121 17 May 15, 2005

Rates of Advertisement – Event #1

0

1000

2000

3000

4000

5000

6000

7000

8000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 1299AS 22822AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

The Anatomy of a Leak: AS9121 18 May 15, 2005

Rates of Advertisement – Event #1

0

2000

4000

6000

8000

10000

09:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 174AS 1239AS 701

AS 2914AS 3356AS 6461AS 209

AS 5511

The Anatomy of a Leak: AS9121 19 May 15, 2005

Prefixes Carried – Event #1

0

20000

40000

60000

80000

100000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 9121AS 6762

0

20000

40000

60000

80000

100000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018

AS 702

0

20000

40000

60000

80000

100000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 1299AS 22822

AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

0

20000

40000

60000

80000

100000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 174AS 1239

AS 701AS 2914AS 3356AS 6461

AS 209AS 5511

The Anatomy of a Leak: AS9121 20 May 15, 2005

Prefixes Carried – Event #1

0

10000

20000

30000

40000

50000

60000

70000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018AS 702

The Anatomy of a Leak: AS9121 21 May 15, 2005

Prefixes Carried – Event #1

0

2000

4000

6000

8000

10000

12000

14000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 1299AS 22822AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

The Anatomy of a Leak: AS9121 22 May 15, 2005

Prefixes Carried – Event #1

0

2000

4000

6000

8000

10000

12000

14000

10:3510:2510:1510:0509:5509:4509:3509:25

# of

uni

que

pref

ixes

time

AS 174AS 1239AS 701

AS 2914AS 3356AS 6461AS 209

AS 5511

The Anatomy of a Leak: AS9121 23 May 15, 2005

Notes on the Data

• All prefix counts are lower bounds, biased by the sampling

• It is likely that non-peer autonomous systems carried

considerably more bad prefixes than what observed

• To validate the results, data from RouteViews and RIPE were

also used

The Anatomy of a Leak: AS9121 24 May 15, 2005

Operational Lessons

• Holiday staffing: not easy but matters

• Resetting a maxpref ’d session: should not be prevented by

change management

• Current contact and escalation info for all peers: essential

• Tight maximum prefix settings: helps but not enough

• Transitively trusting all peers’ on-net customers: fundamentally

unsafe

The Anatomy of a Leak: AS9121 25 May 15, 2005

Future Work: Beyond maxpref

• It is impossible for large autonomous systems to prefix-filter

their peers

– Hard on some hardware: too many prefixes

– Impossible on the people: no way to generate/maintain lists for big ASes

• It is impossible for large autonomous systems to filter on AS-path

origination

– Hard on most hardware: regexp ’s are slow

– Impossible on the people: no way to generate/maintain lists for big ASes

– Wouldn’t help in cases like this anyway

The Anatomy of a Leak: AS9121 26 May 15, 2005

Future Work: Beyond maxpref

But...

• Current model is “trust all peers transitively”

• Bad things will continue to happen

• maxpref settings didn’t help much and won’t in the future

Therefore...

• Alternative solutions must be considered, including prefix

filtering and AS-path origination filtering peers.

The Anatomy of a Leak: AS9121 27 May 15, 2005

Thank you

peering@renesys.com

{alin,bj,todd }@renesys.com

The Anatomy of a Leak: AS9121 28 May 15, 2005

Additional Slides

The Anatomy of a Leak: AS9121 29 May 15, 2005

Rates of Advertisement – Event #2

0

1000

2000

3000

4000

5000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 9121AS 6762

0

1000

2000

3000

4000

5000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018

AS 702

0

1000

2000

3000

4000

5000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 1299AS 22822

AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

0

1000

2000

3000

4000

5000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 174AS 1239

AS 701AS 2914AS 3356AS 6461

AS 209AS 5511

The Anatomy of a Leak: AS9121 30 May 15, 2005

Rates of Advertisement – Event #2

0

500

1000

1500

2000

2500

3000

3500

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018AS 702

The Anatomy of a Leak: AS9121 31 May 15, 2005

Rates of Advertisement – Event #2

0

500

1000

1500

2000

2500

3000

3500

4000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 1299AS 22822AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

The Anatomy of a Leak: AS9121 32 May 15, 2005

Rates of Advertisement – Event #2

0

1000

2000

3000

4000

19:5019:4919:4819:47

# of

uni

que

pref

ixes

time

AS 174AS 1239AS 701

AS 2914AS 3356AS 6461AS 209

AS 5511

The Anatomy of a Leak: AS9121 33 May 15, 2005

Prefixes Carried – Event #2

0

1000

2000

3000

4000

5000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 9121AS 6762

0

1000

2000

3000

4000

5000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018

AS 702

0

1000

2000

3000

4000

5000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 1299AS 22822

AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

0

1000

2000

3000

4000

5000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 174AS 1239

AS 701AS 2914AS 3356AS 6461

AS 209AS 5511

The Anatomy of a Leak: AS9121 34 May 15, 2005

Prefixes Carried – Event #2

0

500

1000

1500

2000

2500

3000

3500

4000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 3561AS 3303

AS 15412AS 3491AS 6939AS 3292AS 7018AS 702

The Anatomy of a Leak: AS9121 35 May 15, 2005

Prefixes Carried – Event #2

0

500

1000

1500

2000

2500

3000

3500

4000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 1299AS 22822AS 3257AS 3549AS 4134AS 6453AS 6347AS 7911

The Anatomy of a Leak: AS9121 36 May 15, 2005

Prefixes Carried – Event #2

0

1000

2000

3000

4000

19:5419:5219:5119:4919:47

# of

uni

que

pref

ixes

time

AS 174AS 1239AS 701

AS 2914AS 3356AS 6461AS 209

AS 5511

The Anatomy of a Leak: AS9121 37 May 15, 2005