archipelago - afrinicmeeting.afrinic.net/afrinic-11/slides/aaf/ark_aaf_af11.pdf · group monitors...
Post on 03-Oct-2020
5 Views
Preview:
TRANSCRIPT
kc claffyCAIDA
AAF WorkshopNov 23, 2009
ArchipelagoMeasurement Infrastructure
Outline
Focus and Architecture
Monitor Deployment
Measurements
Future Work
2
Introduction
Archipelago (Ark) is CAIDA’s next-generation active measurement infrastructure
evolution of the skitter infrastructure
in production since Sep 12, 2007
3
Focus
easy development and rapid prototyping
lower barriers => implement better measurements faster with lower cost• measurement infrastructures notoriously lack funding
raise level of abstraction with high-level API and scripting language• inspiration from Scriptroute, Metasploit, Scapy, Racket
4
Focus
dynamic and coordinated measurements
take advantage of multiple distributed measurement nodes in sophisticated ways• one measurement triggers another measurement• use multiple nodes to divide and conquer• synchronize measurements
for example: Doubletree; tomography; Rocketfuel-like targeted discovery of a single network’s topology
5
Focus
measurement services
build upon the work of others; share services between measurement activities• for example, on-demand traceroute/ping service; IP-to-AS
mapping service
similiar in goal to service-oriented architecture (SOA) but at finer granularity and without the complexity
6
Architecture
Ark is composed of measurement nodes (machines) located in various networks worldwide
many thanks to the organizations hosting Ark boxes
please contact us if you want to host an Ark box
Ark employs a tuple space to enable communication and coordination
a tuple space is a distributed shared memory combined with a small number of easy-to-use operations
a tuple space stores tuples, which are arrays of simple values (strings and numbers), and clients retrieve tuples by pattern matching
7
Architectureuse tuple space for decentralized (that is, peer-to-peer) communication, interaction, and coordination
8
monitor1
central server
monitor2
monitor3
monitor4 monitor5
Monitor Deployment
38 monitors in 24 countries
9
13 North America2 South America
12 Europe1 Africa5 Asia3 Oceania
Continent20 academic10 research network4 network infrastructure2 commercial network1 community network1 military research
Organization
Measurements
IPv4 Routed /24 Topology
IPv4 Routed /24 AS Links
IPv6 Topology
DNS Names
DNS Query/Response Traffic
Spoofer Project Collaboration
10
IPv4 Routed /24 Topology
ongoing large-scale topology measurements
ICMP Paris traceroute to every routed /24 (7.4 million)
running scamper• written by Matthew Luckie of WAND, University of Waikato
group monitors into teams and dynamically divide up the measurement work among team members
13-member team probes every /24 in 48 hrs at 100pps
only one monitor probes each /24 per cycle
3 teams active
11
IPv4 Routed /24 Topology
12
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
IPv4 Routed /24 Topology
13
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
software failure
IPv4 Routed /24 Topology
14
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
power supply died
IPv4 Routed /24 Topology
14
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
power supply died
replacementpower supply
IPv4 Routed /24 Topology
14
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
power supply died
replacementpower supply
IPv4 Routed /24 Topology
14
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
power supply died
replacementpower supply
IPv4 Routed /24 Topology
15
0
5
10
15
20
25
30
Sep07
Nov07
Jan08
Mar08
May08
Jul08
Sep08
Nov08
Jan09
Mar09
amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)
Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data
power supply died
replacementpower supply
died
IPv4 Routed /24 AS Links
AS links from Routed /24 Topology traces
map IP addresses to ASes with RouteViews BGP table
16
IPv4 Routed /24 AS Linksstatistics for 1 month of AS links from three sources (Dec 2008):
17
“avg neighbor deg” = avg neighbor degree of the avg k-degree node averaged over all k
“mean clustering” = (avg number of links between neighbors of k-deg nodes) / (max possible such links for k) averaged over all k
nodes linksmax
degreeaveragedegree
averageneighbordegree
meanclustering
Ark
DIMES
RouteViews (rv2)
23,425 56,760 2,509 4.85 467.3 0.354
22,995 74,140 3,590 6.45 705.4 0.446
30,760 65,775 2,328 4.28 487.2 0.241
3 AS Links Sources: 1 Month
18
10-5
10-4
10-3
10-2
10-1
100
100 101 102 103 104
CCDF
Node degree
DIMES AS links (2008-12)Ark AS links (2008-12)
RouteViews (rv2) AS links (2008-12)
3 AS Links Sources: 1 Month
19
100
101
102
103
100 101 102 103 104
aver
age
neig
hbor
deg
ree
Node degree
DIMES AS links (2008-12)Ark AS links (2008-12)
RouteViews (rv2) AS links (2008-12)
3 AS Links Sources: 1 Month
20
10-4
10-3
10-2
10-1
100
100 101 102 103 104
clust
erin
g
Node degree
DIMES AS links (2008-12)Ark AS links (2008-12)
RouteViews (rv2) AS links (2008-12)
AS Links Growth
AS links seem to accumulate linearly without bound
in skitter, Ark, DIMES; possibly in BGP
even with fixed traceroute sources and destination list (which happened with skitter for 4 years)
AS graph densification: average degree increases
for example:
1 year of Ark (2008): 104k AS links, 28k ASes
2 years of DIMES: 356k AS links, 29k ASes
7.5 years of skitter: 209k AS links, 27k ASes
21
AS Links Growth
hard to determine the “natural” time period to aggregate AS links
1 month? 6 months? years?
when do we get a representative AS graph?
22
Ark AS Links Growth
23
23000
24000
25000
26000
27000
28000
29000
1 2 3 4 5 6 7 8 9 10 11 12 55000
60000
65000
70000
75000
80000
85000
90000
95000
100000
105000#
node
s
# lin
ks
Months of accumulation
# nodes# links
Ark AS Links Growth
24
2250
2500
2750
3000
3250
3500
3750
4000
4250
4500
1 2 3 4 5 6 7 8 9 10 11 12 4.5
5
5.5
6
6.5
7
7.5m
ax d
egre
e
aver
age
degr
ee
Months of accumulation
max degreeaverage degree
Ark AS Links Growth
25
450
500
550
600
650
700
750
800
850
1 2 3 4 5 6 7 8 9 10 11 12 0.34
0.36
0.38
0.4
0.42
0.44
0.46
0.48
0.5
0.52
0.54av
erag
e ne
ighb
or d
egre
e
clust
erin
g
Months of accumulation
average neighbor degreeclustering
Ark AS Links: 1, 6, 12 Months
26
10-5
10-4
10-3
10-2
10-1
100
100 101 102 103 104
CCDF
Node degree
Ark AS links (2008, 1 to 12)Ark AS links (2008, 7 to 12)
Ark AS links (2008, 12 to 12)
Ark AS Links: 1, 6, 12 Months
27
101
102
103
100 101 102 103 104
aver
age
neig
hbor
deg
ree
Node degree
Ark AS links (2008, 1 to 12)Ark AS links (2008, 7 to 12)
Ark AS links (2008, 12 to 12)
Ark AS Links: 1, 6, 12 Months
28
10-3
10-2
10-1
100
100 101 102 103 104
clust
erin
g
Node degree
Ark AS links (2008, 1 to 12)Ark AS links (2008, 7 to 12)
Ark AS links (2008, 12 to 12)
Ark IPv6 Topology
ongoing “large-scale” IPv6 measurements since Dec 12, 2008
10 monitors: 3 in US, 7 International
another IPv6 box coming Real Soon Now
ICMP Paris traceroute to every routed prefix
each monitor probes a random destination in every routed prefix in every cycle; 1,553 prefixes <= /48
reduced probing rate to take 2 days per cycle
running scamper
29
Ark IPv6 Topology
statistics for 8 weeks of AS links from six sources:
Dec 12, 2008 to Feb 7, 2009
30
nodes linksmax
degreeaveragedegree
averageneighbordegree
meanclustering
IPv68 weeks
IPv44 weeks
520 1,181 94 4.54 36.3 0.265
23,425 56,760 2,509 4.85 467.3 0.354
Ark IPv6 AS Links
31
10-5
10-4
10-3
10-2
10-1
100
100 101 102 103 104
CCDF
Node degree
Ark IPv4 AS links (2008-12, 4 weeks)Ark IPv6 AS links (2008-12, 8 weeks)
Ark IPv6 AS Links
32
101
102
103
100 101 102 103 104
aver
age
neig
hbor
deg
ree
Node degree
Ark IPv4 AS links (2008-12, 4 weeks)Ark IPv6 AS links (2008-12, 8 weeks)
Ark IPv6 AS Links
33
10-3
10-2
10-1
100
100 101 102 103 104
clust
erin
g
Node degree
Ark IPv4 AS links (2008-12, 4 weeks)Ark IPv6 AS links (2008-12, 8 weeks)
DNS Names
automated ongoing DNS lookup of IP addresses seen in the Routed /24 Topology traces
all intermediate addresses and responding destinations
using our in-house bulk DNS lookup service (HostDB)• can look up millions of addresses per day
257M lookups since March 2008
34
DNS Traffic
tcpdump capture of DNS query/response traffic
only for lookups of Routed /24 Topology addresses
continuous collection of 3-5M packets per day
can download most recent 30 days of pcap files
a broad sampling of the nameservers on the Internet due to the broad coverage of the routed space in traces
how many nameservers have IPv6 glue records? DNSSEC records? support EDNS? typical TTLs?
35
Alias Resolution
Goal: collapse interfaces observed in traceroute paths into routers
toward a router-level map of the Internet
alias resolution work led by Ken Keys
36
Spoofer Project
collaboration with Rob Beverly on MIT Spoofer Project
how many networks allow packets with spoofed IP addresses to leave their network?
Ark monitors act as targets for spoofed probes sent by willing participants
forwards received probe data to MIT server
37
Spoofer Project
38
monitor monitor
monitor
MIT
CAIDA
UDP port 53
tuple space
Ark Statistics Pagesper-monitor analysis of IPv4 topology data
RTT, path length, RTT vs. distance
39
www.caida.org/projects/ark/statistics
Future Work
release Marinda tuple space under GPL
implement large-scale RadarGun measurements
more in-depth analysis of data for stats pages
investigate AS link densification
DNS open resolver surveys?
high-level packet generation, capture, and analysis API
allow semi-trusted 3rd parties to conduct measurements
40
Thanks!
www.caida.org/projects/ark
41
For more information and to request data:
top related