measuring the internet

7/29/2019 measuring the internet

1/17

Measuring the Internet

in real time.

Scott Kirkpatrick, Systems Group (HUJI), and

EVERGROW partners: DIMES (TAU) and ETOMIC

(Budapest, Navarra)
http://www.netdimes.org/


2/17

The problem

Isnt measuring the Internet like, sooo

last century??

No, conceptual and practical problems remain; new ones are becomingapparent.

Peer to peer applications, soon to accommodate streaming

New clients: DSL, cable, hotspots, blackberries, cellphones with cameras,

sensor networksMaking information about uses of digital content accessible and relevant to

scholars from disciplines other than computer science.


3/17

Some history

Measurement activities

Analysis largely based on single snapshots:

Alta Vista, Alexa late 1990s

Single source searches for topology

RouteViews (U Oregon) takes union of major BGP

tables

No one studied bandwidth

Some work became proprietary (Google, et al.),

other activities lost in the dot.com meltdown

(DEC/Compaq SRC web archeology tools)


4/17

Accomplishments to date:

Power Laws

Growth plus preferential attachment is sufficient

Data has been questioned, alternate models

Robustness and reliability

Seen as a percolation process

Two attacks random failure, targeted DOS

Yet 11/13 DNS servers recently were broughtdown, and internet continued to function.

Analysis of strong connectivity of content


5/17

Todays concerns, e.g., content distribution

How to assure that fresh copies of a large piece of information reach

users around the world? (news, movies, catalogs, software

releases). Answer will involve spreading copies around.

Classic (Valliant) multicast strategy always start with a leap into

the complete unknown, then proceed greedily. Peer to Peer approaches redistribute the servers work by having

recipients provide what they have to subsequent requesters. But

they greatly INCREASE the load on networks.

JULIA attempts to send the least information over the longest links,

most information exchanged between nearby clients. How to know enough about the instantaneous properties of the internet

to make this possible?


6/17

Todays measurement activities are different

Todays issues are not connectivity to the backbone, but among the leaves.

Not characteristics of the distribution tree, but of the mesh that links neighbors

in a single geography.


7/17

ETOMIC approach for bandwidth

Use GPS timers, not network time signal

Carefully timed packet train creates momentary overload

Chirping them allows identification of the capacity

15 systems came up in 2004, deployed across Europe

HUJIs is in ROSS 1st floor, disguised as a PlanetLab station

Manage experiments from server in Navarra, Spain Present status can see time of flight from 70 meters to

3000 km.


8/17

Distributed Internet MEasurement and Simulation

Let the network measure itself

Convince people to download lightweight client, provide

measurement scripts from central server (@Home)

Fly under the radar of network administrators Today we initiate one traceroute request every 30 sec.

Client is Java/Windows. Linux and Mac clients under

construction.

DIMES website provides competitive stats on success ofeach agent

Operational since 1 Sept 2004.


9/17

DIMES growth curve

500

agents

Still finding ~1000 links/week. Not steady state. Agents increasing,

and new scripts are constantly being introduced.


10/17

Power laws are real, seen in AS graph


11/17

Comparison with BGP data

BGP ~18,000 ASs, DIMES sees only 14,500

DIMES has >12,000 edges not in BGP

BGP has ~7000 edges not yet in DIMES map

Joining the two datasets, we see of 6 or more,

definitely not a tree.


12/17

Now assign sites to their k-shells

Find all sites with k ngbrs or less and remove them.

Continue until only sites with >k ngbrs remain.

Whats left is the (k+1)-core.

Sites in the k-core, not in the (k+1)-core are the k-shell.

The k-core is w.h.p k-connected (at least on random

graphs). K-core is found in linear time. K-connectivity

has linear proof only for k


13/17

K-shells also follow power law


14/17

K-shell membership suggests roles for

sites

Highest k: DNS, Akamai; medium k is ripe for peer to peer participation.

At least 21 distinct routes connect all the DNS servers in the world.


15/17

Current status and next steps

Currently

500 agents, 45 countries, all continents

> 100,000 measurements/day

Most detailed AS-level map ever, still growing

Each week captures ~2/3 of the whole map, so dynamics is accessible

Refining the analysis to resolve POPs

To achieve 2000 agents (YE2005), 10000 (YE2006), 50000 (YE2007) wewill add function, such as:

Choose best download site; measure your ISP

Internet Weather reports

Smart P2P clients eMule first

Virus-immune system


16/17

Agents by Country

Argentina

Australia

Austria

Belgium

BrazilCanada

China

Czech Republic

Denmark

Egypt

Estonia

Finland

France

Germany

Greece

Hong Kong

Hungary

India

Israel

ItalyJapan

Korea (Republic of)

Latvia

Lithuania

Luxembourg

USA

Israel

UK

Sweden

Ger.

Den.

Hun.

Fr.


17/17

http://www.netdimes.org
http://www.netdimes.org/http://www.netdimes.org/

measuring the internet

Documents