1

Addressing and Routing in Multi-substrate Overlay

Networks

Jorg Liebeherr

University of Toronto

Joint work with: Majid Valipour and Boris Drazic

2

Substrate (“underlay”) network

Overlay network

What is an overlay anyway?

• An overlay network is a virtual network of nodes and logical links built on top of an existing network

• A virtual link in the overlay corresponds to a path in the underlay

Multi-substrate Network

• Colors indicate type of substrate• Overlapping circles and links indicate bidirectional connectivity

Resulting Topology

4

Approaches to Routing

1. “Vanilla” Ad-hoc routing– Extensively explored

– Not cognizant of substrates

– Scalability limits

2. Single structured overlay– Simple addressing and routing

– Scales to large networks

– Not always viable

3. Hierarchical– Within and between groups of

same color

– Needs address and routing solutions

5

6

Overlays over multiple substrates

• Goal: Improve ability to build an overlay over multiple substrates

• Problem: How to efficiently propagate information about address bindings?

• Solution: Protocol mechanisms for exchanging address bindings Cross Substrate Advertisement (CSA)

Address Bindings in Single-substrate Overlay

• Address binding: [A; SA(A)]

7

C

A DB

Substrate Network

SA(A)SA(C)

SA(B)SA(D)

Overlay nodeidentifier

Substrate address

C

A DB

SA(A) SA(B)

Substrate S3Substrate S2Substrate S1

SAS1(A)

SAS1(C)

SAS2(C)

SAS2(B) SAS3(A)

SAS3(B)

SAS3(D)

Address Bindings in Multi-substrate Overlay

• More complex address binding: [A; SAS1(A), SAS3(A)]

8

Overlay nodeidentifier

Substrate address

9

Cross Substrate Advertising (simplified)- Outgoing -

Overlay SocketNode

Node Adapter

Cross-Substrate Advertisement

Adapter

Interface 1

Interface 2

Interface 3

ID = A

Source: A

SAS1(A) SAS2(A) SAS3(A)

SAS1(A)

SAS2(A)

SAS3(A)

List of substrate addresses

Source: A SAS1(A) SAS2(A) SAS3(A)

10

Cross Substrate Advertising (simplified)- Incoming -

Overlay SocketNode

Node Adapter

Cross-Substrate Advertisement

Adapter

Interface 2

Interface 3

ID Substrate addresses

A SAS1(A), SAS1(A), SAS1(A)

Source: A

Source: A SAS1(A) SAS1(A) SAS1(A)

12

Evaluate Methods for Relayed Address Exchange

Exchange address lists periodically Attach address lists to

protocol messages

Add address list to each message with info on node

Request address list when info is needed

Add preferred address

to each message with info on node

“Push” “Pull”“Push Single”

Gossip Protocol-driven dissemination

Experimental Evaluation

• Local Emulab Testbed• 20 Linux nodes

• Software:• Hypercast with CSA

• Delaunay Triangulation protocol

• Multiple UDP/IP substrates

CPU 2xQuad-Core Xeon 5400 (2 Ghz)

RAM 4G DDR2

Interface 8x1Gbps (4 Intel card, 4 NetFPGA)

Mapping of Nodes to Substrates

• K x K substrates (K+1) x (K+1) regions• Nodes distributed uniformly across regions

K=2

R11

R22

R12 R13

R23

R31

R21

R33R23

Performance metrics

• Stability: Do nodes reach a stable state?– % of nodes satisfying stability criterion for local neighborhood

• Connectivity: Do nodes form a single overlay?– # of partitioned topologies

15

A single stable overlay topology has formed when (1) 100% of nodes are stable; and(2) there is one topology

Stability

K=8 (64 substrates), 648 nodes

Push/Pull

None

Push-single

Gossip

Push-single and gossip

250 ms

500 ms

1000 ms

Connectivity

K=8 (64 substrates), 648 nodes

Num

ber

of

part

ition

s

Num

ber

of

part

ition

s

Protocol overhead

Received Traffic (average per node): K=17 (289 substrates), 2592 nodes

Approaches to Routing

1. “Vanilla” Ad-hoc routing– Extensively explored

– Not cognizant of substrates

– Scalability issues

2. Single structured overlay– Simple addressing and routing

– Scales to large networks

– Not always viable

3. Hierarchical– Within and between groups of

same color

– Needs address and routing solutions

19

Reachability Domain

20

• Substrate: Set of compatible attachment points• Reachability Domain: Maximal set of nodes with path in same substrate

Nodes in the same reachability domain must be able to agree on the scope and identifier of a reachability domain.

Hierarchical routing: First find destination RD, then node within RD

Dynamic Reachability Domains

21

• Number and location of reachability domains are dynamic

New

reachability domain

Location of

reachability domain

changes

Keeping consistent routing tables may not be feasible

Strawman: Landmark Domain Routing

22

• Inspired by Landmark Routing and Compact Routing

• Key concepts: – Use least volatile reachability domains as landmark domains

– Route to destination via landmark domain• Nodes know how to route to landmark domain• Locator is reverse path from landmark domain

– Locator has one entry per traversed reachability domain

Reachability Domain

23

• Route from A K

• K’s Locator: {RD6; H, J}

Step 1: Use next-hop information to reach node in landmark domain

Step 2: Use locator for exit node in landmark domain

Step 3: Follow reverse path to K

Preliminary Evaluation

24

• Random static topology• 100 nodes per substrate with up to 70 reachability domains• Multiple landmark domains

Related Works

• Adhoc Routing– Hierarchies or clustering for scalability

e.g., [Ramanathan & Steenstrup `98] [Pei et.al. `00] [Eriksson et.al.`04]

– Structured overlays for ad-hoc routing e.g., [Viana et.al. ‘04]

• Scalable Routing– Landmark e.g., [Francis `88]

– Compact Routing e.g., [Thorup & Zwick `01]

– Metric Spaces e.g., [Krioukov `10]

• Overlay-based Routing– Identifier-based e.g, [VRR ’06] [IHR ’08] [ROFL ’06]

– Locator-based + directory e.g., [SEATTLE `08] [SpoVNeT ’08]

25

Summary

• Self-organizing overlay protocols over multiple substrates

• Cross-Substrate Advertisement improves ability to form a structured overlay over multiple substrates

• Hierarchical routing between reachability domains

• Landmark domain routing awaits full exploration:– Provable bounds on path stretch– Scalability vs. dynamics