1

Canon in G Major: Designing DHTs with Hierarchical Structure

Prasanna Ganesan (Stanford University)

Krishna Gummadi (U. of Washington)

Hector Garcia-Molina (Stanford University)

2

Distributed Hash Table (DHT)

• Hash table over dynamic set of nodes– Insert(key), Lookup(key)– Join(node), Leave(node)

• Partition hash space by node ID• “Flat” overlay network

– O(log n) operations – Homogeneous components– No central point of failure

0 8

3 5

1

6

7

1311

10

3

Why Hierarchies?

• Hierarchies exist!

• Scalability, fault isolation, autonomy, etc.

• Goal: Inherit best of both worlds with a hierarchical DHT

4

What is a Hierarchical DHT?

Stanford Washington

USA

CS EE

Path Locality

Efficiency, Security,

Fault isolation

Path Convergence

Caching, Bandwidth

Optimization

Local DHTs

Access Control

5

Problem Statement

• Convert flat DHTs to hierarchical (Canon-ization)– Chord Crescendo– CAN Can-Can– Symphony Cacophony– Kademlia Kandy

• Caveat: Preserve homogeneity and state vs. routing trade-offs of flat DHTs

6

Roadmap

• Chord and Crescendo

• Routing in Crescendo

• Storage and Caching

• Experimental Evaluation

7

Chord

• Circular N-bit ID space

• Node x links to succ( x + 2^i )

0111

00110101

10000000

0001

8

Crescendo

• Key idea: Recursive structure– Construct bottom-up; merge smaller DHTs– Lowest level: Chord

CS EE

Stanford

9

Merging two Chord rings

0

12

10

52

13

8

3

2

13

8

3Black node x: Connect to y iff

• y closer than any other black node

• y = succ(x + 2^i)

10

Crescendo Details

• Generalize two-ring merge– Merging multiple rings– Multi-level hierarchies

• Making it incremental – New node joins bottom-up

• How many links per node?– Roughly log n ; independent of hierarchy

m nodes

log m links

n blue nodes

n/m nodes

log(n/m+1) links

Total=log m + log (n/m +1)

= log(m+n)

11

Routing in Crescendo

• Greedy clockwise routing!

0

12

10

52

13

3

8

• Path locality by greedy routing!

• Path convergence at closest node to destination of same color

• Local DHTs by construction!

12

Extensions

• Can apply to other DHTs

• Mix-and-match DHTs– Stanford runs Chord, Washington runs

Kademlia

• Additional support for proximity

13

Storage and Caching

• Hierarchical Storage– Specify storage domain subtree

• Access control provided by routing itself

• Caching: Cache at convergent nodes at each level – Nodes form a distributed cache

14

Experiments

• c-level hierarchy– Uniform/Zipf distribution at each level

• Basic metrics– #links / node – #hops / query

15

Number of Links vs. Number of Levels

9

10

11

12

13

14

15

16

17

1000 10000 100000

Number of Nodes

Nu

mb

er

of

Lin

ks

Chord

Levels=2

Levels=3

Levels=4

Levels=5

16

Levels vs. Routing Hops

4

4.5

5

5.5

6

6.5

7

7.5

8

8.5

9

1000 10000 100000

Number of Nodes

Nu

mb

er o

f H

op

s

Chord

Levels=2

Levels=3

Levels=4

17

Path Locality

• GT-ITM topology as hierarchy

• Compare Crescendo with Chord– With proximity adaptation

• Path locality– Latency to different level destinations

18

Path Locality

0

100

200

300

400

500

600

700

800

900

1000

Top First Second Third Fourth

Query Locality

Lat

ency

(m

s)

Chord (Prox.)

Crescendo

Crescendo (Prox.)

19

Conclusions

• Generic mechanism for hierarchical DHTs– Locality: Fault isolation, Security, Efficiency– Convergence: Caching, Bandwidth savings– Local DHTs: Hierarchical Access Control– Preserves degree vs. routing trade-offs

• Potentially useful in large-scale deployment

20

Number of Links vs. Number of Levels

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25

Number of Links

Fra

cti

on

of

No

de

s

Levels=1

Levels=2

Levels=3

Levels=5

21

Path Convergence

• Overlap fraction Length(Overlap)/ Length(Total)

Length: Latency or Number of Hops

22

Overlap Fraction

0

0.2

0.4

0.6

0.8

1

1.2

Top Level 1 Level 2 Level 3 Level 4

Domain Level

Ove

rlap

Fra

ctio

n

Chord (Latency)

Crescendo (Latency)

Chord (Hops)

Crescendo (Hops)

23

Caching

• Cache on convergent nodes at each level– Nodes in the level form a distributed cache

• Advantages– Exploit locality of reference in hierarchy– Queries guaranteed to hit cache– Interesting cache replacement possibilities