distributed shared memory, related issues, and new challenges in large-scale dynamic systems vincent...

Distributed Shared Memory, Related Issues, and New Challenges in Large-Scale Dynamic Systems

Vincent Gramoli

ICDCS 2007June

Fernandez, Gramoli, Jimenez, Kermarrec, Raynal

Roadmap

Large-Scale Dynamic Systems Context and Motivations

Distributed Shared Memory Facing Dynamism

Facing Scalability

A Probabilistic Solution Facing Dynamism and Scalability

A New Challenge Distributed Slicing

ICDCS 2007June

The Scale-Shift of Distributed Systems

Internet Network growth

IPv4 to IPv6

Internet

ICDCS 2007June

Personal devices multiplyAll tend to be connected together

ICDCS 2007June

Network devices

17 billions of network devices by 2012as predicted by IDC

ICDCS 2007June

Drawback of this Scale-shift

Heterogeneity Each device acts independently

Each device has distinct lifetime

Out-of-control Global monitoring is impossible

The system is unpredictable

Dynamism At any time some participants may leave/fail

And some others may join

Unbounded number of leaves/failures/joins

ICDCS 2007June

Problem: How to Communicate?

Shared-Memory Paradigm Simple programming style

Appealing design for algorithmsMEM

ICDCS 2007June

Appealing design for algorithms

Message-Passing Paradigm Better suited for delayed messages

Fault-tolerant system

ICDCS 2007June

Appealing design for algorithms

Message-Passing Paradigm Better-suited for delayed messages

Fault-tolerant system

Emulating Shared-Memory in Message-Passing Simplicity

Fault-tolerance

ICDCS 2007June

Distributed Shared Memory: the emulation

Consistency Criterion, a set of rulesAtomic object:

• If an operation ends before another starts, then it can not be ordered after

• Write operations are totally ordered and read operations are ordered w.r.t. write operations

• A read returns the last value written (or the default one if none exist)

If objects are atomic, then the system looks like a shared-memory model!

ICDCS 2007June

Quorum-based DSM [ABD]

Quorums Mutually intersecting sets of nodes

Q1 ∩ Q2 ≠ ØQ1 ∩ Q3 ≠ ØQ2 ∩ Q3 ≠ Ø

Each node maintains: A local value v of the object A unique version number t of this value

ICDCS 2007June

Operations A node reads the object value by

• Asking the value, tag of all nodes of a quorum

• Choosing the value with the largest tag

• Replicating this value to all nodes of a quorum

A node writes a new object value by• Asking the tags of all nodes of a quorum

• Choosing a higher tag than any tag returned

• Replicating its value with the new tag to a quorum

Get <v,t>

Set <v,t>

Get <v,t>

Set <v’,t’>

t’ = t++

ICDCS 2007June

Writing a value v1

Input: v1

ICDCS 2007June

Writing a value v1

max tag?

ICDCS 2007June

Writing a value v1

Q3v1,t1 (with t1 > t)

ICDCS 2007June

Reading a value

Q3value? tag?

ICDCS 2007June

Reading a value

ICDCS 2007June

Reading a value

Output: v1

ICDCS 2007June

Dynamic DSM [RDS]

Dynamism => Unbounded number of failures

Solution: Reconfiguration Replacing quorums periodically with quorums

of active nodes.

Problem: Q1 ∩ Q2 = Ø

and Q1 ∩ Q3 = Ø and Q2 ∩ Q3 = Ø

ICDCS 2007June

Dynamic DSM [RDS]

All must agree on the next set of quorums Quorum-based consensus algorithm: Paxos

Reconfiguration must not block operations Up-to-date information is passed from old

quorums to new quorums during reconfiguration Operations that discover a new quorum set

must restart using it.

ICDCS 2007June

Dynamic DSM [RDS]

Algorithm Reconfiguration is based on Paxos (3 phases leader-based

consensus alorithm) l is the leader c is the current configuration configs is the set of active configurations A ballot has a unique identifier b and a value v, which is a

configuration

Paxos phases: Prepare: l creates a new ballot and chooses/gets the value to

propose. Propose: l proposes <b,v> and gathers votes from a majority. Propagate: l propagates decision

ICDCS 2007June

Dynamic DSM [RDS]

Recon(c,c’)

ICDCS 2007June

Dynamic DSM [RDS]

Prepare phaseRecon(c,c’) •Creates a new larger ballot b

ICDCS 2007June

Dynamic DSM [RDS]

<1a, b>

Prepare phaseRecon(c,c’)

ICDCS 2007June

Dynamic DSM [RDS]

<1a, b>

<1b, b, configs, <b’’, c’’>>

•Updates its ballot’s value v with the one received •Updates its configs set

Prepare phaseRecon(c,c’)

ICDCS 2007June

Dynamic DSM [RDS]

<1a, b>

<1b, b, configs, <b’’, c’’>>

<2a, b, c, v>

Propose phaseRecon(c,c’)

ICDCS 2007June

Dynamic DSM [RDS]

<1a, b>

<1b, b, configs, <b’’, c’’>>

<2a, b, c, v>

<2b, b, c, v, tag, val>

Recon(c,c’)

Propose phase

•Updates their tag and val•Adds v to their configs set 27

ICDCS 2007June

Dynamic DSM [RDS]

<1a, b>

<1b, b, configs, <b’’, c’’>>

<2a, b, c, v>

<2b, b, c, v, tag, val><3a, c, v, tag, val>

<3a, c, v, tag, val>

Recon(c,c’)

Propagation phase

•Update their tag and val•Remove configuration c from their configs set

<3a, c, v, tag, val>

ICDCS 2007June

Dynamic DSM [RDS]

Good News: The overhead latency to cope with dynamism is low

ICDCS 2007June

Dynamic DSM [RDS]

Bad News: In both solutions, congestion may delay the latency

ICDCS 2007June

First Conclusion

Communication complexity must be reduced to face scalability!

ICDCS 2007June

Scalable DSM [SQUARE]

Object replicated on failure-prone nodes The replicas r1, …, rk share a 2-dim

coordinate spacer1 r2 r3 r4

r5 r6 r7 r8

… rk-1rk

ICDCS 2007June

Communication through neighborhood Each replica ri can communicate only with its nearest

neighbors

ICDCS 2007June

Topology takeover mechanism [CAN] Upon node failure/departure the space sharing is

modified accordingly

If a node ri fails, a takeover node rj replaces it

ICDCS 2007June

Dynamic Quorums Vertical Quorum: All replicas responsible of an

abscissa x Horizontal Quorum: All replicas responsible of

an ordinate yx

For any horizontal quorum H and any vertical quorum V:

H V ≠ Ø

ICDCS 2007June

Read Operation:1) Get up-to-date values and tags of a horizontal

quorum,2) Replicate this value on a vertical quorum.

Write Operation:1) Get up-to-date value on a horizontal quorum,2) Replicate the value to write (and a higher tag) on

the same vertical quorum

Operation Execution

ICDCS 2007June

Memory thwarts if the requested replica is overloaded:Other replicas on its diagonal are contacted in turn until a non-overloaded one is found

Memory expands if all contacted replicas are overloaded:A node outside the memory is added, and the object value is replicated at this node.

Memory shrinks if a replica gets underloaded:The replica simply leaves the memory after neighbors notification.

Self-Adjusting Memory

ICDCS 2007June

Good News: The memory self-adapts well in face of dynamism

ICDCS 2007June

Good News: The load is well-balanced over the replicas

ICDCS 2007June

Bad News: The operation latency increases with the load (request rate)

ICDCS 2007June

Second Conclusion

No way to avoid the tradeoff betweencommunication and time complexity!

ICDCS 2007June

Scalable and Dynamic DSM [TQS]

Motivations for Probabilistic Solutions Tradeoff between time and message

complexity prevents deterministic solutions

Allowing more Realistic Models• Any node can fail independently

• Even if it is unlikely that many nodes fail at the same time

Quality of Service (QoS) is often expressed in terms of percentage of success

ICDCS 2007June

Dynamic System n interconnected nodes Nodes join/leave the system A joining node is new c is the churn:

• At each time unit, └cn

┘ nodes leave the network

• At each time unit, └cn

┘ nodes enter the network

ICDCS 2007June

Probabilistic Atomicity• If an operation ends before another starts, then it is

ordered after with probability e-β2 (with β a constant). If this happen, the preceding operation is considered as unsuccessful.

• Write operations are totally ordered and read operations are ordered w.r.t. write operations

• A read returns the last successfully value written (or the default one if none exist) with probability 1- e-β2 (with β a constant). If this does not hold, then the read is unsuccessful.

ICDCS 2007June

Gossip-based algorithm in parallel Shuffle set of neighbors using gossip-based algorithm

(e.g. Cyclon)

Quorum contact- Disseminate message with TTL l to k neighbors, such

that #contacted nodes = β n / (1-c) Δ/2

- Decrements TTL received if first time received.- Forward received messages to k neighbors if their

TTL is not null.

ICDCS 2007June

Read Ask a quorum of nodes their values Replicate the most up-to-date value to a

quorum

Write Ask a quorum of nodes their tags Chooses a strictly higher tag Replicate the value to write with the new tag

ICDCS 2007June

Assumptions: success regularity: at least one operation succeeds

every Δ time units. The underlying gossip-based algorithm provides each

node with a neighbor chosen uniformly at random.

Results: Expected messages: O( nD) without shuffling, Expected latency: O(log nD), Where D = (1-c)-Δ is the dynamic parameter.

Given the application requirement in terms of QoS, the quorum size can be tuned.

ICDCS 2007June

Third Conclusion

Trading deterministic for probabilistic guarantees seems to be the solution!

ICDCS 2007June

What could be the next step?

Today in Peer-to-Peer: Every node must participates Avoid Free-Riding/Lurking at any cost!

However, an old story says: Gnutella performance was limited by the

performance of its lowest capable nodes

ICDCS 2007June

What could be the next step?

Some classifying solutions are efficient: Recommendation helps decision (e.g. eBay) Supernodes/utlrapeers help sharing files (e.g. Kazaa) Supernodes help NAT/FW by-passing (e.g. Skype)

Generally, we can benefit from heterogeneity Streaming service needs nodes with highest

bandwidth Non-critical service can run on unstable nodes File-sharing service requires nodes with many files …

ICDCS 2007June

Problem

Classifying nodes into categories, slices Based on individual characteristics: attributes A slice corresponds to a portion of the system

Typically, answering the question:

ICDCS 2007June

Distributed Slicing [RANK]

HOW AM I COMPARED TO OTHERS?

Classifying nodes into categories, slices Based on individual characteristics: attributes A slice corresponds to a portion of the system

Typically, answering the question:

ICDCS 2007June

…using their attribute values (assume a single attribute for simplicity reason)

ICDCS 2007June

Attribute values ai

ICDCS 2007June

Attribute values ai

NormalizedIndices pi

ICDCS 2007June

#4#3#2#1

NormalizedIndices pi

0 1Slices si

Attribute values ai

ICDCS 2007June

Existing solutions use gossip-based mechanism

ICDCS 2007June

Performance achieved so far: d, is the distance from pi’ (the position

estimate of i) to the closest slice boundary. For confidence coefficient of 99,99%, the

required number of attribute value drawn is:

mi ≥ z pi’ (1 – pi’) / d2, with z <16, a constant.

ICDCS 2007June

Fourth Conclusion

Distributed Slicing is a new Challenge!

ICDCS 2007June

References[RANK] Distributed Slicing in Dynamic Systems A. Fernandez, V. Gramoli, E. Jimenez, A-M. Kermarrec, and M. RaynalICDCS 2007[TQS] Timed Quorum System for Large-Scale Dynamic EnvironmentsV. Gramoli and M. RaynalIRISA TR1859 2007[SQUARE] SQUARE: Scalable Quorum-based Atomic Memory with Local ReconfigurationV. Gramoli, E. Anceaume, and A. VirgilittoACM SAC 2007[Cyclon] Cyclon: Inexpensive Membership Management for Unstructured P2P OverlaysS. Voulgaris, D. Gavidia, and M. van SteenJournal of Network and System Management 13(2) 2005[RDS] Reconfigurable Distributed Storage for Dynamic NetworksG. Chokler, S. Gilbert, V. Grmoli, P.M. Musial, and A.A. ShvartsmanOPODIS 2005[CAN] A Scalable Content Addressable Network.S. Ratnasamy, P. Francis, M. Handley, R.M. Karp, and S. ShenckerACM SIGCOMM 2001[ABD] Sharing Memory Robustly in Message-Passing Systems.H. Attiya, A. Bar-Noy, and D. DolevJACM 1995

distributed shared memory, related issues, and new challenges in large-scale dynamic systems vincent...

distributed shared memory

raynal problem

raynal drawback

sharedmemory model

new object value

local value

algorithms mem p1

scaleshift heterogeneity

Documents

vincent reynouard...

distributed slicing in dynamic systems a. fernández, v....

william j. vincent and judy s. vincent v. jack c....

square scalable quorum-based atomic memory with local...

constitution of saint vincent and the grenadines, 1979...

travellers to travellers to st. vincent st. vincent

denis vincent and denis vincent helico

vincent van gogh. vincent van gogh the potato eaters 1885

epfl - march 7th, 2008 interfacing software transactional...

tm input acceptance vincent gramoli, derin harmanci, pascal...

vincent john vincent

sonny vincent press clipssonny vincent press … · sonny...

vincent van gogh. vincent van gogh birth name: vincent...

st vincent leadership prezi - svdp usa vincent leadership...

fair report - vincent van duysen | vincent van duysen

a non-blocking, contention-friendly skip list school of...

arthur et le dragon auteur : eve vincent illustrateur : eve...

chapter 3 vincent ologak. chapter 3 vincent ologak

sargent - vincent cichowicz vincent dimartino and armando

large-scale networks - redbelly blockchain · large-scale...