full consistency lag and its applications

Confidential and Proprietary. © 2013 Bazaarvoice, Inc.

EmoDBFull Consistency Lag and its applications

www.bazaarvoice.com


Fahd SiddiquiStaff Software Engineer, Data Infrastructure

Bazaarvoice

linkedin.com/in/fahdsiddiqui

[email protected]

$ whoami


SaaS serving software that collects and displays user generated content, crunches analytics, and extracts insights.

Thousands of clients

Hundreds of millions of pieces of content

Hundreds of millions of unique visitors per month

Tens of billions of pageviews per month

Austin-based company founded in 2005

Austin San Francisco New YorkEngineering offices


Global Monthly Unique Visitors

1B

1B

500M

1B

400M

200M

250M

450M

1B

600M


Overview

• Define Full Consistency Timestamp (FCT) and Full Consistency Lag (FCL)

• Applications of FCT in Distributed Systems

• Distributed Compaction

• Algorithm for determining FCT and FCL

• Final thought on FCT and gc_grace_seconds


Data Infrastructure


EmoDB Overview

Append-only key value store

Exposes cross region databus for listeners to subscribe and watch for changes to data events

RESTful API

Multi-master, multi-datacenter, fault tolerant, horizontal scale on r/w

Backed by multiple multi-region Cassandra rings


Full Consistency Timestamp (FCT)

• A system is said to be fully consistent, when all nodes in a distributed system have the same state of the data. So, if record A is in State S on one node, then we know that it is in the same state in all its replicas

• An eventually consistent system may never be in a fully consistent state.

• Full Consistency Timestamp (FCT) can be defined as a timestamp such that any event, e, whose timestamp, t <= FCT, then e is guaranteed to be consistent on all nodes.



More formally,

∀ e, te | te < FCT => e C ∈

where C: Set of consistent events

e: event

te: Timestamp for event, e



In the inconsistent state:

FCT = T(∆2)


Some applications of FCT

• Invalidate Cache

• Synchronize events

• Monitor real-time staleness

• Distributed compactions (not the same as Cassandra compaction)


Invalidate Cache

• Invalidate caches across data centers without relying on Global CL.

• For instance, write to DC-1 (at any Consistency Level) and then call DC-2 to invalidate cache. DC-2 should not invalidate cache before the change is actually replicated to it.


Invalidate Cache

• Using FCT:

• Invalidate call to DC-2 includes update timestamp, t

• DC-2 waits until t <= FCT, before invalidating cache to make sure changes are replicated.


Synchronize Events

• Synchronize events when we need to be certain that writes are consistent in all data centers.

• For example, Process P1 writes to DC-1, changing the value of "{maintenance: true}". Process P2 should act upon it, iff, the value is propagated to all data centers DC-2, DC-3, DC-n.


Synchronize Events

• Using FCT:

• Process, P2, polls to see if anything is required for it to do

• P2 checks FCT, and makes sure that update is propagated everywhere.

• P2 does work


Monitor real-time staleness

• At any given time, we need to know exactly how stale our data is.

• It is OK to report conservatively, and report higher latency than there is. But we should not under-report real time data staleness.



Chart Full Consistency Lag ( T - FCT )

Note: We update FCT every 5 minutes in the above chart


Distributed Compaction

• In an append-only system, such as EmoDB, a row:

• Row is composed of deltas

• Writers append deltas, and readers resolve deltas to produce a resolved object



• To reduce read latency, we need to compact the deltas that we know are replicated to all data centers.

• Compaction should happen as frequently as possible without data loss.

• An older delta is in flight from a different data center while we are in the process of compaction may result in data loss



• Using FCT:

• Partially compact rows.

• Any deltas before FCT can be safely compacted


FCT - Algorithm

• Hinted Handoffs

• Introduced in the Dynamo paper as a way of handling failure.

• Cassandra uses it for fault-tolerant replication.

• If a write is made and a replica node for the key is down, then the coordinator stores a hint about dead replicas in the local system.hints table.


FCT - Algorithm

• Algorithm:

• Tf : Find oldest hint found in the entire ring.

• rpc_timeout: Maximum timeout in cassandra that nodes will use when communicating with each other

FCT = Tf – rpc_timeout


FCT - Algorithm

• Finding oldest hint in the ring:

• Let's take a quick look at Hints table in system keyspace:

cqlsh:system> desc table hints; CREATE TABLE system.hints ( target_id uuid, hint_id timeuuid, message_version int, mutation blob, PRIMARY KEY (target_id, hint_id, message_version) ) WITH COMPACT STORAGE ...

• hint_id is time UUID and is a clustering key


FCT - Algorithm

• Finding oldest hint in the ring:

… which means that finding the minimum hint_id is quick

SELECT min(hint_id) AS old_hint_id FROM hints;

• T f = minimum(Oldest hint from each node in the ring)

• Note that we can only update FCT if all nodes are up.


FCT – Failure cases

• A node is down:

• A pre-requisite of FCT is that all nodes in the ring have to be up at the time.

• If a node is down, FCT is not updated and remains to the old value.

• Any process that depends on FCT is not disrupted. It just uses the last “good” FCT.

• Not a failure



• Network failure between datacenters:

• A pre-requisite of FCT is that all nodes in the ring have to be up at the time.

• In an event of network outage, FCT remains untouched, and FCT is not calculated. This will simply increase FCL (Full Consistency Lag)

• Not a failure



• Node is down for longer than hints storage window:

• Most impactful risk.

• A node may come up after a long time, and never get hints as they are expired. FCT is updated, which isn’t correct.

• Mitigate this risk by:

• Alerting when FCL gets close to the hints storage window and/or automatically disabling FCT updates.



Chart Full Consistency Lag ( T - FCT )

Note: We update FCT every 5 minutes in the above chart


FCT and gc_grace_seconds

• Cassandra compaction is a local operation.

• gc_grace_seconds is an arbitrary configured value that decides the maximum time that a node’s compaction will keep tombstones around. If this value is too short, you may resurrect deleted deltas. Too long, and you are being inefficient. Cassandra’s default value is 10 days.

• Equipped with FCT, we can get rid of gc_grace_seconds, and simply introduce an “auto” feature.

• In other words, if the FCT is known, then gc_grace_seconds is FCL.


Other works in this area

• "Quantifying Eventual Consistency with PBS" - Peter Balis, et al. The VLDB Journal (2014) 23:279–302

• Probabilistic Bound Staleness (PBS)

• Provides expected bounds on staleness due to eventual consistency.


@Bazaarvoice

@BazaarvoiceDev

http://www.bazaarvoice.com/

http://blog.developer.bazaarvoice.com/

Learnmore

full consistency lag and its applications

Technology