Steve Marx

Developer advocate

Dropbox

“Data synchronization is the process of

establishing consistency among data from

a source to a target data storage and vice

versa and the continuous harmonization

of the data over time.”

– Wikipedia

Cloud storage

DEMOLet’s build cloud storage.

if request.method == 'PUT' then

storage.data = request.body

end

return storage.data

Simple cloud storage

Concurrency

Concurrency anti-pattern

This algorithm doesn’t work:

1. GET latest from the server

2. PUT update

What if things change between 1 and 2?

if (file.exists) {

file.open();

}

Optimistic concurrency

Optimistic concurrency

• Try update “optimistically”

• Conditionally update if nothing changed

• Else change is rejected

• HTTP has Etag and If-Match headers

DEMOImproved cloud storage, now without data loss!

local hash =

crypto.sha256(storage.data or '').hexdigest()

if request.method == 'PUT' then

if request.headers['If-Match'] == hash then

storage.data = request.body or ''

return 200

else

return 409

end

end

Conflict resolution

Transactions

Algorithm for applying an operation:

1. Read current value

2. Apply operation to value

3. Send to server

4. If rejected, GOTO 1

function increment() {

superagent.put('https://sync.webscript.io')

.set('If-Match', etag).send(counter + 1)

.done(function (res) {

if (res.status !== 200) {

counter = parseInt(res.text);

etag = res.headers.etag;

increment();

}

});

}

Transaction downsides

• Commits end up serialized

• Transactions lost on shutdown?

• Firebase and Dropbox Datastores

(Python)

Working offline

Diff and merge

• Keep track of a common ancestor

• 3-way merge, git-like:

– Diff server data against ancestor

– Apply patch to client data

– (or something equivalent)

Merging counts

Ancestor: 3

Server: 4

Client: 5

Merged: 3 + (4-3) + (5-3) = 6

DEMOMerge all the things.

Merging downsides

• Requires tracking common ancestor

• Sometimes ambiguous

Summary

• Syncing is hard, so think about:

– Dealing with concurrency

– Thinking about offline

– Resolving conflicts in a consistent way

• Or use someone else’s platform

(Dropbox!)

Image credits

Thanks!

https://www.flickr.com/photos/kalyan02/5458325252

https://www.flickr.com/photos/gara/3560355335

https://www.flickr.com/photos/somoslarevolucion/3609196127

http://geekandpoke.typepad.com/geekandpoke/2012/09/simply-

explained-concurrency.html

https://www.flickr.com/photos/onesevenone/6361400909

@smarx

smarx@dropbox.com