mongodb world 2016: smart strategies for resilient applications
TRANSCRIPT
Smart Strategiesfor
Resilient MongoDB Applications
A. Jesse Jiryu DavisStaff Engineer
@jessejiryudavis
What Do Drivers Do?
Network blip: set state "unknown", throw error Primary failover: same Network down: same Command error: just throw the error
what state?
Server DiscoveryAnd Monitoring
Server 1: type "Primary" Server 2: type "Secondary" Server 3: type "Secondary"
Server 1: type "Unknown" Server 2: type "Secondary" Server 3: type "Secondary"
Network blip
Server DiscoveryAnd Monitoring
Server 1: type "Primary" Server 2: type "Secondary" Server 3: type "Secondary"
Server 1: type "Unknown" Server 2: type "Secondary" Server 3: type "Secondary"
Drivers check twice per second for up to 30 seconds
Server DiscoveryAnd Monitoring
Server 1: type "Primary" Server 2: type "Secondary" Server 3: type "Secondary"
Server 1: type "Unknown" Server 2: type "Secondary" Server 3: type "Secondary"
Primary failover
Server DiscoveryAnd Monitoring
Server 1: type "Unknown" Server 2: type "Secondary" Server 3: type "Secondary"
Drivers check twice per second for up to 30 seconds
Server 1: type "Secondary" Server 2: type "Primary" Server 3: type "Secondary"
Server 1: type "Secondary" Server 2: type "Secondary" Server 3: type "Secondary"
Network blipPrimaryfailover
Network down
Commanderror
May undercount Undercounts Correct Correct
Bad Retry Strategy:
Don't retry.
Network blipPrimaryfailover
Network down
Commanderror
First retry succeeds, may overcount
First retry succeeds
Wastestime
Wastestime
Bad Retry Strategy:
Retry five times.
Network blipPrimaryfailover
Network down
Commanderror
First retry succeeds, may overcount
First retry succeeds
Correct Correct
Bad Retry Strategy:
Retry once,except command error.
final hurdle
Network blipPrimaryfailover
Network down
Commanderror
First retry succeeds, won't overcount
First retry succeeds
Correct Correct
Retry once,except command error.
Make all ops idempotent.
Idempotent Insert
doc = {_id: ObjectId(), ...}try: insertOne(doc)except network err: try: insertOne(doc) except DuplicateKeyError: pass # first try worked throw
Idempotent Delete
try: deleteOne({key: uniqueValue})except network err: deleteOne({key: uniqueValue})
Idempotent Update
try: updateOne({ _id: '2016-06-28'}, {$set:{sunny: true}}, upsert=True)
except network err: try again, if that fails throw
not idempotent.
NON-Idempotent Update
the hard one.
updateOne({ _id: '2016-06-28'}, {$inc: {counter: 1}}, upsert=True)
Idempotent Update1. Add unique token:
{ _id: '2016-06-28', counter: N, pending: [ ObjectId("...") ]}
2. Remove token and increment counter:{ _id: '2016-06-28', counter: N, pending: [ ObjectId("...") ]}
2. Remove token and increment counter:{ _id: '2016-06-28', counter: N + 1, pending: [ ObjectId("...") ]}
oid = ObjectId()try: updateOne({_id: '2016-06-28'}, {$addToSet: {pending: oid}}, upsert=True)except network err: try again, then throw
try: updateOne({_id: '2016-01-01', pending: oid}, {$pull: {pending: oid}, $inc: {counter: 1}}, upsert=False)except network err: try again, then throw
pipeline = [{ $match: {'pending.0': {$exists: true}}}, { $project: { counter: { $add: [ '$counter', {$size: '$pending'} ]}]
for doc in collection.aggregate(pipeline):collection.updateOne( {_id: doc._id}, { $set: {counter: doc.counter}, $unset: {pending: true} })