internal anatomy of an update
DESCRIPTION
TRANSCRIPT
Update AnatomyThe guide to how an update works
Me = Scott Hernandez● Worked on Morphia (+java), community● Over 3 years with MongoDB● Worked on update re-write in 2.6● Working on replication team now● Lots of experience with many parts
User Components
update({f:1}, {$set: {a: 2}}, {multi:true})● Query● Update Modifiers (or replacement doc)● Options
o multi, upserto writeConcern
New Code (in 2.6)● New update and query code● Replaced all modifier implementations● New validation and error messages● Using new “MutableBSON” document class● New oplog entry generation code
Internal Components● Update Command (-> request)● Update Executor: Runner + Driver● *Modifiers (Inc, Set, Unset, Bit, …)● Validation, Modification, and Replication
Two Paths To Update
Wire Protocol Op
User
Write Commands
Internal Update Request
… don’t forgetFindAndModify ;)
Update Request1. Create
o Validate argso Prepare what we can (reduce lock time)o Record system state (repl, sharding, etc)
2. Send for execution
What is a modifier{$set: {a: 1} }, {$min: {c: “b”} }
● Modifier: “set”, “min”● Field Path: “a”, “c” (+ “b”)● Value: 1, “b”
Implementing a modifierdb/ops/modifier_interface.h:● init ( modExpr, opts ) // validation● prepare ( doc, fieldName, execInfo ) // work● apply ( ) // update document● log ( ) // oplog entry created
Modifier::init● Validate form of update, and args● Parse path● Indicate if positional, and supported
Modifier::prepare● Work with doc, but don’t change it● Validate field, type, operation● Substitute positional path element, if appl.● Do work, stored to the side● Return ExecInfo (no-op, fields affected)
Modifier::apply● Apply prepared work to doc
Modifier::log● Calculate transformation logic for oplog entry● Update idempotent oplog entry
o Seto Unseto Full Replacement
db/ops/modifier_* + tests● add_to_set.h/cpp● bit.h/cpp● compare.h/cpp● current_date.h/cpp● inc.h/cpp● obj_replace.h/cpp● pop.h/cpp
● pull.h/cpp● pull_all.h/cpp● push.h/cpp● rename.h/cpp● set.h/cpp● unset.h/cpp
Update Executor● Finish prep● If no docs + upsert, prepare insert doc● Run driver (mods) against each doc● Ensure, validate, log for replication● Return UpdateResult
Update FlowQuery + Update
Cur
sor ● Init
● Prepare● Apply
{ $inc: {a:1}, { $inc: {b:1}}
● Init● Prepare● Apply
{ $set: {c: {e:1} }}
● Log
● Log
Entry
Oplog
No results, upsert:true
Inse
rt
Entry
Update Flow Notes● Yields based on timing/in-memory● Each document update is atomic● Periodically allows journal to flush, if needed
Questions?Round 1: No, then we have more slides
2.6 Stuff● New modifiers● Improvements● Behavioral Changes
New Modifiers● Mul(tiply) existing field by constant: $mul{a:1} + {$mul:{a:10}} -> {a:10}● Min/Max of existing field w/constant: $min/$max{a:1} + {$min:{a:100}} -> {a:1} (no-op)● Store current date/ts in a field: $currentDate{} + {$currentDate:{ dt: true}} -> {dt: new Date()}
Improvements● $push
o $sort elements (works with numbers/strings/etc)o $sort and/or $slice (doesn’t require $sort for $slice)o $slice off either end (no middle yet)o $position lets you specify where in the arrayo $each, $sort, $slice no longer order dep.
● $bito xor support
Improvements● Error messages with context
o The field 'l' must be an array but is of type NumberLong in document {_id: ObjectId('52a0a61672a5f2f771099b54')}
o Cannot apply $inc to a value of non-integral type. {_id: ObjectId('52a0a61672a5f2f771099b54')} has the field 'd' of non-integral type Date
Improvements● No reordering of fields (except: _id is first)● Better building of doc during insert
o update({a:1, b:1}, {$inc:{c:1}}, 0,1) -> {a:1, b:1, c:1} o now uses parsed query tree to get equalities
● Will now return new _id for all types● Accurate tracking of no-ops, modified docs
Improvements● Validation
o Immutable Fields (_id, shard keys)o Storage restrictions (no $field, “.”, _id field types)o DBRef validation
● Can now query + set _id/shard-key fieldso update({_id:1}, {$set:{_id:1, ...}}) // oko update({_id:1}, {_id:2, ...}) // error
● Cleanup bad data ($unset/$rename etc)
Questions?No, well thanks for coming!