continuous deployment at scale, baltic devops 2016
TRANSCRIPT
Scale
Deploymentcontinuous
at
@premshree
1.6 million active sellers
24 million active buyers
$2.39 billion GMS (2015)
819 employees
Premshree PillaiSr. Engineer, Etsy
tere!
principles
q & a
tooling + culture
principles
just ship
enable innovation
enable innovation(you are creating products, not code)
optimize for purpose
optimize for purpose
optimize for autonomy
optimize for mastery
experimentation
experimentationA/B testing
quick iteration
quick iteration(improving products/features)
quick iteration(fail fast > stagnant code)
continuous improvement
low MTTR
tooling
continuous deployment
continuous delivery/
commit
commit build
commit build tests
commit build tests user tests
commit build tests user tests release
commit build tests user tests release
commit build tests user tests release
frequent check-ins
frequent check-ins(to master!)
frequent check-insbranching in code
Feature
$server_config['my_feature'] = [ 'enabled' => 'on'];
$server_config['my_feature'] = [ 'enabled' => 'off'];
$server_config['my_feature'] = [ 'enabled' => 1];
$server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user'];
if (Feature::isEnabled('my_feature') {!}
if (Feature::isEnabledFor('my_feature', $user) {!}
Experimentation
ExperimentationA/B testing
$server_config['my_feature'] = [ 'enabled' => 'on'];
$server_config['my_feature'] = [ 'enabled' => [ 'layout1' => 1, 'layout2' => 3, 'layout3' => 3 ]];
if (Feature::isEnabled('my_feature')) { switch (Feature::variant('my_feature')) { case 'layout1': // layout1 code break; case 'layout2': // layout2 code break; }}
continuous integration
continuous integration(and automated tests)
keep the build green
ready to release
ready to releaseanytime
commit build tests user tests release
try
Try
Try(before you commit)
TryLib
commit build tests user tests release
try deployinator
Deployinator
commit build tests user tests release
try deployinator ci/jenkins
commit build tests user tests release
try deployinator ci/jenkins manual tests
commit build tests user tests release
try deployinator ci/jenkins manual tests deploy
commit build tests user tests release
commit build tests user tests princess prod
anybody can push
be anxious
… but do not fear
dev /
ops
♥dev /
ops
dark changes
template changes css tweaks unreferenced code
config pushes
$server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user'];
$server_config['my_feature'] = [ 'enabled' => 50, 'bucketing' => 'user'];
push train
premshree> .join
john
john + premshree
premshree>
john + premshree
premshree> .good
john + premshree
premshree> .bueno
john + premshree*
premshree>
john + premshree*
sally> .join
john + premshree* | sally
sally>
push train(with PushBot)
post-deploy
post-deploy(gaining confidence)
supergrep
dashboards
dashboards(deploy)
push
dashboards(app)
StatsD + Graphite
summary
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
try/git hooks
push train/irc
deployinator
user testing
supergrep/dashboards
poka-yokeポカヨケ
culture
assume best intentions
cultivate empathy
open
failure is an option
failure is an option(but not our intention)
failure is an option(we strongly want NOT to fail)
failure is an opportunity
post-mortems
blameless post-mortems
+ remediation
morgue
mixer
culture + tooling
deploy
fail ok
post-mortem
remediationstimulii
deploys
30+ deploys
30+ config deploys
fin
aitäh
