misadventures in continuous delivery...you know there’s a problem with continuous delivery when…...
TRANSCRIPT
Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP
2
BACKGROUND
● thredUP: the world’s largest online consignment store
● Joined thredUP in 2010 as a full-stack engineer
● Responsible for engineering + product (~60)
● Cross-functional pod system
● Isomorphic React/Apollo SPA, GraphQL orchestration, Ruby/Java/Node on the back-end
● Migrated to k8s three years ago
Confidential 3
4
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
5
ENGINEERING MISADVENTURES
6
WHY FOCUS ON MISADVENTURES?
● To avoid making the same mistakes we did
● Everyone can relate
● Learnings in how to scale are hard to get
Confidential
‹#›
You know there’s a problem with continuous delivery when…
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce?
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”?
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”? Have you had to do any dev work for anything related to “Cyber Week”?
11
CYBER WEEK IN E-COMMERCE
12
CYBER WEEK IN E-COMMERCE
13
CYBER WEEK IN E-COMMERCE
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
15
HOW TO PICK THE ENGINEER
16
OUR DEPLOY PIPELINE
17
CI PIPELINE CHECKLIST
● Is your end-to-end deploy pipeline quick?
● Is it safe? Does it gracefully degrade?
● Can you roll back with ease?
● Is the deployment process sustainable?
● Do engineers have confidence when deploying?
18
CI PIPELINE CHECKLIST
● Is your end-to-end deploy pipeline quick?
● Is it safe? Does it gracefully degrade?
● Can you roll back with ease?
● Is the deployment process sustainable?
● Do engineers have confidence when deploying?
19
LAZY ENGINEERS
“I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.” - Bill Gates
20
CLEVER ENGINEERING
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
22
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
RAISE YOUR HAND IF:
Have you ever tried to decompose a monolith?
24
MONOLITH DECOMPOSITION Has anyone ever done something small and easy like decompose a monolith?
25
MONOLITH DECOMPOSITION ● Migrate RethinkDB to AuroraDB
● Pull out user personalization into separate service
● Pull out shopping cart responsibilities into separate service
● Introduce RabbitMQ so the back-ends can talk to each other
● Consolidate all client API requests to GraphQL orchestration layer
● Pull out auction responsibilities into separate service
26
MIGRATION DEPLOYS (IN THEORY - BEFORE)
27
MIGRATION DEPLOYS (IN THEORY - DURING)
28
MIGRATION DEPLOYS (IN THEORY - AFTER)
29
MIGRATION DEPLOYS (IN PRACTICE - BEFORE)
30
MIGRATION DEPLOYS (IN PRACTICE - DURING)
31
MIGRATION DEPLOYS (IN PRACTICE - AFTER)
32
WHY HOLE-SHOT DEPLOYS ARE PROBLEMATIC
● Staging != Production ● Extended QA in staging -> elongates the development process
● Risk for the business -> creates unnecessary stress for the team
● Repeated stress -> recipe for burn out
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
34
FEATURE BRANCH WORKFLOW (IN THEORY)
35
FEATURE BRANCH WORKFLOW (IN PRACTICE)
36
WHEN YOU HAVE TO ROLL BACK THE FEATURE YOU DEPLOYED
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
38
CONTINUOUS MERGE WORKFLOW
39
CODE REVIEW FOR LONG PULL REQUESTS
40
CONTINUOUS WORKFLOW BENEFITS
● Bite-sized pull requests -> easier code review ● Continuous deploys -> earlier production feedback
● More granular time estimations -> consistency -> predictability -> sustainable
● Last 10% of the project is actually the last 10%
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
42
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
43
“SHOP HER CLOSET” LAUNCH
44
LAST MINUTE CHANGES
45
WHEN YOU REALIZE YOU HAVEN’T ROLLED BACK ON K8S BEFORE
46
WHEN YOU’RE TRYING TO THINK THROUGH A CI BYPASS
47
LAST MINUTE CHANGES
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
6. You have Devs who don’t Ops
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
6. You have Devs who don’t Ops
7. You’re satisfied with continuous delivery
50
CLOSING THOUGHTS
“The most important characteristic of high performing teams is that they are never satisfied: they always strive to get better.” - Jez Humble
THANK YOU
Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP Twitter: @dandemeyere Medium: medium.com/@dandemeyere Email: [email protected]