dockercon eu 2015: placing a container on a train at 200mph
TRANSCRIPT
![Page 1: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/1.jpg)
Placing a container on a train at 200mph
Casper S. JensenSoftware Engineer, Uber
![Page 2: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/2.jpg)
About Me
● Joined Uber January 2015,Compute PlatformDenmark, Aarhus office
● PhD, CSOn a completely unrelated topic
● Linux aficionado
● Docker “user” since February
![Page 3: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/3.jpg)
About UBERWhy all the fuzz?
![Page 4: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/4.jpg)
The UBER app
4
![Page 5: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/5.jpg)
339 Cities
5
![Page 6: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/6.jpg)
61 Countries
6
![Page 7: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/7.jpg)
2.000.000+ Trips/day
7
![Page 8: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/8.jpg)
4000+ Employees
8
![Page 9: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/9.jpg)
![Page 10: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/10.jpg)
Not that hard...
10
You just have to handle
● 24/7 availability across the globe● Very different markets● 1000s of developers and teams● Adding new features like there’s no tomorrow
UberPOOL, UberKITTEN, UberICECREAM, UberEATS, UberWHATEVERYOUCANIMAGINE
● Hypergrowth in all dimensions● Datacenters, servers, infrastructure, etc
Basically, you have to make magic happen every time a user opens the application
![Page 11: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/11.jpg)
Software DevelopmentThe old UBER way
![Page 12: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/12.jpg)
A fair amount of frustration
12
1)Write service RFC2)Wait for feedback3)Do all necessary scaffolding by hand4)Start developing your service5)Wait for infra team to write service scaffolding6)Wait for IT to allocate servers7)Wait for infra team to provision servers8)Deploy to development servers and test9)Deploy to production10)Monitor and iterate
Steps 5–7 could take days or weeks...
![Page 13: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/13.jpg)
It's just not scalable
13
But you have to start somewhere
![Page 14: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/14.jpg)
—Internal e-mail, February 2015
“Make it easier for service owners to manage their local
service environments.”
14
![Page 15: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/15.jpg)
![Page 16: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/16.jpg)
New development process
16
1)Write service RFC2)Wait for feedback3)Do all necessary scaffolding using tools4)Start developing your service
5)Deploy to development servers and test6)Deploy to production7)Monitor and iterate
![Page 17: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/17.jpg)
![Page 18: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/18.jpg)
No silver bullets
![Page 19: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/19.jpg)
All the things you did not consider
19
● Routing● Dynamic service discovery● Deployment● Placement engine● Logging and tracing● Dual build environments● Handling of secrets● Security updates● Private repositories● Replicating images across multiple datacenters
Also, how much freedom do you really want to give your developers?
![Page 20: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/20.jpg)
Change all the things!Let's go through some examples
![Page 21: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/21.jpg)
uDeploy
21
● Rolling upgrades● Automatic rollbacks on failure● Health checks, stats, exceptions,
○ Load-, and system-tests● Service building● Build replication● 4.000+ upgrades/week● 3.000+ builds/week● 300+ rollbacks/week● 600+ managed services
Our in-house deployment/cluster management system
![Page 22: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/22.jpg)
Moving to docker with zero downtime
22
Build multiplexing
We want to keep on trucking while migrating to docker
![Page 23: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/23.jpg)
Build process & scaffolding
23
Declarative build scripts
● Service configuration in git● Preset service frameworks● Many options● Generator creating
○ Dockerfile○ Health checks○ Entry point scripts inside container○ In general, all glue between host and service
● Possible to supply custom Dockerfile
service_name: test-uber-serviceowning_team: udeploybackend_port: 123frontend_port: 456service_type: clay_wheelclay_wheel: celeries: - queue: test-uber-service
has_celerybeat: true
![Page 24: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/24.jpg)
Image replication
24
● Multiple datacenters● Images must be stored within DCs● Build once, replicate everywhere● Traffic restrictions, push but not pull
Current setup● Stock docker registry● File back-end● Docker-mover● Syncing images using pull/push● Use notification API to speed up replication
![Page 25: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/25.jpg)
Service discovery & routing
25
● Previously, we used HAProxy + scripts to do this● Now, we use Hyberbahn + TChannel RPC
https://github.com/uber/{hyperbahn|tchannel}○ Used for docker and legacy services○ Required in order to move containers around in seconds○ Dynamic routing, circuit breaking, retries, rate limiting,
load balancing○ Completely dynamic, no fixed ports
![Page 26: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/26.jpg)
Key Take-Aways
![Page 27: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/27.jpg)
27
● Remove team dependencies● More freedom● Not tied to specific frameworks
or versions (hi, Python 3)● Easy to experiment with new
technologies
● Too much freedom● Non-trivial integrating with a
large running system● Infrastructure must be dynamic
throughout● Containers are only a minor
part of the infrastructure, don't forget that
The good & the bad
![Page 28: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/28.jpg)
Current and future wins● Today, 30% of all services in docker● Soon-ish, 100%
● Great improvements in provisioning time (done)● Framework and service owners can manage their own
environment (done)● Faster and automatic scaling of capacity (in progress)
![Page 29: DockerCon EU 2015: Placing a container on a train at 200mph](https://reader036.vdocument.in/reader036/viewer/2022062522/58817aa41a28abf7478b7a15/html5/thumbnails/29.jpg)
Thank you!Casper S. [email protected]