Scaling docker @OVH

Timothée GERMAIN devops@OVH

From 20 to 2500 running containers

About meTimothée GERMAIN

Devops for Containers team @ OVH since 2015

Work with containers since 2014

Twitter : DarkNihilius1

Docker @ OVH● 8 clusters

○ Internal production : 5○ External beta/gamma : 2○ Preproduction : 1

● 2500 running containers● 800 Hosts● 3000 cpu cores● 12TB RAM● 2 docker registry (internal & external)● 100s of deployment per day● 5 people

The beginning : 1 - 20 containersEverything fit on 1 host

+ Easy to deploy docker run+ Easy to build docker build+ Easy to debug docker exec/logs/stats

- No high availability- Not scalable

Scaling up : +20 containers, 1 team

You need an orchestrator :

● Docker swarm (Docker inc.)● Mesos & Marathon (Apache Software Foundation)● Kubernetes (Google then CNCF)

Don’t fit on 1 host ? Need high availability ?

Mesos & marathon in 60 seconds

Linux kernel : provide abstraction of hardware

Linux kernel

Your application

Linux schedulerScheduler : say which application run when with which resources

Mesos & marathon in 60 seconds

Mesos : provide abstraction of multiple hosts

Mesos

Your (Docker) application

FrameworkFramework (Marathon) : say which application run when with which resources

Needs :

● Isolate team/client● Be cost efficient, mutualize what we can

Scaling up : 20+ containers, 2+ teamMore team !

Scaling up : 20+ containers, 2+ teamMarathon : no multitenancy out of the box

The solutions we try :

1) One marathon per team2) One marathon with logical isolation (middleware)3) One mesos & marathon per team

Multitenancy : One marathon per team+ Good isolation+ Mesos mutualization+ Fair blast radius- Does not scale well

Mesos

Marathon B

Marathon CMarathon A

Multitenancy : One marathon and logical isolation+ Scale+ Give us control on user input- Less isolation- Lots of maintenance- Hard to keep up to date with

marathon API- Big blast radius

Mesos

Marathon

Middleware

Multitenancy : One marathon & mesos cluster per team+ Scale even better+ Little blast radius+ Best isolation+ Modular- Less/no mutualisation- Costly- Lot of parts to deploy and

maintainMesos B

Marathon B

Mesos A

Marathon A

Deploy mesos and marathon using mesos and marathonLot of parts to deploy and maintain ?

we already have a tool for that : Mesos and Marathon !

Mesos B

Marathon B

Mesos A

Marathon A

Mesos Admin

Marathon Admin

Questions ? Thanks for your time

Log and metrics : 1 - 20 containersDocker stats

Docker logs

Direct calls on docker daemon

Log and metrics : +20 containers, 2+ teamNot docker logs nor docker stats