containerdayvietnam2016: docker at scale with mesos

Docker at scale with Mesos

Phm Tun Anh Lead engineer @ VCCloud Platform Services

$ whoami

Lead engineer at VCCloud Platform Services

Build & work with large-scale/production platform services for about 6 years

Services: Storage, DNS, CDN, DDoS Protection, Logging, Private PaaS

Agenda

1. What is Mesos?

2. How to run docker containers on Mesos

3. Lessons learned

4. Q&A

1. What is Mesos?

2. How to run docker containers on Mesos

3. Lessons learned

4. Q&A

https://blog.twitter.com/2016/overview-of-the-twitter-cloud-platform-compute

https://blog.twitter.com/2016/overview-of-the-twitter-cloud-platform-compute

https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/

https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/

Powered by Mesos: Apple

Atlassian

CERN

Cisco

Cloudflare

Foursquare

Groupon

HubSpot

Netflix

Opera

Paypal

Shopee

Shopify

Saleforce

Twitter

Uber

Vimeo

Verizon

Weibo

Yelp

Xiaomi

http://mesos.apache.org/documentation/latest/powered-by-mesos/

http://mesos.apache.org/documentation/latest/powered-by-mesos/

Solomon Hykes, Founder & CTO of Docker, said at DockerCon EU 12/2014:

Mesos is the gold standard for large-scale production clusters running containers

Mesos is:

Open source, top-level Apache project

Proven at massive scale

Multi-resource (CPU, RAM, Disk, GPU) scheduling

Simplified operational model

Cross platform

Native Docker support

PaaS: Marathon, Aurora, Kubernetes, Swarm

Big Data: Hadoop, Spark, Storm

Database: Cassandra, ArangoDB,

ElasticSearch, Chronos, Jenkins, Kafka

Mesos Frameworks:

1. What is Mesos?

2. How to run docker containers on Mesos

3. Lessons learned

4. Q&A

Apache Aurora

Google Kubernetes

HubSpot Singularity

Mesosphere Marathon

Features:

Start, stop, scale, update, rollback apps

Rolling deploy / restart

Constraints

Health checks

App dependencies

Highly available, no SPoF

1. What is Mesos?

2. How to run docker containers on mesos

3. Lessons learned

4. Q&A

Immutable Infrastructure / CodeShip

Some of the major challenges today when building infrastructure are predictability, scalability and automated recovery.

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

Choose a container style:

"thin" single-process container

"fat" container

sshd in container

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

Services Discovery

Mesos-DNS:

..marathon.mesos

record types: A / SRV

simple & stateless

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

Networking

Project Calico:

IP-per-container

Pure Layer 3

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

Managing Secrets

ansible-vault

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

App deployment

ansible-playbook

marathon.json

docker tag name = git commit hash

1 mesos cluster (3 VMs) per dev

deploy to servers deploy to marathon

marathon.json

https://flask-hello.marathon.mesos.vn/

https://flask-hello.marathon.mesos.vn/

Monitoring

Containers RAM used

Stateful Containers

Local persistent volumes

Container style

Service discovery

Networking

Lessons Learned

Managing secrets

App deployment

Docker annoyances

Docker

restart / upgrade docker without stopping the running containers

old images / tags cleanup

containers stdout / stderr log files

docker pull

docker hub

"Normal" container vs cloud-native container:

No manual steps

Service discovery friendly

Config files should be optional

Using env vars for config

1. What is Mesos?

2. How to run docker containers on Mesos

3. Lessons learned

4. Q&A

Links

Mesos: http://mesos.apache.org/

Marathon: https://mesosphere.github.io/marathon/

Project Calico: https://www.projectcalico.org/

Mesos-DNS: https://github.com/mesosphere/mesos-dns

A Healthy Platform Checklist: https://gist.github.com/andy-pham/04dedff872b33c07a31277afb900950b

http://mesos.apache.org/https://mesosphere.github.io/marathon/https://www.projectcalico.org/https://github.com/mesosphere/mesos-dnshttps://gist.github.com/andy-pham/04dedff872b33c07a31277afb900950b