container orchestrators and dcos · 2016-09-21 · -apache mesos, drill pmc -help maintain apache...

44
1 Container Orchestrators and DCOS Timothy Chen

Upload: others

Post on 20-May-2020

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

1

Container

Orchestrators and

DCOS

Timothy Chen

Page 2: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

About me:

- Previously Mesosphere Lead Engineer, (ex-

Microsoft/VMWare)

- Apache Mesos, Drill PMC

- Help maintain Apache Spark on Mesos

- Co-founder on new Container startup (still stealth)

Page 3: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Outline

- The Need of Container Orchestration

- What is DCOS?

- Breaking down DCOS

- Demo

Page 4: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 5: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 6: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 7: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 8: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 9: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 10: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 11: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 12: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 13: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 14: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Facebook -> Tupperware

Google -> Borg / Omega

Yahoo -> YARN (Hadoop)

Twitter -> Mesos / Aurora

Page 15: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 16: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 17: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 18: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 19: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 20: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

20

Page 21: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 22: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 23: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 24: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 25: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Why is DCOS awesome?

- Containers! (High resource utilization, etc)

- Extensible Isolation

- Public and Private Service repositories

- Cloud Agnostic Installer

- Web and Command Line Interfaces

- Service Discovery and Load Balancing

Page 26: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 27: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

27

Page 28: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

System Components

- Admin Router

- Mesos DNS

- Cosmos Service

- Marathon

- Diagnostic + Signal Service

- Minuteman Service

- DNS Proxy

- History Service

- Auth Service

Page 29: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Admin Router

https://github.com/dcos/adminrouter

Page 30: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 31: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Mesos DNS

https://github.com/mesosphere/mesos-dns

Page 32: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Cosmos

https://github.com/dcos/cosmos

- Package management server

- Handles packaging discovery, serving, install

Page 33: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Marathon

- DCOS Init system

- Deployment

- HA

- Health Checks

Page 34: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Diagnostics / Signal Service

- 3DT monitoring agent + REST API, collects node health and

DC/OS specific services health

- Signal service forwards telemetry to consumer (SegmentIO,

etc)

https://github.com/dcos/dcos-signal

https://github.com/dcos/3dt

Page 35: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Minuteman

- DC/OS Load balancing + Service discovery solution

- Distributed L4 load balancer with VIPs

https://github.com/dcos/minuteman

Page 36: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Install DC/OS

- Local Vagrant

- Cloud Provisioned (AWS, Azure)

- Custom installation (GUI, scriptable)

Page 37: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 38: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 39: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 40: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

DC/OS Packages

- A package:

- Mesos framework (Marathon JSON)

- CLI

- Metadata (Description, Icon)

- Universe stores all packages index and versions

- (http://github.com/mesosphere/universe)

- DCOS CLI contacts Cosmos to un/install packages

Page 41: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +
Page 42: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Demo!

http://github.com/mesosphere/tweeter

Page 43: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Next big question moving forward…

How do we run containers much more efficiently at

scale?

Page 44: Container Orchestrators and DCOS · 2016-09-21 · -Apache Mesos, Drill PMC -Help maintain Apache Spark on Mesos -Co-founder on new Container startup ... - 3DT monitoring agent +

Thanks!

@tnachen

[email protected]