© 2016 Mesosphere, Inc. All Rights Reserved.
Edward Hsu, VP of Enterprise DC/OS@edwardihsu
JOURNEY TO THE MODERN APP WITH CONTAINERS, MICROSERVICES AND BIG DATA
Datacenter Operating System (DC/OS)The simplest way to build, run, scale and manage modern enterprise apps
© 2016 Mesosphere, Inc. All Rights Reserved.
INDUSTRY TRANSITIONS
Customer analytics2000s
Electronic customer records1980s
Online customer engagement1990s
Real-time & predictive customer engagement2013+
© 2016 Mesosphere, Inc. All Rights Reserved.
MAYO CLINIC REVOLUTIONIZING HEALTHCARE
Individualized care recommendations based on 5 million patient records
Key technologies:
● Real-time data ingestion● Distributed search engine● Large-scale processing● Distributed database
© 2016 Mesosphere, Inc. All Rights Reserved.
UBER DISRUPTING MULTIPLE INDUSTRIES
Operations in over 300 cities around the world
Key technologies:
● Distributed logging ● Large-scale data processing● Distributed datastore
© 2016 Mesosphere, Inc. All Rights Reserved.
ARCHITECTURAL SHIFT
TRADITIONAL ENTERPRISE APPLICATIONS
MODERN ENTERPRISE APPLICATIONS
Data
Code
Latency
Users
2+ Billion smartphone users
3+ Billion internet users
Enterprise data growth (CAGR): 40%+
Source: KPCB Internet Trends 2016, EMC Digital Universe 2014
© 2016 Mesosphere, Inc. All Rights Reserved.
Streaming
10
THE MODERN ENTERPRISE APP
BIG DATA SERVICESMICROSERVICES
STATELESS CONTAINERS ANALYTICS
Batch Machine Learning
Search
DATABASES
Time Series
SQL / NoSQL
© 2016 Mesosphere, Inc. All Rights Reserved.
MODERN ENTERPRISE APPS
Containers Data Services
MicroservicesOpen source & Social coding1
2
3
4
© 2016 Mesosphere, Inc. All Rights Reserved.
CONTAINERS SIMPLIFY PACKAGING AND DEPLOYMENT
Private Copy
Shared
App 3
Libraries
Guest Operating
System
Infrastructure
App 1
Libs
Host Operating System
Start / stop time One minute to few seconds Milliseconds
Workload density 10 - 100x1x
App 2
Libraries
Guest Operating
System
App 1
Libraries
Guest Operating
SystemApp 2
Libs
App 3
Libs
Virtual Machines Containers
Host Operating System
Hypervisor
Infrastructure
App 4
Libs
App 5
Libs
2
© 2016 Mesosphere, Inc. All Rights Reserved.
JUST HOW FAST ARE CONTAINERS?
Larry Rau from @Verizon with @flo Launching 50,000 containers in seconds with @mesosphere #DC/OS
© 2016 Mesosphere, Inc. All Rights Reserved.
MICROSERVICES ENABLE FAST AND FREQUENT RELEASES
Traditional Architecture
Many functions in a single process
Siloed teams
REST
API
s
Microservices Architecture
Cross-functional teams organized around capabilities
Scales individually
Each element of functionality defined as “microservices”
Scales monolithically
3
© 2016 Mesosphere, Inc. All Rights Reserved.
DATA SERVICES PROVIDE CONNECTION AND PERSISTENCE
Data Ingestion
Response
Devices
ClientSensors
Event Bus
Reactive App
Database
Analytics
Use Case Examples
● Anomaly detection
● Personalization
● IoT Applications
● Predictive Analytics
● Machine Learning
4
© 2016 Mesosphere, Inc. All Rights Reserved. 17
BUT WHAT ABOUT THE INFRASTRUCTURE?
CaaSPaaS
Container AppContainer App
Stateful Service #1
V1
Big Data Analytics #1
V1
Stateful Service #2
V1
Stateful Service #2
V2
PaaS
Container App
Team A
Big Data Analytics #1
V2
Big Data Analytics
#2V1
Team B
© 2016 Mesosphere, Inc. All Rights Reserved. 18
BUT WHAT ABOUT THE INFRASTRUCTURE?
CaaSPaaS
Container AppContainer App
Stateful Service #1
V1
Big Data Analytics #1
V1
Stateful Service #2
V1
Stateful Service #2
V2
PaaS
Container App
Team A
Big Data Analytics #1
V2
Big Data Analytics
#2V1
Team B
● Extremely low utilization - less than 10%, due to static partitioning
● Manual operations - Weeks to provision, wasted time managing individual machines
● Difficult to experiment with new tech - months to provision environments to evaluate new technologies
● High risk - Failure from manual operations and difficulties maintaining high availability
© 2015 Mesosphere, Inc. All Rights Reserved.
WHERE HAVE WE SEEN THIS PROBLEM BEFORE?
You are launching Google Chrome.Which core would you like to use?
Core 1 Core 2
Core 3 Core 4
You are launching a distributed app requiring 500 cores.Which of your 50,000 cores would you like to use?
Core 1 Core 2 Core 3 Core 4 Core 5
Core 6 Core 8 Core 9 Core 10Core 7
Core 11 Core 13 Core 14 Core 15Core 12
Core 16 Core 18 Core 19 Core 20Core 17
Core 21 Core 23 Core 24 Core 25Core 22
Core 26 Core 28 Core 29 Core 30Core 27
Core 31 Core 33 Core 34 Core 35Core 32
Core 36 Core 38 Core 39 Core 40Core 37
Core 41 Core 43 Core 44 Core 45Core 42
Core 46 Core 48 Core 49 Core 50Core 47
© 2015 Mesosphere, Inc. All Rights Reserved.
MODERN ENTERPRISE APPS REQUIRE WE OPERATE AT THE LOGICAL DATACENTER LEVEL
Traditional App Modern Enterprise App
© 2015 Mesosphere, Inc. All Rights Reserved.
YES, THIS TOO CAN BE SOLVED WITH ABSTRACTION
Scheduler
Executor
Task Launch
Task Launch
Task Status
Task Status
Resource Offer
ExecutorExecutorExecutor
Scheduler
Executor
Task Launch
ExecutorExecutor
Task Status
Scheduler
Executor
Task Launch
ExecutorExecutor
Task Status
Distributed System A Distributed System B Datacenter Operating System Approach
Distributed Systems A+B+C+...
Apache Mesos Two-level Scheduling
© 2015 Mesosphere, Inc. All Rights Reserved.
Datacenter Operating System (DC/OS)
DC/OS
...Agent 1 Agent 2 Agent 3 Agent n
Server VM Cloud VM
...Modern App Components
FROM SERVER TO DATACENTER-SCALE
Server Operating System
OS (e.g., Windows, Linux)
...Core 0 Core 1 Core 2 Core n
Server
Process
© 2016 Mesosphere, Inc. All Rights Reserved. 23
MODERN APPS WITH THE DC/OS MODEL
Traditional Approach
Big Data Analytics
Stateful Service
DC/OS Model
Container App
Container App
CaaS PaaS
● Turnkey install of datacenter-wide services● Simplified operations & efficiency● Lowers barriers to experiment with new tech (e.g., Spark)
Container App
Datacenter-scale operating system
© 2016 Mesosphere, Inc. All Rights Reserved.
BRINGING MODERN APPS TO EVERYONE
● 100% Open source
● No limits
© 2016 Mesosphere, Inc. All Rights Reserved.
DC/OS UNIVERSE: STORE FOR APP SERVICES
DC/OS
Anyone can publish on DC/OS Universe app store - partners & OSS contributors
Popular services include Spark, Cassandra, Jenkins, and Kafka
Services include distributed systems that run elastically across the datacenter
Install these services with a single command
Spark
Jenkins
Riak
DataStax Enterprise
Confluent Kafka
ArangoDB
GitLab
Cassandra
JFrog Artifactory
Elasticsearch
MariaDB
Storm
HDFS
Zeppelin
MemSQL
Over 40 Services Made For DC/OS
© 2016 Mesosphere, Inc. All Rights Reserved.
“We’re running real time analytics and visualization of data from connected sensors
Before DC/OS we had no where near the scale we had today, from thousands of to millions of events per second.
Next, we’re doing predictive GIS”
Adam Mollenkopf, Real Time Geospatial Information Systems Capability Lead
© 2016 Mesosphere, Inc. All Rights Reserved.
“We are building a PaaS and services on an existing OpenStack cloud to enable developers to build microservices on Docker containers and use big data technologies...
DC/OS was a force multiplier for us to get these new services out the door fast...
We transitioned the full enterprise infrastructure at 2X the speed of previous efforts.”
Tim PletcherEngineering Director, Application Services
© 2016 Mesosphere, Inc. All Rights Reserved.
“We came across Mesosphere while planning for the coming IoT market..
If I build everything in a silo, I have no chance...
With DC/OS, I have one single contiguous cluster and I can bring my data and store it and I can run all my applications as well..
and I think that's a huge advantage”
Larry Rau, Director Architecture & Infrastructure, Verizon Labs
© 2016 Mesosphere, Inc. All Rights Reserved.
SPEED MATTERS
McKinsey Quarterly 2015: “The perils of ignoring software development”Based on assessment of software development projects at 1,300 companies
Top quartile companies bring new software features to market
3x faster than bottom quartile