mantl data platform, microservices and bigdata services

Dmitri Chtchourov,

MANTL Data Platform, Microservices and BigData Services

Innovation Architect, CIS CTO Group

Agenda

Problem & Opportunity

What do we want to do?

What is in it for us?

How does it work?

What have we done so far?

Anatomy of a Service

Reference Architectures and real use cases

Putting it all together

Problem & Opportunity

Rapid innovation in computing and application development services

No single service is optimal for all solutions

Customers want to run multiple services in a single cluster and run multiple clusters in Intercloud environment

...to maximize utilization ...to share data between services

…Complex/BigData and Microservices together

Technologies matrix* Service Product Cloud/Virtualization CIS/AWS/Metacloud/UCS… Provisioning Open Stack/Terraform Automation Ansible

Clustering & Resource Management Mesos, Marathon, Docker Load Balancing Avi Networks ETL & Data Shaping StreamSets Log Data Gathering Logstash Metrics Gathering CollectD, Avi Networks Messaging Kafka, Solace Data Storing (Batch) HDFS Data Storing (OLTP/Real-time) Cassandra Data Storing (Indexing) Elastic search Data Processing Apache Spark Visualization Zoomdata*Subset example

Cloud Management

Data Collect

Data Storage

Data Processing

Visualisation

Technologies stack

Datacenter and solution today

VM7or

BM7

VM8or

BM8

VM4 or

BM4

VM5or

BM5

VM6or

BM6

VM1 or

BM1

Visualization Service

Data Ingestion Service

Analytics Service

• Configuration and management of 3 separate clusters

• Resources stay idle if service is not active

• Need to move data between clusters for each service

VM2 or

BM2

VM3 or

BM3

VM1 or

BM1

VM2 or

BM2

VM3 or

BM3

What do we want to do?

Data Ingestion Service

Analytics Service

Visualization Service

….to maximize utilization...to share data between services

Shared clusterMultiple clusters

Shared Cluster

CIS/AWS/Metapod/UCS…

VM1 or

BM1

VM2 or

BM2

VM3 or

BM3

VM4 or

BM4

VM5 or

BM5

What is in it for us?

Maximize utilizationDeliver more services with smaller footprint

Shared clusters for all services Easier deployment and management with unified service platform

Shared data between services Faster and more competitive services and solutions

Combine paradigms for flexibility and functionality Run complex services and microservices in the single environment

How does this work?

Mesos Slave

Spark Task Executor Mesos Executor

Mesos Slave

Docker Executor Docker Executor

Mesos Master

Task #1 Task #2 ./python XYZ java -jar XYZ.jar ./xyz

Mesos Master Mesos Master

Spark Service Scheduler Marathon Service SchedulerZookeeper quorum

How does this work?

Mesos provides fine grained resource isolation

Mesos Slave Process

Spark Task Executor Mesos Executor

Task #1 Task #2 ./python XYZ

Compute Node

ExecutorContainer(cgroups)

How does this work?

Mesos provides scalability

Mesos Slave Process

Spark Task Executor

Task #1 Task #2 ./ruby XYZ

Compute Node

Python executor finished,more available resourcesmore Spark

Container(cgroups)

Task #3 Task #4

How does this work?

Mesos has no single point of failure

Mesos MasterMesos MasterMesos Master

VM1or

BM1

VM2 or

BM2

VM3or

BM3

VM4or

BM4

VM5or

BM5

Services keep running if VM fails!

How does this work?

Master node can failover

Mesos MasterMesos MasterMesos Master

VM1or

BM1

VM2or

BM2

VM3or

BM3

VM4or

BM4

VM5or

BM5

Services keep running if Mesos Master fails!

How does this work?

Slave process can failover Tasks keep running if Mesos Slave Process fails!

Mesos Slave Process

Spark Task Executor

Task #1 Task #2 ./ruby XYZ

Compute Node

Task #3 Task #4

How does this work?

Can deploy in many environments Get orchestrated by Openstack, Ansible (scripts), Cloudbreak

True Hybrid Cloud deployment: CIS, AWS, UCS, vSphere, other

CIS/

CIS/AWS/Metpod/vSphere/UCS…

Terraform

REST API REST API

Scripted provisioning

Direct provisioning

Policy, Auto-scaling

VM1or

BM1

VM2or

BM2

VM3or

BM3

VM4or

BM4

VM5or

BM5

How does this work?

Microservices managed and scaled separately Microservices managed by Mesos in a single platform

Microservices architecture for Mesos frameworks and other components

CIS/AWS/Metacloud/vSphere/UCS…Terraform

Spark Executor

N

Spark Executor 1

Spark Scheduler

Kafka Broker N

Kafka Broker 1

Kafka Scheduler

Docker Docker TraefikMicroservices …

REST API REST API

Scripted provisioning

Direct provisioning

Policy, Auto-scaling

VM1or

BM1

VM2or

BM2

VM3or

BM3

VM4or

BM4

VM5or

BM5

What have we done so far?Working with partners on adopting and co-developing services

Partners Co-development Partners

Anatomy of the service/framework

Riak is Basho Technologies distributed highly available database

Optimized Multi-Datacenter operation

We are working together with Basho Labs on developing and testing their Mesos Service version of the product

Riak Use Cases

Online / Commerce● Session Control● Shopping Cart● Product Ratings and Reviews

Internet of Things● Connected Device Data● Sensor Data● Log Data

Content Management● Storing Unstructured Data● Content Personalization● Advertising Data

Gaming● Store Leaderboard Info● Store Bet Transactions● Online Chat

Digital Communications● Online Community Chat● Notification and Alerting● Mobile Messaging

Development phases

Phase 0: Package application in Docker container to deploy on Mesos

Phase 1: Convert application to Microservices Architecture to deploy as Mesos application with multiple components

Phase 2: Create an intelligent scalable Mesos service based on the application

Riak Service: Components

Riak Service: Architecture

Riak Service: Persistence

Riak Service: Operational Simplicity

Riak Service: Highly ScalableE-commerce Application with Varying Traffic


Zoomdata is distributed highly available large scale visualization platform

Optimized very big data set micro-query analytics

We are working together with Zoomdata on developing and testing their Mesos Service version of the product

Zoomdata Service: Components

Zoomdata Service: Mesos + Kubernetes

Mesos Slave

Mesos Master

Mesos SlaveMesos Slave

Zoomdata web app

MongodbSpark Worker

Spark Executor

Spark Executor

Proxy(haproxy, nginx)

KubernetesMongoService/RC

KubernetesSpark-ProxyService/RC

Spark-Proxy

Zoomdata web app

Zoomdata web app

Kubernetes Framework

KubernetesZoomdataSchedulerService/RC

Zoomdata Scheduler

ProxyGen Script

User

● Every component (Zoomdata App, MongoDB, Spark-Proxy, Scheduler) must be started in independent K8s Pod and there must be exactly one MongoDB, Spark-Proxy and Scheduler Pods meanwhile Zoomdata App can be scaled with help Kubernetes Replication Controller.

● There must be defined Kubernetes Service for MongoDB, Spark-Proxy, Scheduler as they will be used in Zoomdata’s App Pod. Every docker container will have env variables for every present Service injected automatically.


StreamSets is an open source continuous big data ingest infrastructure

Accelerates time to analysis with unprecedented transparency and processing to data in motion.

Cluster deployments

JVM, Docker, Spark Streaming on Mesos

Continuous Operations to Minimize downtime

Advantages of Streamsets

Adaptable Data flow - Design and execute intent-driven data flows in a graphical IDE

Instream Sanitization - transform and process the data on the fly

Intelligent Monitoring - Get early warnings, detect anomalies and take action

Link origins to destinations with in-stream data preparation

Streamsets Data Pipeline

MESOS

Streamsets Data Collector (SDC) ArchitectureCluster Streaming modeData Collector runs as an application within Spark Streaming, Spark Streaming runs on Mesos cluster manager to process data from a Kafka cluster.

The Data Collector uses a cluster manager and a cluster application to spawn workers as needed.

Cluster Batch Mode :Data Collector processes all available data from HDFS and then stops the pipeline.

MapReduce generate additional worker nodes as needed.

Standalone modeSingle Data Collector process runs the pipeline. A pipeline runs in standalone mode by default.

MANTL Data Platform OverviewA modern, batteries included platform for rapidly deploying globally distributed services.Mantl’s goal is to provide a fully functional, instrumented, and portable container based PaaS for your business at the push of a button

1) Easy deployment and configuration on different

platforms

2) High availability and self-healing

3) Multi-datacenter support

4) Linear scalability

5) Smart resource management

6) Wide range of supported frameworks

MANTL nodes

Consul for service discoveryMesos cluster managerMarathon for cluster management Docker container runtimeZookeeper for configuration management

Docker containersAny Mesos-based workloads

Traefik for proxying external traffic into services running in the cluster

Security & Operation

Frameworks

Platform Support

Mantl Components

CoreComponents

➢ Data Storage - Riak, Cassandra, HDFS➢ Data processing - Spark➢ Security - Vault

➢ Data ingestion – Kafka➢ Metrics collection - Collectd➢ Logs forwarding - Logstash

➢ Provisioning - Terraform, Ansible➢ Cluster management – Mesos, Marathon➢ Service discovery and configuration management - Consul, Zookeeper,

Traefik➢ Container runtime - Docker

➢ Cisco Cloud Services, Cisco MetaCloud➢ Amazon Web Services➢ Google Compute Engine

➢ Openstack➢ DigitalOcean➢ Bare Metal

➢ Autoscaling and high availability➢ Application load balancer➢ Application dynamic firewall

➢ Manage Linux user accounts➢ Authentication and authorization

for Consul, Mesos, Marathon

Long Running Services

Big Data Processing

Batch Scheduling

Supported Mesos Frameworks

Data Storage

Mesos makes it easy to develop distributed systems by providing high-level building blocks.

ANALYTICS PLATFORM MANAGEMENT

Data Ingestion•Kafka, Streamsets configurators

Data StorageRiak, Cassandra, HDFS

Model DevOps Machine learningMLLib, Spark

Model Deployment•Model loading, versioning

Cluster Management & Scheduling

Cluster managerMesos

Cluster Management long running service

Marathon

Service DiscoveryConsul

Distributed Virtual networkCalico ETCD

ADVANCED ANALYTICS APPS

Analytics Accelerators as Apps•Forecasting, NLP, optimization, enrichment etc.

SPECIALIZED ADVANCED ANALYTICS MODELS

Consulting Services Design, Build, Deploy Maintain, Manage Performance

DASHBOARDS

ZoomData Tableau, Qlik, Spotfire, Excel/BI Cubes …

BUSINESS APPS

Custom ZoomData Visualizations (D3) Custom Applications Customer System Integration

CUST

OM

IZAT

ION

&M

ANAG

ED S

ERVI

CES

CISCO INTERCLOUD

Customization MANTL Data Platform

Sample Architecture for Batch Data Processing

Cassandra

Elastic Search

Spark

Spark Mllib Riak

Kibana Dashboard

VisualisationStorage

Stream Sets

I/P in multiple formats Text, logs and json from various storage source.

Spark application process data and store to elastic search or Cassandra or Riak storage for visualization else it stores in HDFS

Machine learning algorithm for data science application

ZoomdataData

Discovery

D3Web

Application

HDFS

StreamSets Data Collector runs as an application in Spark Streaming to pull data from origin to spark

CSV, Tab delimited etc.

LOG file

JSON

TEXT

Sample Architecture for Data Streaming

Kafka

Cassandra

Elastic Search

Spark

Spark Mllib

Riak

Kibana Dashboard

VisualisationStorage

Stream Sets

Streaming network from different sources

Kafka is used for collecting streaming data and data is consumed through consumer API by Streamset for further processing.

Spark application process data and store to elastic search or Cassandra or Riak storage.

Machine learning algorithm for data science application

ZoomdataData

Discovery

D3Web

Application

Use Case 1 - Shipped Analytics

Collect log metric from cluster to analyze and drive Alert/Recommend engine

• Alert Engine - produces alert messages on a basis of some conditions.

• Trend Engine - produces trend messages related to data aggregation. • Policy Engine – derives from Alert and Trend Engines produces policy

messages which contain recommendations.

Use Case 1 - Shipped Analytics Architecture

Central Cluster

Probe

Probe

Probe

DataCollector

DataCollector

DataCollector

node

node

node

node

node

Use Case 1 - Shipped Analytics Data Flow

• Identify the top technology trends by analyzing public data and open source projects

• Use machine learning to process a wide range of public data available on the world wide web and identify high potential emerging technologies

• Publish results to a web-based dashboards and refresh results regularly

Use case 2 - Emerging Top technology using Public data

Use Case 2 - Analysis Through Public data

Use Case 2 - Dataflow

APIs

RSS feeds

Scraping

Numeric Network Data

Text data (articles, blogs)

Staging tables

Interactive D3 Dashboards

Websites

Data Sources Data Extractors

Data Storage Data Processing Machine Learning Visualization

Below we used the framework to execute the project in CIS Data Platform

Lambda Reference Architecture

Monitoring / Analytics Cluster (local, Texas-3)

Global Monitoring / Analytics Cluster (global, Texas-1)

Monitoring / Analytics Cluster (local, Ams. -1 )

Monitoring / Analytics Cluster (local, Lon.-1)

Local components and deployment is the same as global, just smaller

Real-time and batch processing (Lambda), anomaly detection, visualization

SSL

Kafka

SSL

SSL

MQTT

MANTL Data Platform in Practice: putting it all together

Working on advanced enabling technology – Mesos, K8S, Orchestration

Working on developing individual components – dev & co-dev: Zoomdata, Riak, Streamsets, etc.

Putting together reference architectures and real solutions to test and further develop the technology

Provide innovation and advanced services to customers

Platform to develop and deliver Microservices and Data applications

Next stepsContinue partnerships and co-devlopment efforts with industry leaders to deliver innovation

Continue applying new developed technology to real use cases and PoC with customers and partners

Continue working closely with A&E and Product teams on productization roadmap

Work with A&E team closely on prioritization of our R&D activities to stay closely aligned


Elasticsearch is a highly scalable open-source full-text search and analytics engine

Allows to store, search, and analyse big volumes of data quickly and in near real time

Underlying technology in application to Optimize complex search in Big data

We are working together with Elastic developing and testing their UTILIZING Mesos cluster to run Elasticsearch

Elasticsearch on Mesos Cluster

Elastic framework scheduler

Marathon framework scheduler

Chronos framework scheduler

Zookeper

Chronos ExecutorMarathon ExecutorHA Proxy node

Step 1: Mesos Cluster with Marathon & Chronos running

Step 2:Elastic framework installation on MESOS Master with a configured # of mesos slaves to be launched

Step 3: Deploys the ES executore in MESOS slaves

Step 4: ES nodes discovery and Zookeper pugin in ES nodes

Step 5 Using plugin nodes find each other and search is optimized at cluster level

Elasticsearch executor & Zookeper pugin

rahul.jain

MANTL Architecture – DatacentersControl nodes manage the cluster and resource nodes. Containers automatically register themselves into DNS so that other services can locate them.

Once WAN joining is configured, each cluster can locate services in other data centres via DNS or Consul API

Single Datacentre Multiple Datacentre

Client Client Client

RPC over DNSmask

RPC over DNSmask

LAN gossipover DNSmask

Server Server(Leader) Server

replication replication

Lead forwarding

Internet

Server Server(Leader) Server

replication replication

Lead forwarding

Datacenter 1

Datacenter 2

Remote DC forwarding

WAN gossip

TCP&UDP

Consul

➢ Service discovery➢ Client health-checking➢ Key-value store for

configurations➢ Multi-datacenter support

Mesos features

Mesos makes it easy to develop distributed systems by providing high-level building blocks.

➢ Scalability➢ Fault-tolerance and self-healing➢ Resource isolation➢ Fine Grained resource elasticity

Mesos architecture

Mesos setup for developing application

ZK

ZK

ZK

Zookeeper quorum

JN

JN

JN

Shared edits

DataNode

DataNode

ActiveNameNode

ZookeeperFailover Controller

ActiveNameNode

ZookeeperFailover Controller

DataNode

Heartbeat Heartbeat

Write Read

Active NN state monitoring

Standby NN state monitoring

Monitor and maintain active lock

Monitor and try to take active

lock➢ Used to store and distribute data accross a cluster

➢ Is a base for batch analytic processing

➢ Is highly available and fault tolerant

➢ Automatically scaled and self-healing with Mesos

HDFS framework

* https://github.com/datastax/spark-cassandra-connector

Mesos Framework for Spark and Cassandra

* Smart broker.id assignment* Preservation of broker placement* Rolling restarts* Easy cluster scale-up

Mesos framework for Kafka

Fault tolerant job scheduler handles dependencies and ISO8601 based

schedules Flexible Job Scheduling Supports arbitrarily long dependency chain supports the definition of jobs triggered by

the completion of other jobs

Mesos framework for Chronos

How MANTL Data Platform for business application• Cisco Data Platform can be used to build custom applications or

service for various analysis and Data analytics initiative.• Companies can streamline Data ingestion, process,

manipulate , analyse and visualize data all in single Infrastructure

Yali Load Testing FrameworkYali

Elasticsearch

Kafka

Cassandra

HDFS

Plugins

Kafka

Cassandra

HDFS

Storage

Elasticsearch

Generate data to load test storage

Elasticsearch Plugin Testing Results

Job Host config

Elasticsearch config

Execution threads

Batches Records/batch/thread

Average response from ES, s

Records/s Record size, b

Records generated * 10^6

Execution time, min

Win7, 4 cpu, 16 ram

Cluster: CentOs 6.7, Elasticsearch 2.1.1, VPN network, 2 master(4 cpu, 16 ram), 15 worker nodes(8 cpu, 32 ram)

12 60 50000 78 6804 280 36 84

Local: CentOs 6.4, Elasticsearch 2.1.1, VMware virtual network, single node (2 Core cpu, 8 Gb ram)

4 60 10000 1,6 14768 280 2,4 2,5

Records

mantl data platform, microservices and bigdata services

Technology