cloud computing, docker, mesos, dcos, container, big data, paas

©2016XavientInformationSystems

CloudComputingandDCOS


Agenda

ü IaaS

ü PaaS

ü TechnologyOverview


IaaS


InfrastructureasaService isaformofcloudcomputingthatprovidesvirtualizedcomputingresources.

PublicCloud

• AWS

• Azure

• GCE

PrivateCloud

• VMWare

• OpenStack

• ApacheCloudStack

IaaS


PaaS/DCOS


• Platformasaservice (PaaS)isacategoryofcloudcomputingservicesthatprovidesaplatformallowingcustomerstodevelop,run,andmanageapplicationswithoutthecomplexityofbuildingandmaintainingtheinfrastructuretypicallyassociatedwithdevelopingandlaunchinganapp

• DCOS isanewkindofoperatingsystemthatorganizesallofyourmachines,VMs,andcloudinstancesintoasinglepoolofintelligentlyanddynamicallysharedresources.ItrunsontopofandenhancesanymodernversionofLinux

Paas andDCOS


NotoStaticPartitioning

• Underutilize• Doesnotscale• Hardtodealwithfailures


MesosEcosystem


Architecture


Clusterconfigurationexample


SoftwareonMesos

Aurora isaserviceschedulerthatrunsontopofMesos,enablingyoutorunlong-

runningservicesthattakeadvantageofMesos'scalability,fault-tolerance,andresource

isolation.

Marathon isaprivatePaaSbuiltonMesos.Itautomaticallyhandleshardwareor

softwarefailuresandensuresthatanappis“alwayson”.

Hadoop RunningHadooponMesosdistributesMapReducejobsefficientlyacrossan

entirecluster.

Spark isafastandgeneral-purposeclustercomputingsystemwhichmakesparalleljobs

easytowrite.

Chronos isadistributedjobschedulerthatsupportscomplexjobtopologies.Itcanbe

usedasamorefault-tolerantreplacementforCron.Jenkins isacontinuousintegrationserver.Themesos-jenkinspluginallowsitto

dynamicallylaunchworkersonaMesosclusterdependingontheworkload.

Cassandra isaperformantandhighlyavailabledistributeddatabase.Linearscalability

andprovenfault-toleranceoncommodityhardwareorcloudinfrastructuremakeitthe

perfectplatformformission-criticaldata.

ElasticSearch isadistributedsearchengine.Mesosmakesiteasytorunandscale.


MesosArchitectureinHAmode

Behindthescenes,ApacheMesosusesApacheZooKeeper,adistributedconfigurationstoretopersistitsstate.The"state"ofMesosincludeseverythingfromconfigurationtothestatusandloadofindividualMesosnodes,aswellasframeworkslikeCassandra-Mesos.


PaaSonMesos


Marathonuse

• Constraintscontrolwhereappsruntoallowoptimizingforeitherfaulttolerance(byspreadingatask

outonmultiplenodes)orlocality(byrunningallofanapplicationstasksonthesamenode).Thereare

theredifferentpartsa)fieldb)operatorc)optional

"constraints":[["rack_id","CLUSTER","rack-1"]],"constraints":[["hostname","CLUSTER","a.specific.node.com"]]

"constraints":[["rack_id","GROUP_BY"]]

Example:https://github.com/mesosphere/marathon/tree/master/examples

curl-i -H'Content-Type:application/json'-d@<filename.json>localhost:8080/v2/apps


• Reservationproblem

• DataPersistenceproblem

• Whereisthesecurity?AuthenticationandAuthorization

• Monitoring

• FrameworkRateLimiting

• Vendorsupport

• Managedservices

Let’sgetpractical


• Resourcesarereservedforrolesandrolesusedtoreserveresources• StaticReservationmesos-slave\ --master=<ip>:<port>\ --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096”“8CPUsand4096MBofRAMreservedforadsonthisslave.”

• DynamicreservationinMesos>=0.23.0• CreatethroughFrameworkandOperatorAPI

Framework Operator

Reservation-mechanisms to reserve resourcesinspecificslaves


• Amechanismtocreateapersistentvolumefromdiskresources• PersistentvolumesenablestatefulservicessuchasHDFS andCassandra tostoretheirdata

withinMesos• Authorizationismusttocreatepersistentvolumes• Persistentvolumescanonlybecreatedfrom reserveddiskresources,whetheritbe

staticallyreservedordynamicallyreserved- Important• CreatethroughFrameworkandOperatorAPI(SameasDynamicReservation)

https://mesosphere.github.io/marathon/docs/persistent-volumes.html

https://mesosphere.github.io/marathon/docs/external-volumes.htmlAnexternalstorageserviceenablesyourappstobemorefault-tolerant.Ifahostfails,Marathonreschedulesyourapponanotherhost,alongwithitsassociateddata,withoutuserintervention.https://github.com/emccode/rexray

PersistentVolumes


Persistentvolume


• Authenticationisdisabledbydefault.Whenauthenticationisenabled,operatorscanconfigureMesostoeitherusethedefaultauthenticationmoduleortousea customauthenticationmodule.

• AuthenticationpermitsonlytrustedentitiestointeractwithaMesoscluster• Authenticationcanbeusedatframework,agentandoperatorlevel• Configureauthenticationduringthemesosmasterandslavestart• Authorization,localandcustom• Configuretheactionsthatcertainprincipalsareallowedtoperform

http://mesos.apache.org/documentation/latest/authentication/http://mesos.apache.org/documentation/latest/authorization/

*ThedefaultMesosauthenticationmoduleusesthe CyrusSASLlibrary.SASLisaflexibleframeworkthatallowstwoendpointstoauthenticatewitheachotherusingavarietyofmethods.Bydefault,MesosusesCRAM-MD5authentication.

Security


What’sgoingon?


MesosandContainers

"If a Docker application is a Lego brick, Kubernetes would be like a kit for building the Millennium Falcon and the Mesos cluster would be like a whole Star Wars universe made of Legos." ~ Solomon


Mesos+Docker+Kubernetes

MesosclusterscanbedeployedonnearlyeveryIaaScloudproviderinfrastructureorinyourown

physicaldatacenter.KubernetesonMesosrunson-topofthatandthereforeallowsyoutoeasilymoveKubernetesworkloadsfromoneoftheseenvironmentstotheother.http://kubernetes.io/docs/getting-started-guides/mesos/


CassandraandMesos/Marathon

https://mesosphere.github.io/cassandra-mesos/

https://github.com/mesosphere/dcos-cassandra-service

https://docs.mesosphere.com/1.7/usage/services/cassandra/install-and-customize/


https://mesosphere.com/blog/2015/11/05/jenkins-dcos-mesos/

Jenkins


• Auroravs.Marathonhttp://code.hootsuite.com/marathon-vs-aurora-a-high-level-comparison/

• MesosandKuberneteshttp://stackoverflow.com/questions/26705201/whats-the-difference-between-apaches-mesos-and-googles-kubernetes

• JenkinsandMesoshttp://www.slideshare.net/carlossg/ci-and-cd-at-scale-scaling-jenkins-with-docker-and-apache-mesos

• BloombergandMonitoringhttp://schd.ws/hosted_files/mesosconna2016/98/SeamlessMonitoringForMesosClusters.pdf

• UberandCassandrahttp://schd.ws/hosted_files/mesosconna2016/60/mesoscon-uber.pdf• Mesoscon 2016

https://www.youtube.com/playlist?list=PLGeM09tlguZQVL7ZsfNMffX9h1rGNVqnC• Jenkinsalternatesolutionhttps://github.com/HubSpot/Blazar• Mesosoversubscriptionhttp://mesos.apache.org/documentation/latest/oversubscription/• https://support.mesosphere.com/hc/en-us/categories/200383389-Technical

Read


CloudFoundry

CloudFoundryisanopen-sourceplatformasaservice(PaaS)thatprovidesyouwitha

choiceofclouds,developerframeworks,andapplicationservices


Architecture- PCF


RedHat OpenShift (onRadar)


Q&A


Xavient’s Capabilities: BigDataandAnalytics


Agenda

ü XavientIntroduction

• CompanySnapshot• GlobalPresence• OurDifferentiators

ü XavientBigData&BICapabilities

ü CaseStudies


CompanySnapshot

WhoWeAre

• Founded in US in 2002 | Global Scale |Exponential Growth

• Fortune 200 clientele | Partner of choicefor4out of Top5CSPs inUS

• Comprehensive Technology Coverage &PersonalizedService

WhatWeDo

• Enable Enterprise Class Solutions onWeb, Mobility, Open Source, Cloud,

BigData

• Focus on Telecom, Retail, Healthcare&Banking

• OptimizeBAUSDLC Practices

HowWeDeliver

• Customer First Approach - Long termongoing engagements are testimony toour success

• Talent Farming – Thought leadership |Bootcamps | Embedded Factory

• Maturity & Agility – TechnologyInnovation | Reusable Best Practices |Early Adopters


GlobalPresence

Tampa, FL

Atlanta,GA

Herndon,VA

Sterling, VA

Bethpage, NY

Frisco,TX

Englewood, CO

Denver, CO

St.Louis,MO

Bellevue,WA

SimiValley,CA

(Headquarters)

Noida

Bengaluru

OnshoreDeliveryCenter/

ClientLocation

OffshoreDeliveryCenter

Mississauga, Canada


Xavient – EnablingDigitalTransformation

Mobility

OpenSource

BigData

IoT

UI/UX

• Dataingestionandanalysis fromsensors,edgesdevicesandsmartphoneapps.

• Transitionheavyweight,costlyandcompositesystemstoOpenSourceStackbasedEnterprise classsolutions

• CustomUI/UXbasedonthebusinessneeds.

• Deliveredinnovativesolutionshandlinglargevolumeofdata

• Mobilitysolutionaccelerators&frameworks toreducecosts&go-to-marketfaster


OurDifferentiators

People

Pricing

Processes

Partnerships

Productivity

AmongstthebestglobalITtalent

Mature&Agileprocesses

In-houseR&Dcenter,TSG® forinnovationandautomation

Leveragingpartnershipsandstrategicalliances

CustomercentricGlobal Delivery

Highly-skilledconsultants &SMEs Lowattrition

ExcelinDevOpsmethodologies Extrabandwidthof10-15%

Bigenoughtoscale,smallenough toalign

LeadwithSolutionsandValue

Turningclientchallengesintosuccessstories

Flexiblepricingmodelsforcostefficiency


Xavient’sBigDataandBICapabilities


Xavient’sBigDataandBIPracticeDifferentiators

Enterprise readysolutionwithcomplete

securityinplace.

100+ExpertsworkingonBigData/BI

projects.

Comprehensiveexperienceacrossthe

datastack

Readytouseaccelerators

StrategicallianceswithleadingvendorsDiversifiedIndustryExposure


Xavientendtoendapproach

Application

Security

Compute

ResourceManagement

Storage


ServicesthroughEVERYphaseofadoption

XavientisfocusedtoprovidelongtermpartnershipandsupportinallphasesofyourBigDataprojects

Services

Architecture

Development

Implementation

Operation

Managed

Services


Xavient’sBigDataTeam

DeveloperandDataScientist

10+yearsindustryExperience3yearsBigDataexperience

DataArchitect/Engineer


Sr.EngagementManager

8+yearsindustryExperience2+yearsBigDataexperience

Sr.Consultant


• Develop dataingestionflowinbatchandrealtime

• Designanddevelopstatisticalproceduresandalgorithmsarounddatasources

• Recommend andbuildmodelsforvariousdatastudies,datadiscoveryandhypothesis

• Work withdeveloperstointegrateandpre-process dataforinputsintomodels

• Datacleanupandtransformation aspreparationforanalysis

• Focus areasinclude:statistics,appliedmath,machinelearning,modeldesignanddevelopment

• Configure andsetupvariousdevelopmentanddatasciencetools.

• Platformarchitecture analysisanddesign,including:datacenterlayout,hardware andnetworkconsiderations

• Securitymodeldesignandimplementationforauthentication,authorization,administrationandaudit

• Definerecommendedprocesses fordataingestion,management,transformation andegress.

• Perform installationforenterprise-scaleclusterenvironments.

• Resource managementatclusterandinfrastructurelevel.

• Facilitatetheselection ofappropriateBigDatatechnologyandtools

• Conduct technicalworkshops• Definetheoverallarchitecture forthesolutions/business applicationsandpreparetheenvironment

• Initialday-to-dayclusteroperations

• Overallresponsibilityforprojectsuccess

• Primarymanagement contact forthecustomer.

• CoordinatealltheProfessionalServicesresources

• Ensuretheprojectisdeliveredaccordingtoscheduleandbudget

• Reportstatus andmanageescalations.

• Leadtheimplementationofbusinessusecase(s).

• Provideexpertiseandguidanceduringtheprojectlifecycle.

• Providedesignanddevelopmentexpertiseforlarge-scale,clustereddataprocessing systems

• Developingest,transformation andegresscapabilitiesleveraginglatesttools suchasKafka, Storm, Pig,Hive,etc.

• Resolvetechnicalissuesintheenvironment

• Assist withthepreparationoftechnicaldeliverables,andreviewanddemonstrate thesystem andapplications

• Providetechnical supporteitheronsiteorremote.


CaseStudies


BigDataCase1:EnterpriseSocialNetworkingand

CollaborationPlatform

• Real timedata streaming• Search capability• Front endUI• Build recommendationmodel based onuser search

Xavient’sSolutionXavient’sSolution

• End to end design and implementation of enterpriseready solution/app to provide social networkingplatform

• Mobile app solution• Chat, Message exchange capabilities by leveraging

NoSQL databases• Resourcemanagement at the enterpriselevelTechnologies: Kafka, Storm, HDFS, Hive, HBase, ElasticSearch, Java, ApacheMesos/Mesosphere, ApacheMyraid

Xavient’sSolution

• Real time reports& analyticswith granular data• Health checkandmonitoringofkeyareas• Informed andquickdecisions• Both online& offlinemode access• Multiple Currency selection (USD& local currency)

KeyBenefits

BusinessRequirement

AnEnterpriseSocialNetworkingandCollaborationPlatform


EnterpriseBasedDistributedArchitecture

(MultipleWeb&MultipleApp)Load

Balancer

WEB-n

ENCACHE

WEB-1

ENCACHE

APP

ENCACHE

ELASTICSEARCH-1

ELASTICSEARCH-n

………………

…………

…………

PostgreSQL-1

UTILITYSERVER

DOCSERVER

CHAT REDISSERVER…………

PostgreSQL-n

…………

MongoDB-1

MongoDB-n

…………


BigDataCase2:Implementing aTrueEnterpriseDataLake

BusinessRequirement

• Store all the data at oneplace• Eliminate datasilos• Build COE

Xavient’sSolutionXavient’sSolution

• Developedoverallroadmapanddesignforenterprise‘datalake.’

• POCandHadoopknowledgesharingsessions• Securitysetupforstrongauthenticationandauthorization

• DataingestionfromRDBMStoHDFS/Hive.H• DefinedplanandhelpedimplementaCoE

Technologies used: HDP, Hive, Sqoop, AWS,Oozie

Xavient’sSolution

• Nomore silosof data• Fast response time delivered for long runningqueries

• Cost per TB dropped• InternalHadoop asa Service cloud• Store data for longer period of time• New technology stack increases employeeretention

KeyBenefits

AboutClientClient is a leading Satellite TVService Provider inUS


• Operationaldashboards• Customerscorecards• CDRanalysis• ProactiveMaintenance• Infrastructureinvestment• Bandwidthallocation• Productdevelopment

EDW

MPP

CRM

ERP

DATASOURCES

DATAREPOS ANALYSIS

BillingData

SubscriberData

ProductCatalog

NetworkData

Clickstream

OnlineChat

SensorData

ServerLogs

CallDetailRecords

MerchantListing

DMP

SocialMedia

Ingest

SqoopFlume

HDFSPutWEBHDFS

Hadoop

Cluster

Batch

MapReduce

Script

PigSQL

HiveOnline

HbaseAccumulo

Real-Time

StormIn-Memory

SparkOthers

MetadataManagementHCatalog

MultitenantProcessing:YARN

HadoopOperatingSystem

LinearScale

Compute&HDFSStorage

Governance

tag,filter&process

Operations(Ambari)

Security(Knox,Kerberos,ApacheRanger)

1

N

. . . . . . ..

. .

.

. .

. . . . . . . . . .

. . .. . . .. . .

BigDataCase2:EnterpriseDataLake(Contd..)


BigDataCase3:Dropcallandnetworkchangemonitoring in

RealTime

.

BusinessRequirement

• Xavient’sSolutionXavient’sSolution

Xavient’sSolutionKeyBenefits

• StorealltheCDRdataatoneplace“DataLake”• Realtimemonitoringofdroppedcalls• Realtimemonitoringofnetworkchanges

• Dataingestionflowwasbuiltusingflume,kafka andstormtostoredatainHDFS,HiveandHBaseincombinationwithApachePhoenix

• Solr andBananadeliveredthedashboardcapabilities• ApachePhoenixwastousedtorunqueriesagainstHBasetofinddropcallsandnetworkchange

• Longerdataretention• Fasterresponsetimefromthelargerdataset• CostreducedperTBbecauseofHadoopstack• Effectiveandfastalertingduringtheoutages• Abilitytorunpredictiveanalysisbasedonrealtimealerting

.AboutClientClient is a leading wirelessnetworkoperator in the US


DataFlowandArchitecture

HDFS

InputFeed

Hive

Flume,Kafka,Storm

SearchUI

QueryUI

OutputFeed

SearchIndex


CaseStudies:BusinessIntelligence


BICase1:Executive DashboardForStrategicInsights

BusinessChallengesBusinessRequirement

Xavient’sSolution Xavient’sSolutionXavient’sSolution KeyBenefits

• Multi-channeldashboard• RealtimeanalyticsonCross-functionaldatawithvisualrepresentation• QuickerdecisionmakingforSeniorExecutives

• Crossplatformdashboardreportingapplication• Configurablebackgroundsyncforofflinedataaccess• CustomizedKPIsprovidingbird’seyeview• Dynamicreportingfrommultipledigitalmarketingchannelsviz.revenue,billings,costsandnumberofcampaigns

Technologies used: Informatica, Oracle, HTML5, Jquery,Java, Objective C

• Realtimereports&analyticswithgranulardata• Healthcheckandmonitoringofkeyareas• Informedandquickdecisions• Bothonline&offlinemodeaccess• MultipleCurrencyselection(USD&localcurrency)

AboutClientThe client is a Digital Marketing& Advertising ServiceProvider


BICase2:InventoryandMerchandisingApp

BusinessChallengesBusinessRequirement

Xavient’sSolutionXavient’sSolution KeyBenefits

• Cross-Platformmobileapplicationthatgivesat-a-glanceviewofallSales,InventoryandGrossmarginforallproductsinastorefromanywhere

• Performancedashboardforassociatestomeasure,monitorandactonthesellablearticles

• Extendplatformtocatertocorporatevendorsforvariousproducts

• InterfacethemobileappwithSAPformetricsandreporting

• developedamobileapplicationto:• Browseproductcatalogue• Checkinventoryreportsandstatus• Tracksales(actualandplanned)• Readcustomerreviews• Findastore• Viewcurrentpromotionsofretailer

Technologiesused:PHP/JavaScript/MySQLObjectiveC(foriOSapp)Java(forAndroidapp)

• ConvenientandBettercustomerservice• Simpleuserfriendlydesignlayout• Customizedplatforms• Competitivepricing• BetterSalesandStockmanagement• SpeedscriptingtoincreaseTimetoMarket• Morevalueaddedservices

AboutClientOne of the leading retail chainsthe in US.


ThankYou