database containerization platform checklist

ROBINSYSTEMS

DatabaseContainerizationPlatformChecklist

AdeeshFulayDirectorofProducts

@AdeeshFadeesh@robinsystems.com

Agenda

› DataManagementChallenges› WhyContainers?› TheFeatureChecklist› RobinSolutionandDemos› Summary

DataManagementLandscapeandChallenges

Traditional ModernDataPlatforms

§ CostContainmento Highsoftwarelicense&supportcosts

§ DevOpsAgilityo Quickdatabaseclone,snapshotand

time-travel

§ PerformancePredictabilityo Unpredictableperformancein

consolidatedenvironments

§ OperationalComplexityo Painfuldistributedapplication

deployment,scalingandongoingmanagement

§ HighHardwareCAPEXo Clustersprawl,DataDuplication

§ EnterpriseGradeRobustnesso Backup,restore,failover,security

ExistingInfrastructureChoicesareInadequate 4

Bare-Metal VMs Purpose-BuiltAppliances

Containers

Inefficient

Complex

PerformanceOverhead

VM,DBsoftwareSprawl

VeryExpensive

Applicationspecific

Containers– WHAT?

› NextGenerationofVirtualizationtechnology› OS-based,anddeliversbaremetalperformance› OperationallyverysimilartoVirtualMachines› DedicatedIPaddress,filesystems,separateuserspace,etc.

› Agile› Containerscanbelaunchedinseconds

› Betterhardwareconsolidation› 10xormorehigherconsolidationbenefits

› EliminateOSduplicationandapplicationsoftwaresprawl

0

20000

40000

60000

80000

100000

120000

RandomReads RandomWrites RandomMixed

IOPS

BareMetal Container VM

Source:IBMResearchhttp://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf

Containers– 2xFasterthanVMs

PoorDockerAdoptionforDatabases

Dockerprimarilydesignedforstatelessapplications› Ignoresmostimportantcomponent– storage

OrchestrationtoolstreatDatabasesasstatelessaswell

https://goto.docker.com/Docker-Survey-2016.html

CattlevsPets!

› Petsaregivennameslikepussinboots.acme.com› Theyareunique,lovinglyhandraisedandcaredfor› Whentheygetill,younursethembacktohealth› Examples:Databases,bigdata,andotherstateful apps

› Cattlearegivennumberslikeinst0042.acme.com› Theyarealmostidenticaltoothercattle› Whentheygetill,yougetanotherone› Examples:Allstatelessapplications

FutureapplicationarchitecturesshoulduseCattle,butPetswithstrongconfigurationmanagementareviableandstillneeded

*OriginalslidefromCern regardingdifferentservicemodels

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

OriginalBlog: http://www.robinsystems.com/blog/stateful-database-containerization-platform-checklist/

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

1.MultipleContainerTypes(andmaybeevenVMs)

• TheNgnix• Node.js• Postgres

• Redis• MariaDB• Node.js

• ElasticSearch• Wordpress• Node.js

SYSTEMCONTAINERS

• Eachcontainerrunsanentireservicestack(multipleapplicationspercontainer)

• MeanttobeusedaslightweightVM

• Filesystemneutral

• Examples– LXC,OpenVZ,SolarisZones

• Supportsin-placepatch/upgrade&SSHaccess

GreatforTraditionalApplications

Ngnix

Node.js

Postgres

Redis

MariaDB

Node.js

ElasticSearch

Wordpress

Ngnix

APPLICATIONCONTAINERS

• Eachcontainerrunsasingleapplication(singleconcernpercontainerphilosophy)

• Meanttobeusedasapplicationpackage

• Containersuselayeredfilesystems(Overlayfs,AUFS,etc)

• Examples– Docker,Rocket

• Patch/Upgradeentailsreplacingcontainerimage

GreatforModernApplications

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

2.Application-awareInfrastructureCURRENT

DBAburdenedwithmanydecisions

DESIREDSimplifiedApplicationManagement,

InvisibleInfrastructure

Fluid,Elastic,HighPerformanceInfrastructurePlatform

Deploy

Scale

SnapshotClone

Restore

Capacityplanning

Storage

NetworkMemory CloudPerformance

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

3.DataPersistenceManagement

› Application=multiplecontainers

› Storageallocatedpercontainer,andnotperhostorVM

› MultipleVolumes/container› Blockstorage:formattoxfs,ext4,orraw› Eachcontainercanbindtomultiplevolumes

– HDD,SSD,NVMe

› Rapidcreate/destroy› Millisecondsresponsetimetocreate/delete

tokeepupw/containers

› Highervolume› 10-100xmorecontainersbeingcreatedthan

VMs

› Diskrepair,replication,compression,…

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

4.CloningProdtoTest/Dev

PROD Test Master

Downstream environments Dev & Test

Storage Usage for one application

› FullClonesofyourproductiondatabase

› Timeconsuming

› ResultsinDataSprawl

› CreateTestmastertoavoidperformancepenalty

› Needawaytocreateasmanyenvironmentsbutnotpaythestoragepenalty?

4.Snapshot,Clone,andRestore 17

ProdCluster

FullCluster,DistributedApplicationSnapshotsOS+DBsoftware+DBconfiguration+data

InstantClusterClone

ProdScaleTestEnv inMinutes

Need to test a change? Corruption or data loss?

2xstorageLongdatacopyManualre-configuration

ErasecommitlogsErasedatafilesRestorebackupRestartcluster

Lotmorethanjuststoragesnapshots

ProdCluster ProdCluster ProdCluster ProdCluster ProdCluster

VirtualTestCluster

InstantRestore

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

5.DatabaseWorkloadTypes

DataWarehousing(OLAP,Reporting,PredictiveAnalytics)

OLTP(Highlytransactionalsystem)

› LowLatency› Impactsuserexperience› SSDssuperiortoHDDs

› HighBandwidthi.e.Throughput› (Throughput=IOPSxBlocksize)› Dealwithlargevolumesofdata

› Multipledatabaseshittingthesamedisks› SustainedhighIOPS&PredictablelowlatencyConsolidated

=

5.NeedforQualityofService(QoS)

› NoisyNeighborproblem› Containersgiveprocessisolationbutnotperformanceisolation

› Predictableperformance› GuaranteedIOPSforapplication/container

› Scaleup/down› Traditionaldatabasesscaleup;moderndistributeddatabasesscaleout› Buthowdowecatertotransientspikes?› Containersonlyallowforcpu andmemory,butwhataboutIOPS?

Compute

Disk1 Disk2 DiskN

Consolidationw/containers

Compute

Disk1 Disk2 DiskN

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

6.As-IsFailover

ContainerFailover,Migration

APPLICATIONCLUSTERStatelessfailoverpolicieswillnotNOTworkfordatabases✔Containerrescheduledtoadifferenthostx ContainergetsanewIPx Losesdataasthevolumeisnotremountedonthenewhost

KeyUseCases:› Automaticfailover› Manualnodeevacuationforplannedmaintenance(e.g.OSorhardwareupgrades)› Clusterrebalance

22

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

7.NetworkAutomation

› Dockernetworkingisearlystageanddiverse(bridge,host,overlay,n/wdriver)› Applicationshavetochoosethebestoption› Docker,bydefault,usesLinuxEthernetBridgesforbasicrouting.Thishasalotofperformanceissues.SamewithOverlaynetwork.› Usethehostnetworkstackinstead(-net=host)

Databasesnetworkrequirements:› Spanmultiplehostsi.e.clustered› Eachcontainershouldhaveuniquename&IPaddresses› PurposefulDBnames(appdb.us.acme.com)registeredinDNS› Exposewellknownports(MySQL:3306,Oracle:1521)› Minimalnetworkperformanceimpact

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

8.Multi-Tenancy&Isolation

Tenant1 Tenant2

Tenant2

Tenant1

HardIsolation

SoftIsolation

› Sharednothing;eachtenantgetsdedicatedcompute

› DedicateddiskIOcapacity

› Addhardwareforextracapacity

› Poorconsolidationduetosilos

› Sharedcompute,multi-tenancyprovidedbycontainers

› Shareddisk,butdedicatedIOcapacity

› Containersresizedforextracapacity

› Higherconsolidation

TheDatabaseContainerizationPlatformChecklist

MultipleContainerFormats

Application-aware

infrastructure

Data Persistence(volume)

management

Snapshot,clone,restore

QualityofService

As-isfailover Networkautomation

Multi-tenancy&Isolation

Easeofsetupand

management

GuidingPrinciple:Dowhat’srightfortheapplication/database!

9.EaseofSetupandManagement

7CLUSTERMANAGER(mesos,kubernates)

8SCHEDULER

(marathon,cronos)

2STORAGEcapacitytier

(ceph,gluster,..)

4NETWORKING

(openvswitch,socketplane)

10MONITORING(graphite,sensu)

5CONFIGURATION(zookeeper,etcd)

9UI

(custom)

1SERVERHARDWARE(supermicro,dell,…)

3STORAGE

performancetier(SSD,Flash,NVMe)

11TROUBLESHOOTING

(splunk,logstash)

6CONTAINERS(docker,LXC)

1. Whensomethingbreaks,hardtoidentifytherootcause

2. Ifperformancedegrades,hardtoidentifyhotspots

3. Extendingtheplatform requiresconfiguring multiplecomponents

4. Keepingitoperationalrequiresknowledgeofallcomponents

11ComponentAssembly

WhatisRobin? 29

Industry’sFirst

APPLICATION-DEFINEDDATACENTERSoftware

Container-based,Software-definedInfrastructureforData-CentricApplicationsContainers+Storage+Networking+ApplicationOrchestration

SimplifyApplicationManagement

GuaranteePredictablePerformance

MaximizeCapacityUtilization

HowRobinWorks 30

Application-definedCompute

Application-definedStorage

EXISTINGCOMMODITYHARDWARE

ROBINCONTAINER-AWARE,STORAGEANDDATAPLANE

ROBINCONTAINER-BASEDCOMPUTEPLANE

Cloud-extendCompute&Storage

ROBINFABRIC

CONTROLLER

Fluid,Elastic,HighPerformanceInfrastructurePlatform

APPLICATIONORCHESTRATIONNETWORKING DATALIFECYCLEMANAGEMENT

HowRobinWorks 31

Application-definedCompute

Application-definedStorage

EXISTINGCOMMODITYHARDWARE

ROBINCONTAINER-AWARE,STORAGEANDDATAPLANE

ROBINCONTAINER-BASEDCOMPUTEPLANE

Cloud-extendCompute&Storage

ROBINFABRIC

CONTROLLER

Fluid,Elastic,HighPerformanceInfrastructurePlatform

DramaticallyAgile,SimpleApplicationManagement

Elastic,GuaranteedQoS

ConsolidatewithBareMetalPerformance

Deploy

Scale

SnapshotClone

Restore

APPLICATIONORCHESTRATIONQoS GUARANTEEDATALIFECYCLEMANAGEMENT

PlatformforALLDatabase&BigDataApplications

Stateless Stateful DataApps EnterpriseApps

Demos

› ClusterLifecycleManagement› Initialdeploy&scaleout

› QualityofService› Dynamic&real-timecontrolofCPU,Memory,&IOPS

› DataLifecycleManagement› Snapshot,clone,andrestore

› Reducehardwarefootprintby50%ormore

› Baremetalperformance

› GuaranteedApplication-to-SpindleQoS

Business

Benefits

ReduceCAPEX

› Simplifiedapplicationdeployment

› Self-servicedatalifecyclemanagement

ImproveProductivity

› 10xfasterapplicationdeployment

› Improvedapplicationuptime

ReduceOPEX

www.robinsystems.com info@robinsystems.com

THANKYOU