couchbase live europe 2015: introducing multi dimensional scale: a new architecture for workload...
TRANSCRIPT
Multi-Dimensional Scaling
A new Architecture for Workload Optimization
in Couchbase Server 4.0
Cihan BiyikogluDir. Product Management
1
©2015 Couchbase Inc. 2
Agenda
Brief History of Scaling Scaling up and out
NoSQL Workloads and Scalability Model Core Data operations, Indexing and Query
Introducing Multi-Dimensional Scaling Services and Independent Scaling
Demo Q&A
©2015 Couchbase Inc. 3
Disclaimer
Couchbase Server 4.0 is still in development. Detail presented in this presentation may change based on customer feedback and other factors by the time the final version of the product is released.
©2015 Couchbase Inc. 5
Question
Few million people are looking for a setup to efficiently live and interact. What is the most efficient way to build this infra?A) Build one giant high-rise?B) Build some mid-rises?C) Build many single-family homes
©2015 Couchbase Inc. 6
Scaling Up
Build one big high-rise Vertical Scaling
Cluster processors – hyper-threading to cores Locally partition workload among processors Communicate over memory
Great for fast processing but limited in scalability and elasticity
©2015 Couchbase Inc. 7
Scaling out
Build a large community of single-family houses Horizontal Scaling
Cluster commodity HW Partition workload among nodes Communicate over network
Great for scaling and elasticity but slower communication
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 10
NoSQL Workloads
One Database Many Workloads Core Data Processing: GETs & SETs for given key Indexing: Index maintenance and lookups Querying: Combine index and data with complex just-in-
time data re-shaping, ordering, grouping, aggregations and more
Varying resource requirements - CPU, RAM, I/O, Network
Varying methods to optimize latency & throughput for each
10
©2015 Couchbase Inc. 11
Scalability Model TodayHomogenous Scaling
Each node get a slice of the workload Simple to do…
But... Workloads compete and interfere with each other Cant fine tune each workload
- Core Data operation are partition-able so great with wider fan-out- Indexing and Query not always partition-able so worse with wider fan-out
Index Service
Couchbase Cluster
Query Service
Data Service
node1 node8
©2015 Couchbase Inc. 13
Modern Architecture
What is Multi-Dimensional Scaling? MDS is the architecture that enables independent scaling of data, query and indexing workloads.
Index Service
Couchbase Cluster
Query Service
Data Service
node1 node8
©2015 Couchbase Inc. 14
Modern Architecture
Isolated Service for minimized interference Independent “zones” for Query, Index and Data Services
Minimize indexing and query overhead on core KV operations
Index ServiceGlobal
Secondary Indexes
Couchbase Cluster
Query Service
Data ServiceViews and Geo Views
node1 node8
©2015 Couchbase Inc. 15
Modern Architecture Independent Scalability for Best Computational Capacity per
Service
Heavier indexing (index more fields) : scale up index service nodesMore RAM for query processing: scale up query service nodes
Couchbase Cluster
node1 node8 node9
Data Service
Index ServiceQuery
Service
©2014 Couchbase Inc.
Full Cluster Architecture
17
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
©2015 Couchbase Inc. 18
Data Service
Projector & Router
New in Couchbase Server 4.0
Data Service: GET & SET operations and Map/Reduce Views Arch mostly the same as 3.0
Index Service: new Global Secondary Indexes (GSI) Query Service: new Query processor (N1QL)
Query Service
Query Processor
cbq-engine
Index Service
SupervisorIndex maintenance &
Scan coordinator
Index#2
Index#1
Query Processorcbq-engine
Bucket#1
Bucket#2
DCP StreamIndex#4Index#3
...
Bucket#2
Bucket#1
©2015 Couchbase Inc. 21
Recap
MSD enables unprecedented control on scalability with Couchbase Server Separate out competing workloads to independent services Independently scale each service “zone” within the cluster
Couchbase Server with MDS maximizes scalability and performance Improves scale and performance to degrees not possible
with other NoSQL or Big Data engines on premise or in the cloud
Improved price/performance and squeezes more performance and throughput for mission critical systems
©2015 Couchbase Inc. 22
Couchbase Server 4.0
Couchbase Server 4.0
Download the Developer Preview in a few weeks…
Couchbase.com/