scaling your data horizontally on the openstack magnetodb - trove day 2014

MagnetoDBScaling Your Data Horizontally on OpenStack

August 19, 2014

Keith NewstadtTechnical Director, Symantec Corporation@knewstadt

tesora.com 2

Management Platform

Identity Notifications Storage Data Sync Events

Norton Family

8/19/14

Infrastructure

tesora.com 3

Symantec Cloud Platform Engineering

8/19/14

Infrastructure as a Service

Infrastructure

Platform Services

Servers Networks Disks Images Ops Identity

Storage Databases Big Data Monitoring Deployment

tesora.com 4

What Kinds of Databases Do We Need?

8/19/14

Relational

NoSQLGood for relational dataIn teams’ comfort zone

Can be difficult to operate

Scales well horizontallyLearning curve

Can be difficult to operate

tesora.com 5

What Kinds of Models Do We Need?

8/19/14

Customer Operated As a

ServicePlatform provides

operational services

Platform provides the database itself

tesora.com 6

MagnetoDB – NoSQL + as a Service

8/19/14

• Open source, OpenStack standards

• Large data sets• Throughput, availability, latency• Secure multi-tenant• Integrated with Keystone

TablesCreate

Load

Index

Query Scan

Configurable Consistency

Data Types (maps, sets,

etc.)

Conditional Operations

MagnetoDB

tesora.com 7

MagnetoDB – A Simple Architecture

8/19/14

API Proxy Layer

Database Layer

MagnetoDB Driver Layer

• Simple ReST (ish) Web service API• Modeled after DynamoDB, plus…

• We’re using Cassandra• HBase also in the works

• Supports different backend databases

tesora.com 8

Some Symantec Use Cases

8/19/14

MagnetoDB

User Profile Data

Searchable Swift Object Metadata

Import from Big Data Processes

Real-time Metrics

DemoUsing MagnetoDB8/19/14

tesora.com 10

Demo – Top 100 Music Charts

8/19/14

Songs

Key Schema Secondary Index

Artist Artist

Track High

Charts

Key Schema

Year

Week-Position

http://bullfrogspond.com/

MagnetoDB

• Tables• Queries

tesora.com 11

Local VM

Environment

8/19/14

MagnetoDB

https://wiki.openstack.org/wiki/MagnetoDB/QA/Tests_on_env_with_devstack

Q&AKeith Newstadt@knewstadtkeith_newstadt@symantec.com

8/19/14