Building Scalable SQL Azure Applications Using NoSQL Paradigms

Michael Rys Principal Program ManagerSQL Server Engine Team(mrys@microsoft.com, @SQLServerMike)

Agenda

Scale-Out Application Pattern for SQL Server and SQL Azure: MySpace.com MSN Casual Games

Applications do all the work Databases provide platform support:

Reliable messaging infra structure Sharding in the data platform

Demo SQL Azure Federations Roadmap Discussion

MySpace: the Business Problem 223M users

4.4 million users concurrently

900 Terabytes of data Horizontally partitioned by

user Id 450 SQL Servers Required (eventual) data

consistency across databases E.g. show your updated

state in your friends’ profile pages

How to reflect change inmy friends’ DBs?- reliable- scalable

My DB gets updated

MySpace’s Data Consistency Problem

1-1000

1001-2000

2001-3000

3001-4000

4001-5000

5001-6000 Web Tier

Data Tier

I change my status

userId=1024

MySpace’s Solution

Propagate data changes from one DB to other DBs using reliable, async Message Service (Service Broker) Managing routes from each DB to every

other DB would be too complex Global Transactions would hinder scale

and availability And also used for

Clean-up state (e.g. on account close) Deploy business logic (stored procedures)

MySpace’s Service Dispatcher Coordination point between all SQL

Servers Centralizes route management Avoids routes explosion

Load-balanced across 30 SQL Servers Messages are sent randomly to these

Enables multicast/broadcast functionality Supports destination lists and wildcards

e.g. [DB1,DB3, DB4], DB% 18,000 ~2k msgs/sec per dispatcher

SQL Server

MySpace Architecture

1-1000

3001-4000

2001-3000

1001-2000

4001-5000

5001-6000

I change my status

userId=1024

Web Tier

Data Tier

ServiceDispatch

er

Service Broker

Service Broker

TX2

My DBgets updatedTX

1Service Broker

TX3

TX4

TX5

Many other customers using similar patterns

Online electronic stores (cannot give names )

Travel reservation systems (e.g. Choice International)

MSN Casual Games Architecture: Goals

Provide elastically scalable, high-available, agile online platform for integrated, social gaming experience Developed v1 to v3 with 7 devs in 3 months!

Support multiple gaming platforms: Windows Live Messenger Games MSN Games Bing Games

Integrate into social environments Favorite games, Friends’ highscores,

Compete against Friends, etc Social Networks: MSN, Facebook, etc

STS

Data Backend Services

MSN Casual Games Architecture: Overview

Ops

Bing Games

WLM Games

MSN Games

GameBar Host

Auth

l

l

MSN Games

Web Portal

WLM Games

Web Portal

Bing Games

Web Portal

Management Services

STS

Data Backend Services

Data Backend Services

Front Door Router

Services

Azure Data Centers

Social Networ

ks

Social Network

s

Why SQL Azure?

Faster than Table Storage Very low learning curve

Prototyped and written in less than 4 weeks by 1 dev

Testability: Easy to prepopulate

with millions of records Compatible with SQL

Server

How does it scale?• ~2 Million users at launch• ~86 Million services

requests/day • 135 Windows Azure Data

Services Hosting VMs • ca. 18K connections in

Connection Pools, this could grow with traffic

• Ca. 1200 SQL Azure requests/second spread across all partitions during peak load

• ~ 90% reads vs 10% writes (this varies per storage type)

• ~ 200 bytes of storage per user

• ~ 20% of database storage is currently used, but expect this to grow

Partitioning Strategies

• Built to scale: Functional and Data Partitioning

• 398 databases (10 Gig each)

• 100 social user information + 298 leaderboard databases• Social user

information partitioned by UserId

• Leaderboard partitioned by GameId

Data Backend Services

Front Door Router

Services

250 instances

STSSTS

DBUser …

Partitioned over 100 SQL Azure DBs

Social Service

Gamer Services

Game Ingestio

n

Social Services

Gamer Services

Game Ingestio

n

Game Catalog

Find Friends’ Profiles

DBLeaderboard

…

Partitioned over 298 SQL Azure DBs

Find Friends’ ProfilesGet my ProfilePublish feed, read feed

Last PlayedFavoritesGame PreferencesSocial Leaderboards

Disable/Enable Games from accessing services

Game binariesGame metadata

Get Friends highscores

DBUser …

Partitioned over 100 SQL Azure DBs

Write user specific game infos

250 instances

Common Data Services Features

• Fanout: Parallel calls to multiple database partitions

• Quorum: Able to tolerate a percentage of request failures during Fanout

• Retry: Retry on database requests error

Operational Games DashboardStatistics per game

Lessons Learned from both scenarios

Require high availability Be able to scale out:

Functional and Data Partitioning Architecture Provide scale-out processing:

Function shipping Fanout and Map/Reduce processing

Be able to deal with failures: Quorum Retries Eventual Consistency (similar to Read-consistent Snapshot Isolation)

Be able to quickly grow and change: Elastic scale Flexible, open schema Multi-version schema support

Move better support for these patterns into the Data Platform!

What is NoSQL? It is different things to different people But for all about App Availability, Scalability, Agility and low cost! Cost: Cheap to build, Cheap to operate Processing Paradigms

High Availability (scalable Replication, Fast Failover, DR/GeoDR, tunable latency)

Scale-out (Sharding, Map-Reduce, Elasticity) Performance (workload optimized stores, Caching, co-located compute with

partitioned state) Tunable/Eventual Consistency

Data Model Paradigms Data first: Flexible Open Schema Low-impedance mismatch between programming and data model:

Wide Tables: For “Relational” Data HyperTable, Windows Azure Tables, Hbase, SQL Server Sparse Columns

Graph stores: For “Relationship” Graphs, “Semantics” Key/document stores: For “Object” and “hierarchical data”

JSON: MongoDB, CouchBase, Riak etc. XML: Marklogic etc.

NoSQL is all about operational and developer agility at low CapEx and OpEx!

Operational Agility

You want: Availability of service (scalability) Global consistency Network Partition Tolerance

You can only get 2 of 3 (CAP Theorem) In Brave New World:

Online businesses need availability It is distributed, because it is big thus Network Partitioning is unavoidable Hence global consistency must be relaxed

Operational Agility

Performance and Elastic Scale on Demand Automate management lifecycle (or fail) Simple deployment lifecycle No DB or OS Admin telling me what to do

Developer Agility

Code First and revise quickly Application-model first (before database) Flexible open data models You don’t know exactly what you are

looking for Lower Pain of adoption and maintenance No DB or OS Admin telling me what to do

Primary Shard

Readable Replica

Readable Replica

Primary Shard

Readable Replica

Readable Replica

Primary Shard

Readable Replica

Readable Replica

OLTP Workloads(mostly touching 1 to low number of shards)

Dynamic OLAP Workloads(Scale-out queries, often using eventual consistent scale-out frameworks like Hadoop)

Generic “Web 2.0” Scale-Out Architecture

Introducing SQL Azure Federations Provides Data Partitioning/Sharding at

the Data Platform Enables applications to build elastic

scale-out applications Provides non-blocking SPLIT/DROP for

shards (MERGE to come later) Auto-connect to right shard based on

sharding keyvalue Provides SPLIT resilient query mode

SQL Azure Federation Concepts

25

Federation “Orders_Fed”

ShardedApplication

Azure DB with Federation Root

Federation Directories, Federation Users, Federation Distributions, …

Federation- Represents the data being sharded

Federation Root- Database that logically houses federations,

contains federation meta data Federation Key

- Value that determines the routing of a piece of data (defines a Federation Distribution)

Federation Member (aka Shard)- A physical container for a set of federated

tables for a specific key range and reference tables

Federated Table- Table that contains only atomic units for the

member’s key range Reference Table

- Non-sharded table Atomic Unit

- All rows with the same federation key value: always together!

Member: PK [min, 100)

Member: PK [100, 488)

Member: PK [488, max)

(Federation Key: CustomerID)

AUPK=

5

AUPK=25

AUPK=35

AUPK=105

AUPK=235

AUPK=365

AUPK=555

AUPK=254

5

AUPK=356

5

Con

nectio

n

Gate

way

Demo: Map-Reduce scale-out over SQL Azure Federations

• Sharded GamesInfo table using SQL Azure

Federations

• Use a C# library that does implement a

Map/Reduce processor on top SQL Azure

Federations

• Mapper and Reducer are specified using SQL

SQL Azure: A Not Only SQL Data PlatformSQL Azure is adding data platform support for NoSQL paradigms in the data platform:

No CapEx, Low OpEx (which should/will be even lower )

High-Availability (each DB has two replicas)

Sharding support with federations: Data platform provides online SPLIT/DROP Filtered connection to provide split resilient programming model

Flexible Data Models: XML support Sparse columns/Column sets

More to come in the future… More scale and tunable HA (to support OLTP/OLAP model) Taking Federations further (orthogonality, merge, fanout) Integration with Hadoop eco-system More data-first (data-driven columnsets, JSON)

Related Resources

Windows Gaming Experience Case Study: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000008310

Related Whitepapers: CACM: Scalable SQL: http://

cacm.acm.org/magazines/2011/6/108663-scalable-sql NoSQL and the Windows Azure Platform: http://

download.microsoft.com/download/9/E/9/9E9F240D-0EB6-472E-B4DE-6D9FCBB505DD/Windows%20Azure%20No%20SQL%20White%20Paper.pdf

SQL Federation blog: http://blogs.msdn.com/b/cbiyikoglu/archive/2011/03/03/nosql-genes-in-sql-azure-federations.aspx

Contact me: mrys@microsoft.com @SQLServerMike http://sqlblog.com/blogs/michael_rys/default.aspx

Appendix

SQL Azure Federations Details

30

CREATE FEDERATIONExisting Database

Gat

eway

CREATE FEDERATION sales (customer_id bigint RANGE)

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

31

Federation with a Single Shard

Gat

eway

Existing Database

salesDatabase root contains:• Federation root = DB level object

containing federation scheme• Federation users• Federation metadata incl. federation

map

Federation Member

Range: Min...Max

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

CREATE FEDERATION sales (customer_id bigint RANGE)

32

Introducing Two Connection Modes

• Filtered Connection– Guarantees that any queries or DML will produce the

same results independent of changes to the physical layout of the federation members

– Scoped to an “Atomic Unit”• Unfiltered Connection– Scoped to a Federation Member– Management Connection

33

Create Schema on Member: Management Connection

© 2011 Microsoft Corporation. Microsoft Materials - Confidential. All rights reserved. CITA #

MSFT101120_A

Gat

eway

Existing Database

sales

Federation Member

Range: Min...Max

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

USE FEDEDERATION sales (customer_id=0) WITH FILTERING=OFF, RESET;

Customer Order Product

federated

federatedNon-

federated

CREATE TABLE …

34

DDL

CREATE TABLE customer ( c_id bigint PRIMARY KEY, … ) FEDERATED ON (customer_id=c_id);

CREATE TABLE order ( item_num int, customer_id bigint, date_sold datetime2, …, CONSTRAINT PK_Order PRIMARY KEY (item_num, customer_id, date_sold), CONSTRAINT FK_Cust FOREIGN KEY customer_id REFERENCES customer (customer_id) ) FEDERATED ON (customer_id=customer_id);

CREATE TABLE product ( product_name varchar(100) NOT NULL, unit_price money, item_num int PRIMARY KEY, … );

35

More Detail

• Supported data types for federation key : bigint, int, GUID, and varbinary (900)– Only range partitioning

• Federation key must be part of unique index• Foreign key constraints only allowed between federated tables and from

federated table to reference table• Not all Azure programmability features supported

– Sequence, timestamp• Additional surface area restrictions

– Indexed views, drop database (members)• Schemas are allowed to diverge over time

– Furthermore, in v1, schema updates to existing members must be done in each member (where the change is needed)

• USE FEDERATION “rewires a connection”– Connection is reestablished– All existing settings and context of the connection is lost (sp_reset_connection)– Must be in a batch by itself

36

Connect to Atomic Unit: Filtered

Gat

eway

Existing Database

sales

Federation Member

Range: Min...Max

customer order product

33

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

USE FEDEDERATION sales (customer_id=3) WITH FILTERING=ON, RESET;

When using into a specific key value, SELECT will only return records from federated tables that match that value. It will still return all records from non-federated tables.Inserts and UPDATES operating outside of the value will fail.

SELECT * from customer

SELECT * from order

SELECT * from product

37

More on Connection Filtering

• Most operations behave differently in filtered vs unfiltered connections

• Connection filtering is a property of the session– Filter injected dynamically at runtime– Cannot inspect source code to determine how it behaves

• E.g., running stored proc written for filtered mode on unfiltered connection could lead to unintended results

• There are several operations that will not work in filtered connection in v1– DDL, DML on reference tables, …

• Fan-out, bulk operations not efficient in filtered mode– For now, filter=off is our best offer

38

Support Matrix

Connection Type Filtered Unfiltered Named (unfiltered)Operation

Dynamic SELECT P P PDML* (federated tables) P P PDML* (reference tables) X P P

DDL X P PViews (not indexed) P P P

UDF - activate P P PStored Proc - activate P P P

Trigger (all modes) - activate P P PCREATE/UPDATE Stats X P P

Bulk Opsopenrowset bulk, bcp, bulk

insert X P P

* not including SELECT & modules

^ autostats will work on all connections

System stored procs, intrinsics will be unaffected (run unfiltered)

39

Splitting a Member

Gat

eway

Existing Database

sales

Federation Member

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

USE FEDERATION ROOT WITH RESET

ALTER FEDERATION salesSPLIT AT (customer_id=50)

Using to the federation ROOT will pop you out of a member back into the database that hosts the federation

Range: Min...Max

customer order product

3

58

3

58

58

40

40

Two New Members

Gat

eway

Existing Database

sales

Federation Member

Range: Min...50

customer order product

33

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

USE FEDEDERATION ROOT WITH RESET

ALTER FEDERATION salesSPLIT AT (customer_id=50)

Federation Member

Range: 51...Max

customer order product

5858

5840

41

Two New MembersExisting Database

sales

Range: Min...50

customer order product

40

Conn

ectio

n:Server=az1cl321.db.windows.net;

Database=MyDB;

User=AppUser;

Passwd=****;

USE FEDEDERATION sales (customer_id=40) WITH FILTERING=ON, RESET;

Range: 51...Max

customer order product

5858

58

Gat

eway

40

SELECT *from customer

SELECT * from order