migrating to sql azure
DESCRIPTION
Deck for screencast April 2011TRANSCRIPT
Lynn Langit
Moving Your Data to the Cloud
Migrating Applications to SQL Azure
Sr. Developer Evangelisthttp://blogs.msdn.com/SoCalDevGal
April 2011
Migrating Applications to SQL Azure
OverviewSQL Azure – now and NEW
App Migration Data Access Tier Considerations
Database MigrationMigrating to SQL Azure from MySqlMoving Schema and Data into SQL AzureTransferring Large Data into SQL Azure
Future
Windows Azure PlatformGENERAL PURPOSE PROGRAMMING
LANGUAGES
Compute
Storage
Management
Marketplace
App FabricCDN
Windows Azure Platform
Relational
databaseData Sync
When Migrating Applications
Data Access APIs SupportSecurity ModelConnections & SizingAdmin and Programmability Surface SupportCollation Behavior
Data Access APIs
Supported APIs
Connection StringADO.Net - Encrypt=True and add @servername to User IDODBC - Encrypt=yes and add @servername to Uid
Supported Unsupported
ADO.Net .Net 3.5 SP1 and 4.0 ODBC - SNAC 10Entity Framework .Net 3.5 SP1 and 4.0SQL Server 2008 Driver for PHP v1.1
OleDB
Data Access APIs cont.
Protocols
Authentication Mode
Supported Unsupported
TCP/IP over port 1433 Dynamic portsNamed pipesShared memory
Supported Unsupported
SQL Auth Windows Auth
Connectivity & Sizing
Robust Connection Management through Retry Logic
FailoverHW Failures or Upgrades
Resource Management OperationsThrottling and Load Balancing
ALTER DATABASE (T-SQL)Changes to Database Name Changes to MAXSIZE/EDITION
ALTER DATABASE [TestDB] MODIFY (MAXSIZE=50GB)
GO
Security ManagementManage Logins - Authentication - SQL Server (only)
CREATE LOGIN [testlogin] WITH PASSWORD='password‘
Manage UserCREATE USER testuser FROM LOGIN testlogin;
Manage Server RolesEXEC sp_addrolemember ‘dbmanager', 'testuser'; [securityadmin role]EXEC sp_addrolemember ‘loginmanager', 'testuser'; [dbcreator]
Granting Database Access to a LoginCreate Login and Database in MasterCreate User and Grant Role in User Database
Viewssys.sql_logins. sys.databases, sys.database_principal
Firewall Management
Portal AccessProgrammatic Access
System Views and Stored Procedures - hereView: sys.firewall_rulesStored Procedure: sp_set_firewall_ruleStored Procedure: sp_delete_firewall_rule
API Implementation Example- here
CollationsDefault Collation - Set to SQL_LATIN1_GENERAL_CP1_CI_AS
Metadata, TempDb and Default for User Data Object Names (tables, sprocs etc), String variables and Columns when no collation is specified. Use column and/or expression level collation when needed
CREATE TABLE [dbo].[ErrorLog]([ErrorLogID] [int] IDENTITY(1,1) NOT NULL,[ErrorTime] [datetime] NOT NULL,[UserName] [nvarchar(4000)] COLLATE Turkish_100_CI_AI NOT NULL,[ErrorNumber] [int] NOT NULL,[ErrorMessage] [nvarchar](4000) COLLATE Japanese_CI_AS NOT NULL,
CONSTRAINT [PK_ErrorLog_ErrorLogID] PRIMARY KEY CLUSTERED ([ErrorLogID] ASC) )declare @str nvarchar(128) COLLATE Greek_CS_AI
Feature Parity
Admin Surface – no physical administrationYou have a master database You do NOT have server level constructs (sp_configure, endpoints, DBCC commands, server level DMVs and System Views…)
Programmability SurfaceMost RDMS features availablePartially available - USE, XML processing, deprecated T-SQL etc. – list hereSome Features are NOT available today
Full-text Search, Remote Data Access and Linked Servers, Distributed Transactions, Change Tracking, Service Broker etc.List is available here
Migrating ApplicationsApplication Data Access Tier Just Works Need to
Investigate
Data Access APIs Support ADO.Net, ODBC, EF and PHP
OleDB
Security Model SQL Auth Windows Auth
Connection Management Apps with transaction retry logic, 3-tiered, stateless or partially stateful apps
Legacy Apps2 tiered apps with online connections
Admin and Programmability Surface Support
Tables, Views, Sprocs, Functions, Triggers, built-in data types, etc.
Server level options, FTS, Service Broker, Distributed Transactions
Collation Behavior Apps explicit with collation options
Non-English apps depending on db/server collation defaults
Database Migration
Moving Schema and / or DataSource Database types
SQL ServerMySQLAccessOther RDMS
Data Migration Tools Overview
WizardsSQL Azure Migration Wizard (CodePlex) – hereMigration Assistant for Access or for MySQL – here or hereImport/Export WizardGenerate Script Wizard
Copy Database commandCREATE DATABASE DB2A AS COPY OF Server1.DB1A
Other Tools SSIS BCPDataSync (CTP)OData (CTP)
From SQL Server to SQL AzureGUI Tools
SSMS for R2 > Generate Script WizardSSMS for R2 > Import / Export WizardSSIS for R2SQL Azure Migration Wizard
Command-line ToolsT-SQL script & bcpCopy Database commandImport/Export Wizard for SQL Azure (command -> GUI)3rd Party tools
SSMS– Generate Script Wizard
Migration of schema and/or data w/ fine grain controlPros - Native support for SQL Azure Schema options: “Engine Type = SQL Azure”
Ensures correct options and settings are applied for the TSQL script generation.Requires explicit action on unsupported objects.
Cons - Verbose – INSERT Statements instead of raw data
Data is scripted with a fixed 100 row batch size.Edit “GO” statements between small batches
Round-trip EfficiencyUse “SET NOCOUNT ON”
Remove unsupported features, such as…
Migrating By Hand
USE DATABASE ON PRIMARY
(Filegroups) Extended Properties NOT FOR REPLICATION Various Index options (WITH) Windows logins Unsupported XML features Remove DEFAULT NEWSEQUENTIALID() Encrypted Stored Procedures No Heap tables (all tables must have clustered indices) UDTs
Generate Script Wizard - DEMO
Migrating Databases to SQL Azure
What Could Go Wrong?
Success!
Migration Assist for MySql / Access (CTP)
Scenario - Auto porting of schema, database code and data from MySql and Access to SQL AzureSQL Server Migration Assistant for MySql and Access
Supports MySQL 4.1 and upSupport Access v 97 and upSQL Server versions supported (all editions)
SQL Azure, SQL Server 2005, SQL Server 2008 and 2008 R2
MySQL to SQL Azure - DEMO
SQL Server Migration Assistant Version for MySQLVersion for Access
MySQL Migration Tool
Access to SQL Azure Migration Wizard - DEMO
SQL Azure - Import/Export
DAC v2.0 adds two new DAC services: import and exportImport and export services let you deploy & extract both schema & data from a single file identified with the “.bacpac” extension
DAC (Data-tier Application) PackageSingle unit for authoring, deploying, and managing data-tier objects (schema)
Development Lifecycle (VS 2010)
Creating / Editing DACsSchema and DB Code DevelopmentCode Analyses, Deployment Policy SettingsSchema Comparison and more…
Management Lifecycle (SSMS 2008 R2)
Extracting DACs (as files)Managing DACs (deploy or upgrade [coming later this year for SQL Azure)Registering existing database as DACs
If ‘regular’ into MSDBIf ‘Azure’ into masterIf using UCP into UCP database
DACPAC – DEMO in SSMS
In SSMS• Extract• Register• Monitor
w/UCPs
In VS 2010• Import• Create
DACPAC – DEMO in VS
Transferring Large Data to SQL Azure
Best Practices for efficient Data movement to SQL Azure
Use the right tools is key BCPSSIS
Network performance considerationsLatency vs Bandwidth
BCP & Bulk Copy APIs for Large Data Transfer
Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checking
-N Native mode so no data type conversion needed.
-c Character mode for interoperability
-b batchsize will let you control transaction size-h”TABLOCK, ORDER(…)” optimize writes
Import/Export Wizard – for data transfer
Scenario - Simplified wizard for migrating schema and data through SSISPros
Great performance out of the boxAllows ‘save as package’ for full control
SSIS Data Flow Task (Large Data Xfer)Diverse Source and destinations -
To/From Flat Files, ADO.Net, OleDBFully programmable flow
Loops, Sorts, Conditional operators, XML/WebServices Processing etcVS Debugging support with data viewers, watches and conditional breakpoints
Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checkingBatch Size: Adjust the transaction sizeBuffer and Blob temp storage area; spread over to multiple drives for large data transfersParallelization – based on execution trees, task will auto parallelize
SSIS package - DEMO
Use supported features only, such as bcp, but not BULK INSERT, etc…
Parallelization & Network Perf
Latency & Bandwidth for high performance import & export
Latency is about laws of physicsLong distances means high latencyParallelization can help mitigate some latency
Bandwidth is key! Once you reach the bandwidth limit parallelization won’t help
Datacenter Locations
BCP Performance – test set up exampleCREATE TABLE [dbo].[table1](
[c01] [int] NOT NULL, [c02] [int] NOT NULL, [c03] [smallint] NOT NULL, [c04] [int] NOT NULL, [c05] [smallint] NOT NULL, [c06] [smallint] NOT NULL, [c07] [char](50) NOT NULL, [c08] [char](24) NOT NULL, … [c17] [char](24) NOT NULL, PRIMARY KEY CLUSTERED ([c01] ASC,[c02] ASC))
CREATE NONCLUSTERED INDEX [index1] ON [dbo].[table1] ([c07] ASC)
Data Distribution:Row Size: 600BytesRow Count: 100KPages: ~7700Total Size: ~60MBEven Distribution on PkeyConstraints and Indexes
Compound Pkey (int+int) None-clustered key on char(50)No triggers or relationships
BCP Upload Perf - Rows/Sec
Low Bandwidth, Medium Latency - 114ms
Low Bandwidth, Very High Latency - 289ms
High Bandwidth, High Latency - 169ms
High Bandwidth, Medium Latency - 54ms
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Parallel Upload Batch Size 1K (sum rows/sec)
Native Sorted - Batch 10K (rows/sec)
Native - Batch 10K (rows/sec)
Native - Batch 100 (rows/sec)
BCP Download Perf – Rows/sec
Medium Bandwidth, Medium Latency - 114ms
Medium Bandwidth, Very High Latency - 289ms
High Bandwidth, High Latency - 169ms
High Bandwidth, Medium Latency - 54ms
0 2000 4000 6000 8000 10000 12000 14000 16000
4x Native Sorted - Batch Size 1K (sum rows/sec)
1x Native Sorted - Batch 10K (rows/sec)
Guidance on Data Transfer to SQL Azure
Split data transfer into multiple concurrent streamsTune BCP batch size and network packet size option Add non clustered indexes after loading data to SQL Azure.
Two additional indexes created before loading the data increased the final database size by ~50% and increased the time to load the same data by ~170%.If, while building large indexes, you see a throttling-related error message, retry using the online option.
COPY Database (same or different servers)CREATE DATABASE destination_database_name AS COPY OF [source_server_name.]source_database_nameCREATE DATABASE Database2A AS COPY OF Server1.Database1A
Check via DMVProgresssys.dm_database_copies
Database COPY
SQL Azure Data Sync
Elastic ScaleService scales as resources requirements grow
No-Code Sync ConfigurationEasily define data to be synchronized
Schedule SyncChoose how often data is synchronized
Conflict HandlingHandle issues where same data is changed in multiple locations
Logging and MonitoringAdministration capabilities for tracking data and monitoring potential issues
CTP 1
CTP 2
Data Sync
Table Order in Sync
Tracking Objects Created
• Tracking Tables• On Source / on
Destination• Triggers• Stored Procedures
SQL Azure Data Sync CTP3
• New UX integrated with the Windows Azure management portal.
• Ability to select a subset of columns to sync as well as to specify a filter so that only a subset of rows are synced.
• Ability to make certain schema changes to the database being synced without having to re-initialize the sync group and re-sync the data.
• Conflict resolution policy can be specified.• General usability enhancements.
Monitoring Tasks• Space Used• Monitor space used via
‘sys.dm_db_partition_stats’• Trap ‘out of space exception’ – SQL
Error 40544• Synchronized Copies• Database copy• SQL Azure Synchronization Provider• Database backup / restore (and point-
in-time restore) on 2011 Q2 Roadmap
Data Migration Tools Summary
WizardsSQL Azure Migration Wizard (CodePlex): http://sqlazuremw.codeplex.com/
Migration Assistant for Access or for MySQL:http://www.microsoft.com/downloads/en/details.aspx?FamilyID=5abe098d-c7e1-46c6-994a-09a2856eef0b&displaylang=en
Import/Export WizardGenerate Script Wizard
Copy Database commandCREATE DATABASE DB2A AS COPY OF Server1.DB1A
Other Tools SSIS BCPDataSync (CTP)OData (CTP)Import / Export Tool(coming spring 2011)
Database Migration Tools
Tools SSMA
Generate Script Wizard DACs SSIS BCP
MySql and Access to SQL Azure
√ (Schema
and Data)
√ (Data Only)
Move SQL Schema √ √ √
Move Data √ √ √
Move Large Data √ √
Try SQL Azure for FREE
www.sqlazure.com/getstarted
Activate your free trial!
What’s Your Idea?
www.sqlazure.com/communityCheck out tutorials and hear from MVPs…
Track Resources
Team Blog – http://blogs.msdn.com/sqlazure
MSDN Dev Center – http://msdn.microsoft.com/azure/sqlazure
Follow us on Twitter - @sqlazure
Windows Azure Platform Training Kit - http://bit.ly/dgvMv0
SQL Azure Labs – http://www.sqlazurelabs.com
TeachingKidsProgramming.org
• Do a Recipe Teach a Kid (Ages 10 ++)• Microsoft SmallBasic Free Courseware (recipes)
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft
cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.