tony petrossian microsoft corporation tony.petrossian@microsoft.com best practices
Post on 14-Dec-2015
218 Views
Preview:
TRANSCRIPT
Tony PetrossianMicrosoft CorporationTony.Petrossian@microsoft.com
SQL Azure
Best Practices
AgendaService Recap“Server” Management Connectivity and TracingUsing multiple databasesMigrating data and schema in and out of SQL Azure
SQL Azure Recap
Relational database serviceBuilt on SQL Server technology foundationHighly scaledHighly secureDatabase “as a Service” – beyond hosting
Customer Value PropsSelf-provisioning and capacity on demandSymmetry w/ on-premises database platformAutomatic high-availability and fault-toleranceAutomated DB maintenance (infrastructure)Simple, flexible pricing – “pay as you grow”
SQL Azure Topology
TDS Gateway
Front-end Node
Protocol Parser
Gateway LogicTDS
Session 1
Front-end Node
Protocol Parser
Gateway Logic
Master Cluster
Master Node
Data Node Components
Partition Manager
Master Node
Data Node Components
Partition Manager
1
2
3
5
7
8
9
Machine 5
SQL Instance
SQL DB
DB5 DB3DB2
Scalability and Availability: Fabric, Failover, Replication, and Load balancing
Machine 6
SQL Instance
SQL DB
Master1Meta 1 DB2 DB3
Machine 4
SQL Instance
SQL DB
DB2 DB3 DB4
Machine 7
SQL Instance
SQL DB
DB4 DB5
Scalability and Availability: Fabric, Failover, Replication, and Load balancing
46
Master1
SQL Azure Connection
Applications connect directly to a database“Initial Catalog = <db>” in connection stringNo support for context switching (no USE <db>)
SQL Azure connection strings follow SQL syntaxExcept for an unusual username format
Format of username for authentication:ADO.Net:Data Source=server.database.windows.net;User ID=user@server;Password=password;...
ODBC:Driver={SQL Server Native Client 10.0}; Server=server.database.windows.net; Uid=user@server;Pwd=password;...
Recommendations
Server ManagementWhere should I create my server?
Co-locate server with Windows Azure app to reduce latency
When should I create a new server?Trade off between geo/admin/billing
Managing ServerThrough the Portal
Add/Drop serverEstablish admin credentialsView usage reportsNetwork access configuration
Through the Master DatabaseFine-tune firewall settings through codeUser loginsUsage and metrics reporting (billing)Create/Drop databases
Admin roles permissionsCREATE/DROP database CREATE/DROP/ALTER loginGRANT/REVOKE rightsModifying server firewall settings
Portal
Master DB
User DB
User DB
User DB
HTTP
TDS
SQL Azure Server
Server: Network Access Control
Each server defines a set of firewall rulesDetermines access policy based on client IPBy default, there is NO ACCESS to server
Controlled using TSQL API against Master DB: sys.firewall_rules, sys.sp_set_firewall_rule, sys.sp_delete_firewall_rule
Portal UXID Name Start IP End IP Create Modify
1 Office 12.1.2.0 12.1.2.255 2010-03-18 …
2010-03-18 …
2 Home 12.2.2.5 12.2.2.5 2010-03-19 …
2010-03-19 …
3 All 0.0.0.0 255.255.255.255
2010-03-19 …
2010-03-19…
Server: Billing and Reporting
sys.bandwidth_usage: usage in KBTime Databas
eDirection
Class Period Quantity
2009-09-17 19:00
TPCH Egress Internal
Peak 55598
2009-09-17 19:00
TPCH Ingress Internal
Peak 76026
… … … … … …
Time SKU Quantity
2009-09-17 19:00
Web 1
2009-09-17 19:00
Business 10
… … …
sys.database_usage: instance count by type per day
Application Design Topics
Most-applicable SQL Best PracticesConnection PoolingQuery ParameterizationBatching
Scaling with data and loadShardingBuilding copies
Deploying and uploading dataBulk copy, BCP.EXESSISMicrosoft Sync Framework Power Pack for SQL Azure
Use Pooled Connections
// When pooling, use connection and return immediately// Do not hold for a long time – pool ensures fast turnaround// on second useusing (SqlConnection conn = new SqlConnection(…)){ conn.Open(); using (SqlCommand cmd = conn.CreateCommand()) { cmd.CommandText = …; … }}using (SqlConnection conn = new SqlConnection(…)){ conn.Open(); …
Increases efficiency by removing re-login
Connections: Retry on failureConnections can drop for variety of reasons
Idleness for more than 30-minutesTransient (network) errorsIntentional throttling
First step: reconnect immediatelyHandles idleness- and transient-disconnects
The SQL Azure Gateway handles connection retry for appConnections attempted for ~30s before failure
What to do on connection failure?Wait (10 seconds), then retry Check the error messages and follow recommendation
Server health can be checked via PortalTSQL APIs will come in later releases
Errors to Catch
Error Severity Description (message text)
40197 17 The service has encountered an error processing your request. Please try again. Error code %d.
40501 20 The service is currently busy. Retry the request after 10 seconds. Code: %d.
40544 20 The database has reached its size quota. Partition or delete data, drop indexes, or consult the documentation for possible resolutions.
40549 20 Session is terminated because you have a long-running transaction. Try shortening your transaction.
40550 20 The session has been terminated because it has acquired too many locks. Try reading or modifying fewer rows in a single transaction.
40551 20 The session has been terminated because of excessive TEMPDB usage. Try modifying your query to reduce the temporary table space usage.
40552 20 The session has been terminated because of excessive transaction log space usage. Try modifying fewer rows in a single transaction.
40553 20 The session has been terminated because of excessive memory usage. Try modifying your query to process fewer rows.
40613 17 Database '%.*ls' on server '%.*ls' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '%.*ls'
Connection Patternwhile (true){ using (SqlConnection connection = new SqlConnection(connStr)) { try { connection.Open(); using (SqlCommand cmd = connection.CreateCommand()) { cmd.CommandText = @"SetBCPJobStartTime"; cmd.CommandType = CommandType.StoredProcedure; cmd.Parameters.Add(new SqlParameter(@"@BCPJobId", BCPJobId)); cmd.ExecuteNonQuery(); } } catch (SQLException Sqexc) { // deal with error } catch (Exception exc) { // deal with error } } // more stuff // …..}
Keep it inside the main loop
Tracing Connectivity Problems
Each session assigned a unique ‘sessionId’Tracks session state and service errors
Retrievable from CONTEXT_INFO()Save this with each new connection
If you need support, support personnel will greatly appreciate that value
Tracing Helper Pattern
Guid? sessionId = null;using (SqlConnection conn = new SqlConnection(…)){ // Grab sessionId from new connection using (SqlCommand cmd = conn.CreateCommand()) { conn.Open(); cmd.CommandText = "SELECT CONVERT(NVARCHAR(36), CONTEXT_INFO())"; sessionId = new Guid(cmd.ExecuteScalar().ToString()); }
// continue processing ...}
Maximize your Performance
Batching: push logic to ServerUse stored procedures and batchingLimit number of round trips to server
Example: batch 10 statements vs. 10 round-trips
Parameterized queriesParameterize queries (limits compiles)Declare all parameters, type and lengthLack of parameter size leads to cache bloat
Declare Parameter Lengths!// Length inferred: leads to cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "1";…cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "22";
// Length supplied: no cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar, 128).Value = "1";
(@1 nvarchar(1)) SELECT c1 FROM dbo.tbl WHERE c2 = @1(@1 nvarchar(2)) SELECT c1 FROM dbo.tbl WHERE c2 = @1
(@1 nvarchar(128)) SELECT c1 FROM dbo.tbl WHERE c2 = @1
Maximize elasticity benefits
SQL Azure balances databases across machines
Divide your data into smaller chunksMakes for better load-balancingEnsures we can place your data on the most-appropriate servers
Using shorter transactionsEnsures we can respond to issues fasterAvoid impacting others sharing the same box
Thus avoiding being throttled by SQL Azure
Multi-Database Case Study
Provider of vehicle valuation data> 13 million visitors to their site per month2.5 GB Database SizeData refreshed every weekReplicate data across 6 databases for increased perfIncrease/Decrease database count based on demand
Kelly Blue Book http://www.kbb.com/
Copies: Managing Lots of Read Access
SQL Azure ClusterNode
14Node
19Node
21Node
99 Node 2Node
76
App
Server
Master DB
DBC1
DBC2
Node 33
DBC3
DBC4
DBC5
DBC6
LB
SQL Azure
Castellan
Castellan Venue
DB
Castellan Venue DB’s
Venue 1Partition(
s)
Castellan Venue
DB
Castellan Venue DB’s
Venue 2 Partition(
s)
Castellan Venue
DB
Castellan Venue DB’s
Venue N Partition(
s)
One application DB, many venue DB’s – each partitioned in to many parts (40+)
...
Azure Roles
http:// TicketDirect .* Dynamic Worker(tasks
uploaded as blobs)
Partitioner Worker
Azure StorageQueues for communication between clients and roles
-- - --- - -
-- - --- - -
Tables to record server & partition information
Blobs to store web and worker role resources
Client Applications
Castellan.old (VB6)
Castellan.Azure• Box Office sales• Ticket Printing • System Administration • Venue/Event Management• Partitioning
.Net Service Bus
WCF
On PremiseSQL Server
Castellan Venue
Distributed Cache Worker
MemCache
Sharding Example: TicketDirect Architecture
Shards: Managing Lots of Transactions
SQL Azure ClusterNode
14Node
19Node
21Node
99 Node 2Node
76
App
Server
Master DB
DB1
DB2
Node 33
DB3
DB4
DB5
DB6
Migrating Databases to SQL Azure
Managing the Schema
Moving between SQL Server & SQL AzureUse SSMS script generation toolSQL Azure Migration Wizard
http://sqlazuremw.codeplex.com
Use Data-Tier ApplicationVS2010, SQL Server 2008 R2, SQL Azure SU2 (April 2010)
Build your own scriptsWatch out for unsupported T-SQLDon’t depend on instance or server level objects
Getting Data In and Out
SQL Azure supports standard SQL data import and export patterns
Use bulk loading patterns where possibleBCP – console .EXE bulk load/export toolSSIS – SQL integration serverBulk APIs in ODBC and ADO.Net
SQL Azure supports data synchronizationWith on-premises DBs and client stores
Always good advice:Break batches up into smaller, consumable chunksAdd retry and tracing logic to ensure robust resume in face of failures
Data Import: ADO.Net Bulk Copy API// Bulk data importusing (SqlBulkCopy bulk = new SqlBulkCopy(new SqlConnection(conn)) { DestinationTableName = "dbo.data", BatchSize = 2000, // Transaction size (length) BulkCopyTimeout = 10000, // Transaction timeout NotifyAfter = 1000, // Progress callback }){ bulk.SqlRowsCopied += new SqlRowsCopiedEventHandler( myProgressCallback); bulk.WriteToServer(sourceDataReader);}
Data Export/Import: BCP.EXE
// BCP exampleSET SRV=somesrv.database.windows.netSET LOGIN=mylohin@somesrvSET PW=somethingSET S_DR=C:\flatsSET DB=TPCH
bcp %DB%.dbo.supplier in %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.nation in %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.region in %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.customer in %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.part in %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|“
bcp %DB%.dbo.supplier out %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.nation out %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.region out %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.customer out %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp %DB%.dbo.part out %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"
Summary
Many SQL Server patterns apply to SQL AzureUse SQL best practices wherever possible
Patterns discussed:Connectivity (to database, not server)Tracing and supportBatching, Pooling and ParameterizationGetting data in and out
Additional Resources
Azure Prices and Packages:http://www.microsoft.com/WindowsAzure/offers/
SQL Azure Information:http://www.microsoft.com/windowsazure/sqlazure/
MSDN Resourceshttp://www.microsoft.com/windowsazure/sqlazure/
Thank you for your Attention!For more Information please contact
Tony PetrossianPrincipal Program ManagerSQL Azure Development
Microsoft Corporationtonypet@microsoft.com
top related