optimizing database access and diving into.net sqlclient mladen prajdić sql server mvp...
Post on 16-Dec-2015
213 Views
Preview:
TRANSCRIPT
Optimizing database access
and diving into .Net SqlClient
Mladen PrajdićSQL Server MVP
mladenp@gmail.com @MladenPrajdic
About me
Welcome to Slovenia
The sunny side of Alps
Disclaimer
I use SELECT * in demos only for
brevity
Agenda
• Building foundation... Client Data Access
• New exciting stuff! Async SqlDataReader
• Consistency is the key! TransactionScope
• Reduce, Reuse, Recycle! Connection Pooling
• Make it go faster! Batching Commands
• What’s it doing? Statistics
• Multiple interleaves? MARS
Correct Connection Use
using (SqlConnection conn = new SqlConnection(_connString)){ conn.Open(); // db access code here // return data reader or objects}
Client Data Access
Building the foundation...
TDS Data Flow
Result set
“Done” token
Result set
“Done” token
Result set
“Done” token
using (SqlCommand cmd = new SqlCommand("SELECT * FROM Table1; SELECT * FROM Table2", conn)){ SqlDataReader rdr = cmd.ExecuteReader(CommandBehavior.CloseConnection); // return the list of data objects return CreateListOfBusinessObjects(rdr); // or return the SqlDataReader itself return rdr;}
TDS Data Flow
Result set
“Done” token
Row Data“Row” token Row Data“Row”
token Row Data“Row” token
Result set
“Done” token
Result set
“Done” token
TDS Data Flow
Result set
“Done” token
Row Data“Row” token Row Data“Row”
token Row Data“Row” token
Result set
“Done” token
Result set
“Done” token
Column Data
Column
Header
Column Data
Column
Header
Column Data
Column
Header
Small datatype INTMedium datatype
CHAR(50)Large datatype NVARCHAR(MAX)
SqlDataReader Data Access• Default access (Non-Sequential)
• Sequential access
// default = non-sequential accessvar rdr = cmd.ExecuteReader();
// sequential accessvar rdr = cmd.ExecuteReader(CommandBehavior.SequentialAccess);
Non-Sequential (Default) Access
Column Data
Column
Header
Column Data
Column
Header
Column Data
Column
Header
0 1 2
Column Data
Column
Header
Column Data
Column
Header
Column Data
Column
Header
0 1 2
We can go back and read all previous columns’ data
Non-Sequential (Default) Access• The full row is stored in client memory• Move back and forward in the row• Possible scalability issues• Possible Out of Memory exceptions
Sequential Access
Column Data
Column
Header
Column Data
Column
Header
Column Data
Column
Header
0 1 2
Column Data
Column
Header
Column Data
Column
Header
Column Data
Column
Header
0 1 2
Forward reading only. Trying to go back throws an error.
ERROR
Sequential Access
• Only the current value is stored in client memory• Move only forward in the row• Get* methods for getting data• GetBytes, GetChars for BLOBs• Save to a stream by reading it in chunks// get first column value that is intrdr.GetInt32(0);// get second column value that is binarybyte[] buffer = new byte[100];// loop this to get the whole column out in chunksrdr.GetBytes(0, 100, buffer, 0, buffer.Length); // save the byte buffer to file or something
Sequential Access
• Full streaming support in .Net 4.5• GetStream, GetTextReader, GetXmlReader• No more chunking
Sequential Access
SqlCommand readCmd = // get varbinary(MAX) from source table new SqlCommand("SELECT BinData FROM Source", readConn);SqlCommand writeCmd = // insert varbinary(MAX) into target table new SqlCommand("INSERT INTO Target(BinData) VALUES (@BinData)", writeConn);// Size is set to -1 to indicate "MAX"SqlParameter streamParameter = writeCmd.Parameters.Add("@BinData", SqlDbType.Binary, -1);SqlDataReader readerSource = readCmd.ExecuteReader(CommandBehavior.SequentialAccess);while (readerSource.Read()){ // Grab a stream to the binary data in the source database Stream dataStream = readerSource.GetStream(0); // Set the parameter value to the stream source that was opened streamParameter.Value = dataStream; // send the data to the target database writeCmd.ExecuteNonQuery();}
Async SqlDataReader
New exciting stuff!
Old “Run in Background” Ways
SqlCommand cmd = new SqlCommand("SELECT * FROM Table", conn);
// either this or a background workerIAsyncResult result = cmd.BeginExecuteReader();
// for the reader to completewhile (!result.IsCompleted){ // do UI operations without problems}
using (SqlDataReader reader = cmd.EndExecuteReader(result)){ // show reader results ShowDbOperationResult(reader);}
What exactly is the new stuff then?• Make C# async programming easier• Since .Net 4.5• Supports easy cancellation
• Uses the new async/await keywords• Reader’s *Async methods
private async void CustomEventHandler(object sender, EventArgs e){ textBox.Text = "Start long operation." + Environment.NewLine; // await returns control to the caller // this means that the UI won't be blocked // since CPU is free to do its stuff double duration = await MyLongIOBoundOperation(); // finish op and show duration in UI textBox.Text = "Completed in " + duration + " sec";}
What is it good for?
• Make code async without complex threading code• No threads created by default• Use default for IO-bound workload• Use Task.Run to create background thread for
CPU-bound workload• Supported by Entity Framework 6
• For more in depth details on this, search for “Asynchronous Programming with Async and Await”• http://
msdn.microsoft.com/en-us/library/hh191443.aspx
What to use when?
• Prefer NextResultAsync in either access modes
• TDS tokens between result sets are processed in async
Result setTDS
TokensAsync
Result setTDS
TokensAsync
Result set
What to use when?
• Prefer ReadAsync in either access modes
• TDS tokens between multiple packets are processed in async
Data PacketTDS
TokensAsync
Data PacketTDS
TokensAsync
Data Packet
Single Row
What to use when?
• Use IsDBNull and GetFieldValue<T> • If ReadAsync() in Default (Non-Sequential) mode
• Since data is already in the client buffer
• IsDBNullAsync and GetFieldValueAsync<T>• Sequential mode
• target column is large• needed to read past large columns
• rdr.GetStream(i).CopyToAsync(outputStream)• for huge columns in Sequential mode
DEMO
TransactionScope
Consistency is the key!
TransactionScope
using (TransactionScope txScope = new TransactionScope()){ using (SqlConnection conn2012AW = new SqlConnection(_connString2012AW)) { // do stuff } txScope.Complete();}
TransactionScope
• Easy to use• Keeps code simple• Great transaction management out of the box• MSTDC escalation• Always, unless connecting to the same database
with non concurrent connections• sys.dm_tran_active_transactions.transaction_uow
• Default isolation level is SERIALIZABLE
DEMO
Connection Pooling
Reduce, Reuse, Recycle!
Connection Pooling
• Connection pools created• per process• per application domain • per connection string• per Windows identity (if integrated security)
• Default max pool size = 100• Audit Login, Audit Logout Events• EventSubClass
• 1 = Nonpooled• 2 = Pooled
Connection Pooling Resets
• sp_reset_connection• Resets the connection for reuse• A long list of what it does on the next slide for
reference
• Does NOT reset the isolation level• Always explicitly specify the isolation level
Connection Pooling Resets
• All error states and numbers (like @@error)• Stops all EC's (execution contexts) that are child threads of a parent EC executing a parallel query• Waits for any outstanding I/O operations that is outstanding• Frees any held buffers on the server by the connection• Unlocks any buffer resources that are used by the connection• Releases all allocated memory owned by the connection• Clears any work or temporary tables that are created by the connection• Kills all global cursors owned by the connection• Closes any open SQL-XML handles that are open• Deletes any open SQL-XML related work tables• Closes all system tables• Closes all user tables• Drops all temporary objects• Aborts open transactions• Defects from a distributed transaction when enlisted• Decrements the reference count for users in current database which releases shared database
locks• Frees acquired locks• Releases any acquired handles• Resets all SET options to the default values• Resets the @@rowcount value• Resets the @@identity value• Resets any session level trace options using dbcc traceon()
DEMO
Batching Commands
Make it go faster!
Batching Commands
• Optimizing network round trips• Consider Bulk Insert if possible• Streaming data insert• SqlBulkCopy Class
• 10k inserts = 10k round trips• 1000 x 10 batches = 10 round trips• 1000 CSV SQL statements in one command• Or… a hidden gem
DEMO
Statistics
What’s it doing?
Client Statistics
• Enable per connection• Cumulative• Good for tracking the actual db access from
app
• Save to a queue, process off-peak• Analyze historical access data
DEMO
MARS
Multiple interleaves?
Multiple Active Result Sets (MARS)• Statement multiplexing/interleaving • SELECT, FETCH• RECEIVE, READTEXT• BULK INSERT / BCP
• Other statements must complete• Transaction savepoints aren’t allowed
• Usually used for SELECT N+1 scenarios
DEMO
?
top related