DAT309:SQL Server CE 2.0 Performance Tuning, Deployment, And Scalability

Kevin Collins

Microsoft

Senior Program Manager

Agenda

SQL Server CE with ADO.NET performance tips

SQL Server CE QP performance tips

Mobile application deployment techniques

Sync scaling strategies

Native Versus Managed

Absolute fastest data access is OLEDBCE via eVC

Much more code and harder to maintain

ADO.NET Verses ADOCE Perf

1. Seek to a random value1. Seek to a random value2. Read entire row2. Read entire row3. Repeat (n) times3. Repeat (n) times4. Close the cursor4. Close the cursor

1. SELECT * FROM tab WHERE id=n1. SELECT * FROM tab WHERE id=n2. Read entire row2. Read entire row3. Close the cursor 3. Close the cursor 4. Repeat (n) times4. Repeat (n) times

SeekSeek TestTest

Number of rowsNumber of rows

00

55

1010

1515

2020

2525

3030

3535

4040

11 1010 100100 10001000

Tim

e (

se

co

nd

s)

Tim

e (

se

co

nd

s)

ADO.NETADO.NET ADOCEADOCE

Single Fetch without parametersSingle Fetch without parameters

Number of rowsNumber of rows

00

5050

100100

150150

200200

250250

11 1010 100100 10001000

Tim

e (

se

co

nd

s)

Tim

e (

se

co

nd

s)

ADO.NETADO.NET ADOCEADOCE

Data Reader Versus Data Set

Whenever possible use Data Reader

Significantly faster than DataSet, especially when using larger result sets

NCS saw 5X performance improvements going to Data Reader

http://www.ncssuite.com/NewsArticles/NZ%20NETArticleR1.pdf

Direct access to SQL CE

No double buffering of data

DataSet updates ALL columns

DownsideNot bound to controls

Not scrollable or updateable

Data Reader – Optimal Usage

When reading entire rows, consider using GetValues() rather than calling GetXXX multiple times

Use strongly typed getters rather than GetValue()

Refer to the columns by ordinal rather than name

Using SqlTypes is more expensive than CLR types

**** List by ordinal versus by nameExample:Example:

object[] row = new object[rdr.FieldCount];object[] row = new object[rdr.FieldCount];while (rdr.Read())while (rdr.Read()){{ // Fewer function calls inside the data provider// Fewer function calls inside the data provider rdr.GetValues(row);rdr.GetValues(row); // Process the row here// Process the row here} }

GetValues() Versus GetValue(i) #12

demodemo

QP Optimizer

Heuristic

Rewrites query into semantically equivalent form that leads to a better execution plan

Based on syntax

Rewrite rules are applied if applicable

Adjacent inner joins are merged together so alternative join orders may be considered... FROM (Table_1 INNER JOIN Table_2 ON Table_1.Col = Table_2.Col) INNER JOIN

Table_3 ON Table_1.Col = Table_3.Col ...

... FROM Table_1, Table_2, Table_3 WHERE Table_1.Col = Table_2.Col AND Table_1.Col = Table_3.Col ...

Cost BasedDetermines

Base table scan type (File scan / Index scan)

What indexes, if any, are used

Join order, join algorithm

Sort / filter placement

How does it do itEnumerates a selected subset of all possible execution plans

Finds out the plan with lowest estimated cost

Generates executable data structures that implement the plan

Schema Tips

Variable length columnsKeeps db size down

Row might move with column growthSize increase

Defragmentation

Denormalize?

Joins of more than 5 tablesQP might not come up with optimal plan

Modifying SQL Statements

These have no impactOrder of tables in old-style FROM clause“… FROM Foo, Bar …” ==

“… FROM Bar, Foo …”

Order of predicates in the same clause“… WHERE Col1 = 2 AND Col2 > 10 …” ==

“… WHERE Col2 > 10 AND Col1 = 2 …”

Order of INNER JOIN tables“… FROM Foo INNER JOIN Bar …” ==

“… FROM Bar INNER JOIN Foo …”

Optimizer re-arranges the order in search for better execution plan

Subquery Versus Join

Evaluate both

Subquery may be written using JoinJoin can evaluate tables in different order from subquery

Subquery may not need to scan all rowsReturns after seeing first occurrence

IN subqueriesAlways re-writen to join

No need to re-write to explicit join

Subquery Versus Join #4Subquery Versus Join #4

demodemo

Subquery Versus Join

Orders that have at least one item with discount of 25% or moreOrders that have at least one item with discount of 25% or more

SELECT "Order ID" FROM Orders O WHERE EXISTS (SELECT "Order ID" FROM "Order Details" SELECT "Order ID" FROM Orders O WHERE EXISTS (SELECT "Order ID" FROM "Order Details" OD WHERE O."Order ID" = OD."Order ID" AND Discount >= 0.25) OD WHERE O."Order ID" = OD."Order ID" AND Discount >= 0.25)

SELECT DISTINCT O."Order ID" FROM Orders O INNER JOIN "Order Details" OD ON O."Order SELECT DISTINCT O."Order ID" FROM Orders O INNER JOIN "Order Details" OD ON O."Order ID" = OD."Order ID" WHERE Discount >= 0.25ID" = OD."Order ID" WHERE Discount >= 0.25

Filters In WHERE Versus HAVING

Non-aggregate filters in WHERE clause instead of HAVING clause

WHERE is evaluated before HAVING

Result is reduced early in query processing

Filters – WHERE Versus HAVING

Order count for customers where id start with ‘A’Order count for customers where id start with ‘A’

• SELECT "Customer ID", COUNT("Customer ID") FROM Orders GROUP BY SELECT "Customer ID", COUNT("Customer ID") FROM Orders GROUP BY "Customer ID" HAVING "Customer ID" >= 'A' AND "Customer ID" < 'B‘"Customer ID" HAVING "Customer ID" >= 'A' AND "Customer ID" < 'B‘• SELECT "Customer ID", COUNT("Customer ID") FROM Orders WHERE SELECT "Customer ID", COUNT("Customer ID") FROM Orders WHERE "Customer ID" >= 'A' AND "Customer ID" < 'B' GROUP BY "Customer ID""Customer ID" >= 'A' AND "Customer ID" < 'B' GROUP BY "Customer ID"

Filters – WHERE Versus Filters – WHERE Versus HAVING #5HAVING #5

demodemo

Create Useful Indexes

Create selective indexesSelectivity is ratio of qualifying rows to total rows

Index on Orders.OrderID is selective

Index on Orders.ShipVia is not selective

SQL CE only uses one index per tableIndexes will slow down DML

Increases database size

Evaluate use of applicationHeavy DML less indexes

Heavy querying more indexes

Useful Indexes #6Useful Indexes #6

demodemo

Multi-Column Indexes

CREATE INDEX Emp_name on Employees (“Last Name” ASC, “First Name” ASC)

It may optimizeWHERE “Last Name” = ‘Doe’WHERE “Last Name” = ‘Doe’ AND “First Name” = ‘John’ORDER BY / GROUP BY “Last Name”ORDER BY / GROUP BY / DISTINCT “Last Name”, “First Name”DISTINCT “Last Name”, “First Name”

It cannot optimizeWHERE “First Name” = ‘John’ORDER BY / GROUP BY / DISTINCT “First Name”ORDER BY / GROUP BY “First Name”, “Last Name”DISTINCT “Last Name”

Joins

Indexes used to speed up JOINsEngine automatically creates index for PK and FK

Not good for duplicate values

If no PK/FK index join columns

Join Columns Index Join Columns Index Versus Non-Indexed #2Versus Non-Indexed #2

demodemo

Optimizable Versus Non-Optimizable Clauses

Optimizable clausesColumn operator <constant or variable> or

<constant or variable> operator Column

>, <, >=, <=, BETWEEN, IN, and LIKE (prefix matching, such as LIKE 'John%')

Non-optimizable clausesNOT, <>, (NOT) EXISTS, NOT IN, NOT LIKE, and intrinsic functions

Turn clauses into optimizable onesUPPER(“Customer ID”) = ‘ANTON’ -> “Customer ID” = ‘ANTON’

ORDER BY / GROUP BY / DISTINCT

ORDER BY / GROUP BY Prefix set of index columns must match ORDER BY / GROUP BY columns with exact order

Index more columns, but not fewer ones

DISTINCTIndex columns must match projection list

ORDER BY – Index Versus Sort

The following queries are run on tables of different cardinality with and without indexesThe following queries are run on tables of different cardinality with and without indexes

SELECT "Customer ID" FROM Customers ORDER BY "Customer ID"SELECT "Customer ID" FROM Customers ORDER BY "Customer ID"SELECT "Order ID" FROM Orders ORDER BY "Order ID"SELECT "Order ID" FROM Orders ORDER BY "Order ID"SELECT "Order ID" FROM "Order Details" ORDER BY "Order ID"SELECT "Order ID" FROM "Order Details" ORDER BY "Order ID"

Outer Join Performance Issues

SQL CE doesn’t re-arrange outer joins

Potentially modify database design Products – Categories scenario

Instead of inserting NULL as “Category ID”, add “Misc” or “No Category” category value

Outer join is avoided when joining the two tables together

Parameterized Queries

Compile once and execute many timesGood for a set of queries that are only different in a few constant valuesHold on to the SqlCeCommand object that caches the compiled planCreate multiple SqlCeCommand objects if you run multiple parameterized queries at the same time

Example:Example:// Create a parameterized query// Create a parameterized querycmd.CommandText = “UPDATE myTable set col1=? where cmd.CommandText = “UPDATE myTable set col1=? where col2=?";col2=?";

// Avoid SqlCeParameter(string, object) constructor// Avoid SqlCeParameter(string, object) constructor cmd.Parameters.Add("p1", SqlDbType.Decimal);cmd.Parameters.Add("p1", SqlDbType.Decimal);cmd.Parameters.Add("p2", SqlDbType.SmallInt);cmd.Parameters.Add("p2", SqlDbType.SmallInt);cmd.Prepare();cmd.Prepare();

// Execute multiple times …// Execute multiple times … cmd.Parameters["p1"].Value = 123456.54;cmd.Parameters["p1"].Value = 123456.54;cmd.Parameters["p2"].Value = 1;cmd.Parameters["p2"].Value = 1; cmd.ExecuteNonQuery();cmd.ExecuteNonQuery();

Parameterized Queries

Base Table Cursor

Bypasses the Query ProcessorFetches all the columns in a rowThe fastest way of reading all the columns from a table

Example:Example:

// Create and execute SqlCeCommand// Create and execute SqlCeCommandSqlCeCommand cmd = new SqlCeCommand(“Authors",cnn);SqlCeCommand cmd = new SqlCeCommand(“Authors",cnn);cmd.CommandType = CommandType.TableDirect;cmd.CommandType = CommandType.TableDirect;IDataReader dr = cmd.ExecuteReader();IDataReader dr = cmd.ExecuteReader();

// Retrieve Results// Retrieve Resultswhile(dr.Read())while(dr.Read()){ { Console.WriteLine("Name = " + dr["au_lname"]);Console.WriteLine("Name = " + dr["au_lname"]);}}dr.Close();dr.Close();cnn.Close();cnn.Close();

Seek/SetRange – Basic Usage

Used to open base table indexThe fastest way of selecting range of values

Example:Example:cmd.CommandType = CommandType.TableDirect;cmd.CommandType = CommandType.TableDirect;cmd.CommandText = "Orders";cmd.CommandText = "Orders";

//Assume: Index contains one column [datetime]//Assume: Index contains one column [datetime]cmd.IndexName = "SomeIndex"; cmd.IndexName = "SomeIndex"; object[] start = new object[1];object[] start = new object[1];object[] end = new object[1];object[] end = new object[1];start[0] = new SqlDateTime(2001, 1, 1); start[0] = new SqlDateTime(2001, 1, 1); end[0] = new SqlDateTime(2002, 2, 3;end[0] = new SqlDateTime(2002, 2, 3;

cmd.SetRange(DbRangeOptions.Match, start, end); cmd.SetRange(DbRangeOptions.Match, start, end); SqlCeDataReader rdr = cmd.ExecuteReader(); SqlCeDataReader rdr = cmd.ExecuteReader(); rdr.Seek(DbSeekOptions.FirstEqual, rdr.Seek(DbSeekOptions.FirstEqual, new SqlDateTime(2001,3,4));new SqlDateTime(2001,3,4)); while(rdr.Read()) { } while(rdr.Read()) { } // Read data in the usual way// Read data in the usual way

Seek/SetRange Versus Scrollability

SqlCeDataReader has limited scrolling abilityDataAdapter

does not leverage this capabilitymay be used with base-table cursor (TableDirect) can use CommandBuilder with base-table cursor

Example:Example:// Set the index range// Set the index range cmd.SetRange(DbRangeOptions.InclusiveStart, start, end); cmd.SetRange(DbRangeOptions.InclusiveStart, start, end); SqlCeDataReader rdr = cmd.ExecuteReader(); SqlCeDataReader rdr = cmd.ExecuteReader();

// Seek to a value (customer name)// Seek to a value (customer name) rdr.Seek(DbSeekOptions.FirstEqual,”Collins”);rdr.Seek(DbSeekOptions.FirstEqual,”Collins”);rdr.Read();rdr.Read();// Process the row in the usual way// Process the row in the usual way // Seek to another value (customer name)// Seek to another value (customer name) rdr.Seek(DbSeekOptions.LastEqual,”Sabino”);rdr.Seek(DbSeekOptions.LastEqual,”Sabino”);rdr.Read();rdr.Read();// Process the row in the usual way// Process the row in the usual way

rdr.Close();rdr.Close();

Use Queries?

SELECT "Customer ID" FROM Orders WHERE "Order ID" = 10123SELECT "Customer ID" FROM Orders WHERE "Order ID" = 10123VSVS

Base TableBase Table

Encryption?

190 188

347 335

0

50

100

150

200

250

300

350

Tim

e in

Se

co

nd

s

SELECT UPDATE

Encryption vs No Encryption

EncyrptedNo Encryption

160K rows

UPDATE Table Set NonIndexCol = ‘KevinCol’

SELECT DateDataType FROM Table

Dell Axim XScale PXA250 (400mhz)

Storage Media?

Varying perf for different storage mediaRAM, SD, CF, IPSM (NOR), Toshiba NAND

SD is faster than MMChttp://www.dpreview.com/reviews/minoltadimagex/page7.asp

CF perf varies widelyhttp://www.dpreview.com/articles/mediacompare/default.asp?sort=fwread

142 155 144209

314348

0

100

200

300

400

Tim

e In

Sec

onds

SELECT UPDATE

Various storage medias

RAM

SD

IPSM

Delta’s from CPU’s and OS versions

255

188

148

246

11288

155

44 33

050

100150200250300

Tim

e i

n s

ec

on

ds

RDA Pull Update SELECTData-Reader

Intermec 700 64mb RAM, db in RAM

XScale 250PPC 02

XScale 255PPC 02

XScale 255PPC 03

Delta’s from CPU’s and OS versions

272

191

102

48

0

100

200

300

Tim

e in

se

cond

s

Update SELECTData-Reader

HTC Phone Edition 32mb RAM SA1100, db on SD

PPC 02

PPC 03

Misc

Database compactCorrect any potential corruption

Tables in contiguous orderIf PK/Unique index present, ordered by index

Refreshes statistics

QP StatisticsAllows for more accurate cost estimate

SupportsIndex cardinality (for selectivity of equality predicate)

Doesn’t supportMin / Max

Histogram

Change temp db location to durable media for large transactions

SQL CE QP doesn’t distinguish between RAM and durable media

Performance white papershttp://www.microsoft.com/sql/ce/

Nabisco Case Study And Demo

1500 Delivery 1500 Delivery TrucksTrucks

107 Distribution 107 Distribution CentersCenters

Web ServerWeb Server SQL Server 2000 SQL Server 2000 FailoverFailover

802.11b RF802.11b RF

IBM DB/2IBM DB/2

XMLXML

SAP Business ObjectsSAP Business Objects

Ruggedized Ruggedized PocketPC DevicePocketPC Device

with SQL Server CEwith SQL Server CE

Nabisco Deployment

Rolled custom CAB fileOriginally used eMVT wizard

Load CAB file and database on CF cardAllows for cold boot recovery in field

Used version flag in publicationIf version changes, then it is replicated down to the device

This flags download of new CAB file from Internet Information Server and updates latest version

Nabisco Scaling

One 4-Proc SQL Server 2K

One 4-Proc IIS Win2K

Start of day averages are ~15 seconds

End of day averages are ~15 seconds

Tech-Ed Europe 2002 Mobile ComNet

1500 PocketPC 2002 Wi-Fi devices

C# application with SSCE v2.0 Schedule maintenance

Attendees download all sessions (synched with ComNet devices) to see session changes and change personal itinerary

Used SQL Server SP3 replication fixesLocking reduced by changing to dynamic queries from SP’s

Used NTLB for Internet Information Server

SyncCombination of merge and RDA

One article in publication

Four tables for RDA Pulls

DataGridsPaging

RepeatersMultiple sessions

Timely links

Key FeaturesScheduling

VLAN14 ExboNet Net

VLAN13 UntrustVLAN12

VLAN11 COMMSNET

Infrastructureat Event Site

WEB11for Local

Downloads

InternetSite

Supplid Router

AD11AD / DNS/ WINS 1

AD12AD / DNS/ WINS 2

Delegates PCs

MSEVSGAT11DHCP, ISA andVPN (Router)

Speakerscomputer

Wireless LAN

PDA withWireless LAN

Candidates Laptopswith Wireless LAN

MSEVSGAT12DHCP, ISA andVPN (Router)

2 x IP Address T.B.A1 x Internet OutGoing

1 x For VPN Routed Connection

VLAN 21 VLAN 21

RedundentLinks

Session Room Back Office

Wireless for Officeand speaker network

Speaker orOffice network LAPTOPS

or PC

Internet

GAT13DHCP, NAT

Buddy ServerIIS

Buddy ServerSQL

Buddy System

InternetLINX

Heart beatLAN

SQL01SQL Cluster

SQL02SQL ClusterDisk

Heart beatLAN

Public Addresses

VLAN 3

VLAN 2

ISA05Secure VPN

Server 1(Online)

ISA03 forHTTP

Publishingfor Production

Sites

AD01AD / DNS / WINS 1

AD02AD / DNS/ WINS 2

WEB03service.aspx

MOM01MOM

Service

Infrastructureat London ASP

Site

PRODSQL01Production SQL

Cluster

PRODWEB02Production WEB

WSRV01.asmx WEB

Service

ISA01 forHTTP

Publishingfor Events Sites

Media ServerMedia Server

ISA02 forHTTP

Publishingfor Events Sites

ISA04 forHTTP

Publishingfor Production

Sites

ISA06Secure VPN

Server 7(Backup)

PRODWEB01Production WEB

MOM02MOM SQL Server

WEB04service.aspx

PRODSQL02Production SQL

Cluster

Disk

NLB

WEB01service.aspx

WEB02service.aspx

MAN01Management LAN

NLB

WSRV02.asmx WEB

Service

The Hardware

New HP supplied for COMMS NetServer Use Model Number of

ServersProcessors and Speed

Memory Disk

ISA DL380 6 2 x 1Ghz 1Gb RAM 4 x 18Gbs

AD DL380 4 2 x 1Ghz 1Gb RAM 4 x 18Gbs

WEB DL360 8 2 x 1Ghz 1Gb RAM 2 x 18Gbs

SQL Clusters DL580 4 4 x 700mhz 2Gb RAM 4 x 18Gbs

Fibre Channel disk

2 N/A N/A 32 x 18Gb disks

SQL DL380 1 2 x 1Ghz 1Gb RAM 4 x 18Gbs

AD DL380 2 2 x 1Ghz 1Gb RAM 4 x 18Gbs

WEB DL360 1 2 x 1Ghz 1Gb RAM 2 x 18Gbs

GATEWAY DL580 2 4 x 700mhz 2Gb RAM 4 x 18Gbs

Performance

Average Sync on device (RDA and merge) ~20 seconds

Average IIS CPU utilization ~5%

Merge Agent duration <1 sec min, 42 sec max, average <1 sec

Scaling Merge Replication

SET @run_much_faster = True

Upgrade to SQL Server 2K SP3 and associated QFE 798 (kb 810185)

Significant performance improvements on concurrent sync PLUS SQL Slammer fix

Defrag indexes on SQL Server

DBCC SHOWCONTIG (MSMerge_Contents) WITH TABLERESULTS, ALL_INDEXES

DBCC INDEXDEFRAG (Driver, MSMerge_Contents, 1-4)

Optimizes join between MSMerge_Contents and base tables

Avg. Bytes free per page should be low

Avg. Page density should be high

Scaling Merge Replication

Use of TempDB is high, don’t use autogrow

Put data files and log files on different disks

Separate out NTEXT/IMAGE data-types from base table

Prevents large unchanged columns from synch

Investigate using Upload_Only optionNabisco utilized this option

Investigate mixing RDA with Merge replication

Investigate lowering retention period from default of 14 days

Investigate using re-publishers

Investigate using IIS Web farm with NTLB

Scaling Merge ReplicationChange to row level tracking

Decreases # of rows in MSMerge_ContentsChange per article “@column_tracking = N‘false‘”

Increase generations per batchDecreases number of times looping through Setup_Belongs

Reduce retention period from default of 14 days@retention=nDownside is potential expired subscriptions

Split large batches of changes by calling sp_MSmakegenerationKeep_partition_changes=True

Avoids sending deletes to all subscribers on partition changesFiltering

Avoid complex join filter or subset filter statementsIndex columns involved in filtersSet the join unique key property = true when joining between PK-FKDe-Normalize table to have filtered column on each table to reduce complex join filters

Publicationshttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsql2k/html/sql_replmergepartitioned.asphttp://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/maintain/Optimize/MERGPERF.asp

Questions?

Community Resources

Community Resourceshttp://www.microsoft.com/communities/default.mspx

Most Valuable Professional (MVP)http://www.mvp.support.microsoft.com/

NewsgroupsConverse online with Microsoft Newsgroups, including Worldwidehttp://www.microsoft.com/communities/newsgroups/default.mspx

User GroupsMeet and learn with your peershttp://www.microsoft.com/communities/usergroups/default.mspx

evaluationsevaluations

© 2003 Microsoft Corporation. All rights reserved.© 2003 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.