optimising queries - series 1 query optimiser architecture

www.dageop.com

Optimising Queries

®

OQ-01 Query Optimiser Architecture

DR. SUBRAMANI PARAMASIVAM

(MANI)

http://www.yoursqlman.com/

About me

Dr. SubraMANI ParamasivamPhD., MCT, MCSE, MCITP, MCP, MCTS, MCSACEO, Principal Consultant & Trainer @ DAGEOP (UK) Email: [email protected] Blog: http://dataap.org/blog

Follow Us

https://www.facebook.com/pages/YOUR-SQL-MAN-LTD/

http://www.youtube.com/user/YourSQLMAN

https://twitter.com/dageop

https://uk.linkedin.com/in/dageop

Proud Sponsor

• SQLBits• SQL Saturdays• MCT Summit• SQL Server Geeks

Summit• Data Awareness

Programme• Dageop’s Data Day

®

www.DataAP.org

SPEAKER

Contents

• OQ-01 Query Optimiser Architecture• Phases

• Strategies

• Data access plans

• Auto-parameterisation

• Avoiding recompilation of queries

www.dageop.comOptimizing Queries


OQ-01 Query Optimiser Architecture



Optimising Queries



Optimising Queries

• Optimising Query is very important task to maintain the server resources.

• Most of the time server is hit by performance issues.

• Clean the SQL Server Cache.

• Well written T-SQL Queries will help to optimise.



Query Optimiser Architecture



Introduction to Query Optimiser Architecture

2 Major Components in SQL Server• SQL Server Database Engine

• Reading data between disk and memory

•Relational Engine (Query Processor)• Accepts queries, analyse & executes the plan.



Introduction to Query Optimiser Architecture

• Analyse Execution Plans (Cannot consider every plan)

• Estimate the cost of each plan

• Select the plan with low cost


Plan1

Plan2

Plan3

Plan4

Plan5

Plan6

Plan7

Plan1 – 80%

Plan2 – 60%

Plan3 – 10%

Plan5 – 90%

Plan6 – 70%

Plan3 – 10%

Analyse Estimate Select

COST X 2Plan & itself


BETTER UNDERSTANDING of Query Optimiser is a must

for both DBA & DEVELOPERS.

Main Challenges are

Cardinality & Cost Estimations



Query Optimiser

• 1st Part - Searching or enumerating candidate plans

• 2nd Part - Estimates the cost of each physical operator in that plan (I/O, CPU, and memory)

• This cost estimation depends mostly on the algorithm used by the physical operator, as well as the estimated number of records that will need to be processed (Cardinality Estimation).

PRIMARY WAY to interact with Query Optimizer is through EXECUTION PLANS



Query Optimiser

• Execution Plan• Trees consisting of a number of physical operators

• Physical operators• Index Scan

• Hash Aggregate

• result operator (root element in the plan)

• Nested Loops Join

• Merge Join

• Hash Join

• Algorithms

• Can be saved as .sqlplan (Can be viewed in SSMS)



Query Optimiser• Operators

• Logical• Relational algebraic operation• Conceptually describes what operation needs to be performed

• Physical• Implement the operation described by logical operators• Access columns, rows, index, table, views, calculations, aggregations, data integrity

checks • 3 methods

• Init() – Initializes a connection• GetNext() – 0 to many calls to get data• Close() – Some cleanup and close the connection.

• Query Optimizer chooses efficient physical operator based on logical.



Query Optimiser


Logical Physical Logical & PhysicalBitmap Create Assert AggregateBranch Repartition Bitmap Clustered Index ScanCache Clustered Index Delete Clustered Index ScanDistinct Clustered Index Insert Clustered Index UpdateDistinct Sort Clustered Index Merge CollapseDistribute Streams Hash Match Compute ScalarEager Spool Merge Join ConcatenationFlow Distinct Nested Loops Cursor

Full Outer Join Nonclustered Index Delete Inserted ScanGather Streams Index Insert Log Row ScanInner Join Index Spool Merge IntervalInsert Nonclustered Index Update Index ScanLazy Spool Online Index Insert Index SeekLeft Anti Semi Join Parallelism Parameter Table ScanLeft Outer Join RID Lookup Remote DeleteLeft Semi Join Stream Aggregate Remote Index ScanPartial Aggregate Table Delete Remote Index SeekRepartition Streams Table Insert Remote InsertRight Anti Semi Join Table Merge Remote QueryRight Outer Join Table Spool Remote ScanRight Semi Join Table Update Remote UpdateRow Count Spool SegmentSegment Repartition SequenceUnion Sequence ProjectUpdate Sort

SplitSwitchTable ScanTable-valued FunctionTopWindow Spool


Query Optimiser

• Invalid Plans• Removal of an index

• Removal of a constraint,

• Significant changes made to the contents of the database

• SQL Server under memory pressure

• Changing configuration options - max degree of parallelism

• Discard the plan cache and new optimization generated



Query Optimiser

• Highly complex pieces of software

• Even 30 years of research, still has technical challenges.

HINTS

• override the operations of the Query Optimizer

• caution and only as a last option

• Influence Query Optimizer to use certain plans



Query Optimiser - Interaction


Query Optimizer

Execution Plans

Trees, Algorithm, Physical operator

ACTUAL ESTIMATED

Graphics, Text, XML format


DEMO



Phases



Phases

• Once we execute the SQL Statement, it will follow certain procedures from SQL Statement to Query Result


Parsing Query Compilation

Query Optimization

Query Execution


Phases

• If the Query is already in the plan cache then it will not generate any new execution plan for that query.

• Parsing:• The Query’s syntax will be validated and query is transformed in a tree.

Checks the objects for its existence which are used in the query.

• After the query validated, the final tree is formed.

• Query Compilation:• Query Tree will be compiled here.



Phases

• Query Optimization• The query optimizer takes as input the compiled query tree generated in the

previous step and investigates several access strategies before it decides how to process the given query.

• It will analyse the query to find most efficient execution plan.

• Query Execution• After the execution plan is generated, it is permanently stored and executed• Note: For Some statements, parsing and optimization can be avoided if the

database engines know that there is only one viable plan. This is called trivial plan optimization.



Strategies



Strategies

• As a DBA, we need to choose right strategy for each SQL statement like scripts should be well written before executing the SQL statement.

• Statistics will be used to understand the performance of each query.

• Enable Set Statistics IO before query run. It will display the following information• How many scans were performed

• How many logical reads (reads in cache) were performed

• How many physical reads (reads on disk) were performed

• How many pages were placed in the cache in anticipation of future reads (read-ahead reads)



Strategies

• We can understand the query performance from statistics IO, If the query is good then the logical reads of the query result should be lower and few physical reads and scans.

• Enable Set Statistics Time, it will display the execution time of the query. It purely depends on the total activity of the server.

• Enable show execution plan, to see how the query performed.

• These are things will helpful for choosing the right strategy.




StrategiesChoose the right strategies based on below information from SQL Server


DEMO



Data Access Plans



Data Access Plans

• If we need to optimize the data access plans then we need to start from creating files for database, Index on the tables and T SQL Statements.

• Steps:• Organize the file groups and files

• Apply partitioning in big tables

• Create the appropriate indexes/covering indexes

• Defragment the indexes

• Identify inefficient TSQL

• Diagnose performance problems



Data Access Plans

• Organize the file groups and files• Initially two files will be created while created a database (.mdf & .ldf). • .mdf file : Primary data file for each database. All system objects will be stored in this

file, including the user defined objects if .ndf file is not there.• .ndf file: These are secondary data files, these are optional. These files will have user

created objects.• .ldf file: These are the transaction log files. This could be one or more files for single

database.

• File Group:• Database files are logically grouped for better performance and improved

administration on large databases. When a new SQL Server database is created, the primary file group is created and the primary data file is included in the primary file group. Also, the primary group is marked as the default group.



Data Access Plans

• To obtain the performance of the data access, Primary file group must be separate and it should be only for system objects.

• Need to create one more file called secondary data file for user defined objects.

• Separating the system objects will improve the performance enhance the ability to access tables in cases of serious data failures.

• For frequently accessed tables containing indexes, put the tables and the indexes in separate file groups. This would enable reading the index and table data faster



Data Access Plans

• Apply partitioning in big tables• Table partitioning is nothing but splitting the large table into multiple small

tables so that queries have to scan less amount for data retrieving.

• Consider partitioning big fat tables into different file groups where each file inside the file group is spread into separate physical disks (so that the table spans across different files in different physical disks). This would enable the database engine to read/write data operations faster.

• Partitioning is very much needed for history tables.

• 2 types• Physical

• Logical



Data Access Plans

• Create the appropriate indexes/covering indexes

• Create Non-Clustered index on frequently used columns

• Create index on column which is used for joining the tables

• Index for foreign key columns

• Create covering index for particular columns which are using frequently.

• Defragment the indexes

• Once index has created , it should maintain properly to avoid defragmentation.

• Maintaining the index will lead to performance gain.



Data Access Plans

• Identify inefficient TSQL• Don’t use * in the select statements. Mention column names while retrieving

the data. It will improve the performance of the query

• Avoid Deadlocks between two objects.

• Diagnose the performance issue• SQL Server has many tools to Monitor and diagnose the issues.

• Accessing the data will be more easier.



DEMO



Auto-Parameterisation



Auto-Parameterisation

• SQL Server Query Optimizer might decide to parameterize some of the queries.

• In this case the specific parameter will not make any impact on the execution plan. It will return the same execution plan.

• In SQL Server 2005 forced parameterization has been introduced and is disabled by default and can be enabled in database level.

• To differentiate from forced parameterization, auto-parameterization is also referred as simple parameterization.



DEMO



Avoiding Recompilation of Queries




• In SQL Server 2005 recompiles the stored procedures, only the statement that causes recompilation is compiled, rather than the entire procedure.

• Recompilation will occur in following ways,• On Schema change of objects

• On Change of the SET options

• On statistics change of tables.




• On Schema Change of Objects• Adding and dropping column, constraints, index, indexed view and trigger.

• On change of the SET options• When executing the stored procedure, the compiled plan is created and it will store

the environment setting of a connection (SET OPTION) .• Recompilation will occur, if the stored procedure run on different environment and

with different SET option then it will not use the existing plan which it is created first time.

• On statistics change of tables• SQL server maintains a modification counter for each table and index.• If the counter values exceed the defined threshold, the previously create compiled

plans is considered stale plan and new plan will be created.




• Temporary table modification counter threshold is 6. Stored procedure will be recompiled when stored procedure create a temp table insert 6 or more rows into this table.

• For Permanent table the counter threshold is 500.

• We can increase the temp table counter threshold to 500 as same as permanent table.

• Use table variable instead of Temporary table.



DEMO



Review

Query Optimiser ArchitecturePhases

Strategies

Data access plans

Auto-parameterisation

Avoiding recompilation of queries



Q & A



®

www.dageop.com

optimising queries - series 1 query optimiser architecture

Data & Analytics