SQL Server Optimization for DevelopersANIL DESAI (ANIL@ANILDESAI.NET | HTTP://ANILDESAI.NET)

AUSTIN .NET USER’S GROUP, 04/14/2014

Presentation Overview

Database Performance Goals and Challenges

Monitoring and Optimizing Performance Understanding indexes

SQL Profiler and Database Engine Tuning Advisor

Tuning Database Queries Understanding the Query Optimizer and Execution Plans

Seeing the effects of indexes

Application Design Best Practices

Best Practices: Optimize for

real-world workloads

Monitor/review performance regularly

Focus on specific issues

Performance Monitoring Process

Establish a

baseline

Identify bottlenec

ks

Make one change at

a time

Measure performa

nce

Repeat (if

desired)

Monitoring and Troubleshooting Scenarios

Server-Level Issues

• Users are reporting slow performance• Intermittent transaction timeouts• “The server/application seem sluggish”

Application-Specific Issues

• “Application A is running more slowly than usual”• “The End-of-Month report is taking too long to

run.”

Other Issues• Ad-hoc reports are running slowly• CPU, memory, disk, or network alerts are being

generated

Which queries are taking the longest

to run?

Which queries are using the most

system resources?

Why does database/web site access slow down

during particular times or during certain

patterns of activity?

Is indexing configured optimally for my live

(production) database workloads?

How will my development updates

affect production performance?

Which are the most cost-effective upgrades I can

make to improve performance?

How can I optimize a

specific query?

Common Datbaase Questions

Database Performance Tools

System/OS

Performance Monitor

Resource Monitor

Task Manager

SQL Server

SQL Server Management Studio

SQL Profiler / SQL Trace

Database Engine Tuning Advisor

Dynamic Management Views

(DMVs)

Query-Level

Database Engine Tuning

Advisor

Query Execution Plans

Database Design Issues

Transaction processing (OLTP) Favors normalized schema

Many tables, each with fewer columns

Optimized for write (transactional) activity

Reporting and Analysis Centralized, consistent storage of required data

Favored by denormalized schema

Fewer tables with many columns in each

Data is aggregated from multiple sources into a data mart or data warehouse

May store aggregates in warehouse

Understanding Indexes

Index types Clustered Index

Non-Clustered Indexes

Columnstore indexes

Indexing strategies Goal is ideal index coverage

Index maintenance can slow-down write operations (Insert, Update, Delete)

Referential Integrity Primary Key (default = clustered index)

Foreign Key references

Constraints

Statistics (manual vs. automatic)

General Index Tuning Best Practices

Make tuning a part of your development process Dev: Use synthetic workloads and test cases

Test: Use real-world databases, whenever possible

Production: Capture real usage statistics for analysis

Collect a representative workload, whenever possible

Consider all applications and workloads when tuning a database

Use naming conventions for indexes and related objects

Use query hints sparingly (NOLOCK)

Purpose / Features: GUI for managing SQL Trace

Monitor important events

Capture performance data / resource usage

Replaying of workloads / transactions

Identifying performance bottlenecks

Correlation of data with System Monitor

Workloads for Database Tuning Advisor

Examples: Generate a list of the 100 slowest queries

Monitor all failed logins (Security)

Using SQL Profiler

Database Engine Tuning Advisor

Automatic workload analysis for Physical Design Structures (PDS) Data Source:

File (Profiler Trace or .SQL files)

Table

Plan Cache

Tuning Options Keep existing PDS

Advanced Features: Partitioning, indexed views, etc.

Demo: Optimizing Indexes

Generate sample queries / tables View query execution plans

View the effects of indexes on common queries

Capture Performance Data with SQL Profiler SQL Profiler traces, events, and filters

Using SQL Load Generator to generate database load

Capturing and storing

Analyzing and optimizing with Database Tuning Advisor Analyzing index usage reports

Saving and applying index recommendations

Tuning Individual Queries

Query Analyzer Features Execution Plan (estimated and actual)

Include Client Statistics (multiple trials)

Analyze in Database Engine Tuning Advisor (single query)

Trace query in SQL Profiler (single query)

Keep query logic close to the database Filter returned data at the database layer

Minimize the size of result sets

Minimize round-trips to the server

Use standard (inner) joins, where possible

Consider strategic denormalization for core sets of data

Query Optimizer Details

Goal: Find the most efficient method to return the data Come up with a plan quickly

Minimize CPU, memory, and I/O requirements

Use statistics and index details to improve plans

Query plan caching

Relational engine vs. storage engine

Execution Plan output Save as .sqlplan file for later analysis

Output in graphical, text, and XML formats

Can store and export plans using SQL Profiler (ShowPlan XML event)

Can use query hints

Understanding Execution Plans

Optimizing individual queries Rewrite query logic

Use other objects (views, stored, procedures, etc.)

Strategic demoralization

Data Retrieval: Table scan, index seek/scan

Index Usage Covering indexes

Join conditions

Execution Plan Example

Execution Plans in Windows Azure

Client Statistics Example

Application Design Best Practices

Create an abstraction layer between business and database objects ADO.NET

Microsoft Enterprise Library

NHibernate

Entity Framework

Use caching wherever possible Server-side (web services)

Application-level (middle tier)

Client-side (desktop or mobile apps)

Minimize transaction times

Dev Best Practices: Application and Data Architecture

•Standards-based communications•Loosely-coupled tiers and componentsArchitecture

•Test harness•Unit tests that use data•Query performance monitoring

Development / Testing

•Windows PerfMon Counters•Instrumentation and Logging•Alerts / pro-active corrections•Auto-scaling

Performance and Monitoring

•Error and event handling•Transaction retry (random interval); Deadlock management•User notifications (responsive UI / cancel and retry options)Reliability

Windows Azure and Cloud Databases

Practical cloud benefits Data redundancy and geographic distribution

Lower management overhead

Potential issues Keeping data close to applications and services

Data synchronization

Network performance issues

Data security, legal issues, and regulatory compliance

Determine where/how to use cloud-based services SaaS vs. PaaS vs. IaaS

Azure Database Services

SQL Azure Database Cost-effective, managed database instances

Can be managed with standard tools (Visual Studio and SSMS)

Some limitations (CLR, Mirroring, Partitioning, Replication, Extended SP’s)

Other Services Azure Virtual Machines (SQL Server templates)

Azure Web Sites (with gallery templates)

Azure HDInsight, Cache Service

Azure Backup and Recovery Manager

SQL Azure Reporting

Network, Active Directory, Service Bus, etc.

Managing SQL Azure Instances

ORM Considerations

General issues Development efficiency vs. hardware/software efficiency

Latency, query inefficiency (outer joins), platform-specific optimizations

Frequency and number of server round-trips

ORM-generated queries can be inefficient Difficult to tune or modify individual queries

Potential Solutions Make sure entity relationships are correct

Can use views or stored procedures to improve performance in some cases

Bypass the ORM for some types of operations

New Features in SQL Server 2014

Memory-optimized tables (In-Memory OLTP)

Buffer Pool Extension (for SSD usage)

Delayed durability Async log writes can result in data loss

Enable at database-level; use with BEGIN ATOMIC … COMMIT

Resource Governor storage I/O limits

Updateable Clustered ColumnStore indexes Primarily for data warehousing; supports data index compression

Azure storage for SQL Server data/log files

Backup to Azure; Backup encryption

Dev Best Practices: Managing Data

Large UPDATE or DELETE operations: Use loops to minimize locking and transaction log growth

Large INSERT operations Disable indexes and triggers (if present)

Use BULK INSERT, bcp, SSIS, or DTS

Change transaction isolation level (if appropriate)

Change recovery model

Use SQL to generate SQL Example: INSERT statements

Schedule or delay non-critical operations

Dev Best Practices: Schema Changes

Generate Scripts Script specific objects using SQL Server Management Studio

Script the entire database using Generate Scripts

Can include schema and/or data

Schema changes Use ALTER commands when possible

Drop and recreate objects, as needed

Make all scripts re-runnable

Check before and after state of all objects

Dev Best Practices: Performance Testing

Build performance testing/optimization into the dev process Develop load tests or test “harnesses”

Using synthetic load generation tools

Use representative test data

Consider caching effects: Index maintenance (fragmentation)

DBCC DropCleanBuffers

DBCC FreeProcCache

Advanced Performance Approaches

Database Federations Vertical and horizontal data partitioning

Cross-Server queries Use Linked Servers to query across databases

Potential performance issues

Data compression (row- or page-level)

Resource governor

SQL Server Analysis Services (SSAS) Pre-aggregation for performance

Dependent on a denormalized schema (optimized for reporting)

Links and References

Presenter: http://AnilDesai.net | Anil@AnilDesai.net

Presentation slides and sample code

Microsoft TechNet Virtual Labs

Sample Databases AdventureWorks Sample Databases (CodePlex)

Microsoft Contoso BI Demo Dataset

Database-related tools SQL Load Generator by David Darden (CodePlex)

Glimpse

Red Gate Software

Spotlight

Summary and Conclusion