seas – 2006 sql server 2005 vldb management & operation

SEAS – 2006SEAS – 2006

SQL Server 2005SQL Server 2005VLDB Management & VLDB Management & OperationOperation

SEAS06 - VLDB Management & Operation 2

The challenge of Very Large DatabasesThe challenge of Very Large Databases Take the VLDB challenge and break into smaller Take the VLDB challenge and break into smaller

manageable designsmanageable designs SQL Server 2005 VLDB technologiesSQL Server 2005 VLDB technologies

Table partitioningTable partitioning Online operationsOnline operations Improved performanceImproved performance Improved concurrencyImproved concurrency Piecemeal managementPiecemeal management

Capacity PlanningCapacity Planning StorageStorage

DisksDisks TempdbTempdb TransactionTransaction LogsLogs

MemoryMemory CPUCPU

VLDB Lessons LearnedVLDB Lessons Learned

Agenda Agenda


Very Large DatabasesVery Large DatabasesTrendsTrends

1 TB

2 TB

5 TB

10 TBDrive capacity doubles every 12-18 months

Drive cost drops

Database size doubles every year for the same cost

Data retention driven by• Regulatory requirements• Business analysis


WinterCorp Study - SQL Server/Windows Growth WinterCorp Study - SQL Server/Windows Growth projectionsprojections

20012001 20032003 20052005 20072007 20092009 20112011

Minimum entry Minimum entry criteria criteria for for WindowsWindows

100 100 GGBB

500 GB500 GB 1 TB1 TB 2TB2TB 4TB4TB 8TB8TB

Total number of Total number of total total entries for entries for all all databasesdatabases

168168 141141 175175 ?????? ?????? ??????

Number of entries, Number of entries, Windows Windows onlyonly

2020 3636 6060 9595 150150 250250

Number of entries, Number of entries, SQL SQL ServerServer

1010 2424 4343 8080 140140 240240

Number of entries, Number of entries, SQL SQL Server, > Server, > 1 TB1 TB

00 1515 4343 8080 140140 240240

Largest SQL Largest SQL Server Server InstanceInstance

800 800 GGBB

5TB5TB 20TB20TB 100TB +100TB + 500TB +500TB + 2000TB 2000TB ++

Projections


Challenge of Very Large Databases Challenge of Very Large Databases

Large scale operations including…Large scale operations including… Index creation and maintenanceIndex creation and maintenance Bulk insertion of new dataBulk insertion of new data Bulk deletion of old dataBulk deletion of old data Concurrent reporting and updatesConcurrent reporting and updates Backup and restoreBackup and restore Disaster recoveryDisaster recovery Detecting and recovering from media errorsDetecting and recovering from media errors Tuning and PerformanceTuning and Performance

…while maintaining availability as dictated by business requirements.







MemoryMemory CPUCPU


Agenda Agenda


Design Philosophy VLDBDesign Philosophy VLDB

Don’t think of it as Large complex Don’t think of it as Large complex database, think of it as smaller database, think of it as smaller manageable componentsmanageable components

Partitioning data into smaller Partitioning data into smaller manageable sizes is your best friendmanageable sizes is your best friend Data Partitioning vs Function Data Partitioning vs Function

PartitioningPartitioning Methods for data partitioningMethods for data partitioning

Database PartitioningDatabase Partitioning Instance PartitioningInstance Partitioning SQL Server PartitioningSQL Server Partitioning


Data separation: Data Partitioning Data separation: Data Partitioning vs Function Partitioningvs Function Partitioning Examples of data partitioning by dataExamples of data partitioning by data

Data broken up by States in the US: Data broken up by States in the US: Alabama, Arkansas, Alaska…..Alabama, Arkansas, Alaska…..

Data broken up by months in a year: Jan, Data broken up by months in a year: Jan, Feb, Mar……Feb, Mar……

Examples of data partitioning by Examples of data partitioning by functionfunction Data separated by business users: sales Data separated by business users: sales

data, consulting data, competitive data….data, consulting data, competitive data…. Data separated by business function: Data separated by business function:

manufacturing data, design data, manufacturing data, design data, inventory data…….inventory data…….


Example #1: VLDB Case Study Example #1: VLDB Case Study Function PartitioningFunction Partitioning

Manage 13TBs of dataManage 13TBs of data Manage large files from several Manage large files from several

external customers / dayexternal customers / day Load millions of rows from multiple files Load millions of rows from multiple files

and run batch reporting at the same and run batch reporting at the same timetime

How do I manage different SLAs from How do I manage different SLAs from different customers with same system.different customers with same system.


Example #1: VLDB case Example #1: VLDB case study – Function Partitioningstudy – Function Partitioning

Account Function Tracking DB

Function X Tables

Function Y Tables

Function Z Tables

Common Database

Common Function Procedures - Data

Individual AccountDatabases

Daily Customer Source files


Function Partitioning is scale Function Partitioning is scale out out Partitioning is a VERY good ideaPartitioning is a VERY good idea

Partitioning by business function is very logicalPartitioning by business function is very logical Application must developed to partitioned data Application must developed to partitioned data

versus using database technology to access versus using database technology to access partitioned data. NOT DIFFICULTpartitioned data. NOT DIFFICULT

Putting all the data in one single database Putting all the data in one single database doesn’t make you a hero.doesn’t make you a hero.

No matter how you look at it 13TB of data is No matter how you look at it 13TB of data is 13TB of data used to solve a business 13TB of data used to solve a business problem.problem.


Example #2: VLDB Case Study Example #2: VLDB Case Study data partitioningdata partitioning

Loads into SQL Server 350 million call Loads into SQL Server 350 million call detail records / day. Bulk loadeddetail records / day. Bulk loaded

Must keep a historyMust keep a history Reporting done all day long based off Reporting done all day long based off

daily, weekly, monthly quarterly and daily, weekly, monthly quarterly and yearly intervalsyearly intervals

Very partitionable by period of timeVery partitionable by period of time 5-7 TB database5-7 TB database This was based off a prototype, project This was based off a prototype, project

being developed.being developed.


Example #2: VLDB case study Example #2: VLDB case study data partitioiningdata partitioining

3 types of databases3 types of databases Type 1 contains 24 - 1 hour tablesType 1 contains 24 - 1 hour tables Type 2 contains 7 - 1 day tablesType 2 contains 7 - 1 day tables Type 3 contains 7 – 1 week tablesType 3 contains 7 – 1 week tables

Online consolidation was much more difficult, so Online consolidation was much more difficult, so all 3 are built in parallel. This is good design if all 3 are built in parallel. This is good design if Hardware has head roomHardware has head room

Data is loaded without indexes and then Data is loaded without indexes and then indexedindexed

Data made available to users once indexedData made available to users once indexed Yesterday, Last Week and Last 7 weeks are Yesterday, Last Week and Last 7 weeks are

available while Today and This week are being available while Today and This week are being loaded.loaded.


Example #2: Example #2: VLDB Design if using Single serverVLDB Design if using Single server

TodayTodayTodayTodayYesterdayYesterday

YesterdayYesterdayThis weekThis week

Last weekLast week

This weekThis week

Last weekLast week

Week n-7Week n-7

PViewPView

ViewViewSQL ServerSQL Server


Example #2: VLDB Example #2: VLDB Design if Design if changed from 1 Server to 4 changed from 1 Server to 4 ServersServers



Last weekLast week

This weekThis week

Last weekLast week

Week n-7Week n-7


Example: #2: Example: #2: Design is use 7 Design is use 7 server partitioningserver partitioning



Last weekLast week

This weekThis week

Last weekLast week

TodayTodayYesterdayYesterday


Scalable Shared DatabaseScalable Shared DatabaseScalable Shared DatabaseScalable Shared Database Read-only copy of a Data Read-only copy of a Data

Warehouse database on a SAN can Warehouse database on a SAN can be shared by up to eight SQL query be shared by up to eight SQL query serversservers Network Load Balancing is possibleNetwork Load Balancing is possible Inexpensive scale-out solution as Inexpensive scale-out solution as

alternative to scaled-up server alternative to scaled-up server Easily support a large or growing group Easily support a large or growing group

of query usersof query users SANs can create a shared read-only SANs can create a shared read-only

copy of the database instantly using copy of the database instantly using SAN Snapshot technologySAN Snapshot technology Copy on write technologyCopy on write technology Refresh the snapshot after new data is Refresh the snapshot after new data is

loadedloaded


Example #3: VLDB Scale out partitioningExample #3: VLDB Scale out partitioning







MemoryMemory CPUCPU


Agenda Agenda


VLDB SQL Server Technologies:VLDB SQL Server Technologies:Table PartitioningTable Partitioning

Designed for:Designed for: Easy management of very large tables Easy management of very large tables

and indexesand indexes Maintain high availabilityMaintain high availability Make effective use of large machines Make effective use of large machines

(16, 32, and more CPUs) for any type (16, 32, and more CPUs) for any type of workloadof workload

Provide performance options for Provide performance options for maintenancemaintenance


Table Partitioning: Common ScenarioTable Partitioning: Common Scenario“Sliding Window”“Sliding Window”

New data is added continually to a very New data is added continually to a very large database (Daily, Weekly, Monthly)large database (Daily, Weekly, Monthly) Data must be scrubbed, indexed, etc.Data must be scrubbed, indexed, etc.

As data ages, it is modified less frequentlyAs data ages, it is modified less frequently Eventually, it may be discarded Eventually, it may be discarded

TimeHotRead-MostlyRead-OnlyObsolete

Sliding Window


LocatingLocating AA RowRow

Key (partitioning column)Key (partitioning column)

PartitionPartitionFunctionFunction

Partition Partition SchemeScheme

FileGroup 1FileGroup 1 FileGroup 2FileGroup 2 FileGroup 1FileGroup 1


Aligned Index Aligned Index One-to-one partition correspondenceOne-to-one partition correspondence

Index PartitionIndex Partition

Table PartitionTable Partition


VLDB SQL Server Technologies:VLDB SQL Server Technologies:Online OperationsOnline Operations

Index creation Index creation Index maintenanceIndex maintenance Online restoreOnline restore Availability during rollback recoveryAvailability during rollback recovery


Online Index OperationsOnline Index OperationsSQL Server 2005SQL Server 2005

Introduced as WITH (ONLINE=ON)Introduced as WITH (ONLINE=ON) CREATE INDEXCREATE INDEX DROP INDEXDROP INDEX ALTER INDEX REBUILD (was DBCC ALTER INDEX REBUILD (was DBCC

DBREINDEX)DBREINDEX) ALTER INDEX add or drop constraint (primary ALTER INDEX add or drop constraint (primary

key, unique)key, unique) ONLINE is supported on partitioned tables ONLINE is supported on partitioned tables

but it is not supported for ALTER INDEX but it is not supported for ALTER INDEX REBUILD on single partitionREBUILD on single partition


VLDB SQL Server Technologies: VLDB SQL Server Technologies: Online RestoreOnline Restore

Microsoft® SQL ServerMicrosoft® SQL Server™™ 2000 2000 Database is not available during restoreDatabase is not available during restore

SQL Server 2005SQL Server 2005 Database remains onlineDatabase remains online

Only data being restored is unavailableOnly data being restored is unavailable Works on Enterprise Edition “by default”, on Works on Enterprise Edition “by default”, on

non-enterprise the whole database is always non-enterprise the whole database is always offlineoffline

OptionsOptions File / filegroup restoreFile / filegroup restore Damaged page tracking and restoreDamaged page tracking and restore


VLDB SQL Server Technologies: VLDB SQL Server Technologies: Improved PerformanceImproved Performance

Bulk insertion and deletion of dataBulk insertion and deletion of data Most improvement through partitioning (ALTER TABLE Most improvement through partitioning (ALTER TABLE

SWITCH mentioned before)SWITCH mentioned before) File initializationFile initialization

Skip zeroing out the file(s)Skip zeroing out the file(s) Permission grantable only by WindowsPermission grantable only by Windows®®

administratoradministrator SE_MANAGE_VOLUME_NAME special privilegeSE_MANAGE_VOLUME_NAME special privilege

TuningTuning ITW renamed to DTA (Database Tuning Advisor)ITW renamed to DTA (Database Tuning Advisor) Time-bound tuning Time-bound tuning


VLDB SQL Server Technologies: VLDB SQL Server Technologies: Improved ConcurrencyImproved Concurrency

Transaction isolation levelsTransaction isolation levels

Database SnapshotDatabase Snapshot


Data Access ConcurrencyData Access Concurrency

Transaction isolation levelsTransaction isolation levels SerializableSerializable Repeatable ReadRepeatable Read Read CommittedRead Committed Read Committed SnapshotRead Committed Snapshot Read UncommittedRead Uncommitted SnapshotSnapshot


VLDB SQL Server Technologies: Snapshot VLDB SQL Server Technologies: Snapshot IsolationIsolation

Increased data availability for read applicationsIncreased data availability for read applications Allows non-blocking consistent reads in an OLTP Allows non-blocking consistent reads in an OLTP

environmentenvironment Writers do not block readersWriters do not block readers Readers do not block writersReaders do not block writers

Permits writes, which can cause conflictsPermits writes, which can cause conflicts BUT… includes mandatory conflict detection, BUT… includes mandatory conflict detection,

therefore…therefore… Consider Snapshot Isolation when the cost of locking Consider Snapshot Isolation when the cost of locking

data outweighs the cost of occasionally rolling back a data outweighs the cost of occasionally rolling back a transactiontransaction

Whitepaper Whitepaper ““SQL Server 2005 Beta 2 Snapshot Isolation” on SQL Server 2005 Beta 2 Snapshot Isolation” on

http://www.microsoft.com/technet/prodtechnol/sql/2005/SQL05B.mspxhttp://www.microsoft.com/technet/prodtechnol/sql/2005/SQL05B.mspx


VLDB SQL Server Technologies: VLDB SQL Server Technologies: Database SnapshotDatabase Snapshot

Snapshot of a database at a point in timeSnapshot of a database at a point in time Created instantlyCreated instantly Read onlyRead only Transactionally consistent, time-consistentTransactionally consistent, time-consistent

Base database continues to changeBase database continues to change Database Snapshot does not restrict access to the Database Snapshot does not restrict access to the

base databasebase database Multiple Snapshots are allowedMultiple Snapshots are allowed Database Snapshots can exist foreverDatabase Snapshots can exist forever

Constrained by resourcesConstrained by resources


Database SnapshotDatabase Snapshot Database snapshots do consume Database snapshots do consume

resources on the server. Example: Buffer resources on the server. Example: Buffer PoolPool

Tested – TPC-C run lost less than 15% Tested – TPC-C run lost less than 15% perf with single Database snapshot takenperf with single Database snapshot taken

11PagePage

22 33 44 55 66 77 88 99 1010 1111 1212 1313 1414 1515 1616

dbSnap – Read-Only Database SnapshotdbSnap – Read-Only Database Snapshot

CREATE DATABASE dbSnap AS SNAPSHOT OF mydbCREATE DATABASE dbSnap AS SNAPSHOT OF mydbmydb – Databasemydb – DatabaseUSE mydbUSE mydb

UPDATEUPDATE (pages 4, 9, 10) (pages 4, 9, 10)

USE mydbSnapUSE mydbSnapSELECT SELECT (pages 4, 6, 9, 10, 14)(pages 4, 6, 9, 10, 14)

44 99 101011 22 33 44 55 66 77 88 99 1010 1111 1212 1313 1414 1515 1616


VLDB SQL Server technologies: VLDB SQL Server technologies: Backup & RestoreBackup & Restore

Common Challenge vs. TimeCommon Challenge vs. Time Backup large data volumeBackup large data volume Restore/recover large data volumeRestore/recover large data volume

SQL Server SolutionsSQL Server Solutions SQL Server backup/restore SQL Server backup/restore

improvementsimprovements Online piecemeal restoreOnline piecemeal restore Database SnapshotsDatabase Snapshots

33rdrd party solutions party solutions SAN based solutionsSAN based solutions Software based solutionsSoftware based solutions


FilegroupsFilegroupsBasic unit of availabilityBasic unit of availability

Earlier Versions of SQL ServerEarlier Versions of SQL Server Entire databaseEntire database

SQL Server 2005SQL Server 2005 Database can be online, yet not all Database can be online, yet not all

filegroups are availablefilegroups are available


ExampleExampleServer startupServer startup

DatabaseDatabase

PrimaryPrimaryFilegroupFilegroup

Filegroup AFilegroup A

Filegroup BFilegroup B

Filegroup A is Filegroup A is unavailable due unavailable due to missing file to missing file

Database is available Database is available

DBA locates file DBA locates file and recovers the and recovers the

filegroup filegroup

Filegroup is now available Filegroup is now available


Piecemeal Backup-RestorePiecemeal Backup-Restore

Partial database availability Partial database availability during restoreduring restore

Based on file / filegroup backupBased on file / filegroup backup Works with all recovery modelsWorks with all recovery models

Full, Simple, Bulk-LoggedFull, Simple, Bulk-Logged Supported only for databases with Supported only for databases with

multiple filegroupsmultiple filegroups Filegroups must be recovered to consistent pointFilegroups must be recovered to consistent point


DatabaseDatabase

PrimaryPrimaryFilegroupFilegroup

AvailableAvailable

ExampleExamplePiecemeal restore for disaster recoveryPiecemeal restore for disaster recovery

Online restore of filegroups Online restore of filegroups in priority orderin priority order


Filegroup AFilegroup A

PrimaryPrimaryBackupsBackups

LogLog Filegroup AFilegroup A



Online Piecemeal RestoreOnline Piecemeal Restore PrinciplePrinciple

Filegroups can be restored in stagesFilegroups can be restored in stages Single object restoration (through filegroup) into temporary Single object restoration (through filegroup) into temporary

databasedatabase Logical correction possible to corect errorLogical correction possible to corect error

BenefitBenefit Granular and quick restoration of table and partition at Granular and quick restoration of table and partition at

filegroup levelfilegroup level StepsSteps

Backup tail of active transaction logBackup tail of active transaction log Using piecemeal restore to create temporary work Using piecemeal restore to create temporary work

databasedatabase Restore primary db/log to temporary work db with Restore primary db/log to temporary work db with

“norecovery”, “partial”“norecovery”, “partial” Restore filegroupRestore filegroup Restore from backup of current logRestore from backup of current log

Logically recovers dataLogically recovers data

See SQLCAT blog by Tom DavidsonSee SQLCAT blog by Tom Davidson – – http://blogs.msdn.com/sqlcat/archive/2005/10/27/485580.aspxhttp://blogs.msdn.com/sqlcat/archive/2005/10/27/485580.aspx


Short Restore SequenceShort Restore SequenceFor Read-Only Data For Read-Only Data

Log backups need not be restored when:Log backups need not be restored when: File / filegroup is unchanged since backupFile / filegroup is unchanged since backup File has been rolled forward to where it is read-only File has been rolled forward to where it is read-only

Useful for read-only tables or partitionsUseful for read-only tables or partitions Set filegroup to read-onlySet filegroup to read-only

Recommended for FULL and BULK-LOGGED databasesRecommended for FULL and BULK-LOGGED databases Prevents inadvertent changePrevents inadvertent change Prevents changes by background tasksPrevents changes by background tasks

Required for SIMPLE model databasesRequired for SIMPLE model databases Log backup for roll forward not availableLog backup for roll forward not available


VLDB SQL Server VLDB SQL Server technologies: technologies: Statistics Statistics

ManagementManagement Upgrade from SQL Server 2000Upgrade from SQL Server 2000 SS2K statistics are rendered invalid SS2K statistics are rendered invalid SS2K5 will automatically build statistics after SS2K5 will automatically build statistics after

upgrade upon query executionupgrade upon query execution If you don’t run it manually initial If you don’t run it manually initial

performance may/will sufferperformance may/will suffer SS2K5 statistics has changedSS2K5 statistics has changed

Now use formula O(sqrt(tablecard))Now use formula O(sqrt(tablecard)) For very large tables this most likely will For very large tables this most likely will

be less than 1%be less than 1% Recommendation: For large tables use Recommendation: For large tables use

10% sampling, but you must test and try 10% sampling, but you must test and try different values. The right answer could different values. The right answer could be 5% or 50%.be 5% or 50%.





Capacity Planning Capacity Planning Storage and object managementStorage and object management

DisksDisks TempDBTempDB Index ManagementIndex Management Transaction LogsTransaction Logs

MemoryMemory CPUCPU


Agenda Agenda


Storage Planning…Storage Planning…

Planning for SANPlanning for SAN Most VLDB uses SAN storageMost VLDB uses SAN storage Standard disk/controller number may not Standard disk/controller number may not

apply due to large cache and pre-fetch apply due to large cache and pre-fetch algorithmalgorithm

RecommendationRecommendation Using SQLIO or IOMETER to test/benchmark Using SQLIO or IOMETER to test/benchmark

the SAN prior to implementing SQL Serverthe SAN prior to implementing SQL Server Define storage throughput requirement for Define storage throughput requirement for

the application – e.g. my data warehouse the application – e.g. my data warehouse need to push through 500mb/secneed to push through 500mb/sec


Understanding your SAN IO Understanding your SAN IO SubsystemSubsystem

Disks

Mirrored Cache

CPU CPU CPU …… CPU CPU CPU ……

SAN CacheSAN Cache

SANServer

SQL Servermemory

SQL Server

Controllers

Client


Example – Example – insert stmt / commit to insert stmt / commit to loglog

Disks

Mirrored Cache


SAN CacheSAN Cache

SANServer

SQL Servermemory

SQL Server

Controllers

Client

DataPage

Log Record OnlyLog Record Only


Example: write completedExample: write completednote: Data not on disknote: Data not on disk

Disks

Mirrored Cache


SAN CacheSAN Cache

SANServer

SQL Servermemory

SQL Server

Controllers

Client

Datapage

IO COMPLETEIO COMPLETE


Next System Checkpoint – Next System Checkpoint – data page(s) get written to data page(s) get written to SAN – SAN – note: again not on disknote: again not on disk

Disks

Mirrored Cache


SAN CacheSAN Cache

SANServer

SQL Servermemory

SQL Server

Controllers

Client

DataPage


Periodically the SAN will Periodically the SAN will flush its data to disk (de-flush its data to disk (de-staging)staging)

Disks

Mirrored Cache


SAN CacheSAN Cache

SANServer

SQL Servermemory

SQL Server

Controllers

Client


Storage PlanningStorage Planning SQL Server guidelines – IO throughputSQL Server guidelines – IO throughput Capacity planning considerations:Capacity planning considerations:

# of spindles to handle mb/sec read # of spindles to handle mb/sec read requirements of the applicationrequirements of the application

# of spindles to handle de-staging of the SAN # of spindles to handle de-staging of the SAN cachecacheDeviceDevice SpeedSpeed CommentsComments

Disk SpeedDisk Speed 200-250 IO(s)/sec200-250 IO(s)/sec- - 13-16mb/sec(64K)13-16mb/sec(64K)- 2mb/sec (8K)- 2mb/sec (8K)

Note: HW vendors will tell you disk are 500+ IOs/sec Note: HW vendors will tell you disk are 500+ IOs/sec and 125+mb/sec. These numbers are raw file speeds and 125+mb/sec. These numbers are raw file speeds with very large block sizes of 256K or greater. SQL with very large block sizes of 256K or greater. SQL Server reads at 8K, 64K or larger and writes at 8K or Server reads at 8K, 64K or larger and writes at 8K or 64K64K

Controller Controller 125-150 mb/sec125-150 mb/sec Note: based on per channel / per controller basisNote: based on per channel / per controller basis

SQL Server random SQL Server random readread

4-5 ms4-5 ms Note: This is just the IO completion, not the complete Note: This is just the IO completion, not the complete transactiontransaction

SQL Server writeSQL Server write 1 ms1 ms Notes: Notes: Assuming write through cache on controller and SAN. Assuming write through cache on controller and SAN. This is cost of Windows write in perfmon – AVG DISK This is cost of Windows write in perfmon – AVG DISK SEC / WRITE counter.SEC / WRITE counter. Log only as data pages are lazy writtenLog only as data pages are lazy written Numbers do not include de-staging of SAN CacheNumbers do not include de-staging of SAN Cache


Transaction Log Transaction Log Planning…Planning… Database scalability is limited by the Database scalability is limited by the

maximum throughput of the transaction maximum throughput of the transaction log (Disk I/O)log (Disk I/O)

Plan on 1 ms / log write (utilizing write Plan on 1 ms / log write (utilizing write through cache) – Avg Disk Sec/Write through cache) – Avg Disk Sec/Write counter in perfmon (using SAN)counter in perfmon (using SAN)

Note: SQL Server will combine multiple Note: SQL Server will combine multiple write requests into a single log write (See write requests into a single log write (See diagram next page) but keep orderdiagram next page) but keep order

Fastest configuration is multiple files and Fastest configuration is multiple files and multiple disks striped to handle de-multiple disks striped to handle de-staging of cache and multiple log staging of cache and multiple log operations (Replication, Triggers, etc.)operations (Replication, Triggers, etc.)

Ballpark – (Depends on many variables): Ballpark – (Depends on many variables): plan on 40-60K database transactions / plan on 40-60K database transactions / single logsingle log


Database Log ThoughputDatabase Log Thoughput

.

.

.

Update, insert, deleterequests

SQL Server

SS Memory

Datapages

1. Insert happens

2. Data page written to SS Memory

3. Write request issued to writeout log to IO Subsystem

Log Buffer

4. By the time Log write IO is issuedSeveral other log writes piggyback


TEMPDB Planning…TEMPDB Planning… SQL Server 2005 tempdb requires SQL Server 2005 tempdb requires

more spacemore space Some new features in SQL Server 2005Some new features in SQL Server 2005 Some features that use tempdb in Some features that use tempdb in

earlier versions of SQL Server may earlier versions of SQL Server may require more tempdb disk space in SQL require more tempdb disk space in SQL Server 2005.Server 2005.

Some features that do not use tempdb Some features that do not use tempdb in earlier versions of SQL Server use in earlier versions of SQL Server use tempdb in SQL Server 2005tempdb in SQL Server 2005

Good MSDN article: Capacity Good MSDN article: Capacity Planning for tempdb: Planning for tempdb: http://msdn2.microsoft.com/en-us/lihttp://msdn2.microsoft.com/en-us/library/ms345368.aspxbrary/ms345368.aspx


TempDB capacity TempDB capacity planningplanning On line index: On line index:

2x-3x size of index – Sort size, temp index and 2x-3x size of index – Sort size, temp index and rollbackrollback

Versioning: Versioning: [Size of Version Store] = 2 *[Size of Version Store] = 2 *

[Version store data generated per minute] *[Version store data generated per minute] *[Longest running time (minutes) of your [Longest running time (minutes) of your transaction] * number of concurrent transaction] * number of concurrent transactions/userstransactions/users

Note: Version store data generated per minute and Note: Version store data generated per minute and version store size are now a perfmon parametersversion store size are now a perfmon parameters

Recommendation repeated: Be sure to tune Recommendation repeated: Be sure to tune TempDB for proper sizing as well as performanceTempDB for proper sizing as well as performance


TEMPDB capacity TEMPDB capacity Planning…Planning…

aa more reliable methodmore reliable method Determine tempdb space usedDetermine tempdb space used

select select

sum(user_object_reserved_page_count)*8 as sum(user_object_reserved_page_count)*8 as user_objects_kb,user_objects_kb,

sum(internal_object_reserved_page_count)*8 as sum(internal_object_reserved_page_count)*8 as internal_objects_kb,internal_objects_kb,

sum(version_store_reserved_page_count)*8 as sum(version_store_reserved_page_count)*8 as version_store_kb,version_store_kb,

sum(unallocated_extent_page_count)*8 as sum(unallocated_extent_page_count)*8 as freespace_kbfreespace_kb

from from sys.dm_db_file_space_usagesys.dm_db_file_space_usage

where database_id = 2where database_id = 2


TEMPDB capacity TEMPDB capacity planing.....planing.....

What tasks are using tempdb?What tasks are using tempdb? Determine tempdb space used by Determine tempdb space used by TaskTask

SELECT t1.session_id,SELECT t1.session_id,(t1.internal_objects_alloc_page_count + task_alloc)(t1.internal_objects_alloc_page_count + task_alloc)

as allocated,as allocated,(t1.internal_objects_dealloc_page_count + (t1.internal_objects_dealloc_page_count +

task_dealloc) as deallocated task_dealloc) as deallocated from sys.dm_db_session_space_usage as t1, from sys.dm_db_session_space_usage as t1, (select session_id, (select session_id, sum(internal_objects_alloc_page_count)sum(internal_objects_alloc_page_count) as task_alloc,as task_alloc, sum (internal_objects_dealloc_page_count)sum (internal_objects_dealloc_page_count)

as task_dealloc as task_dealloc from sys.dm_db_task_space_usage group by session_id) from sys.dm_db_task_space_usage group by session_id)

as t2as t2 where t1.session_id = t2.session_idwhere t1.session_id = t2.session_id and t2.session_id >50and t2.session_id >50 order by allocated DESCorder by allocated DESC


Index Operations Lessons LearnedIndex Operations Lessons Learned

Online index build logs the entire contents of the Online index build logs the entire contents of the target index pages. Hence log space requirements target index pages. Hence log space requirements increase significantly. increase significantly. You can specify to use TempDB to avoid log sizing and You can specify to use TempDB to avoid log sizing and

contention. contention. Recommendation: specify sort_in_tempdb option for Recommendation: specify sort_in_tempdb option for

online index build and tune temp_db to avoid excessive online index build and tune temp_db to avoid excessive log activitylog activity

Online index not available on a per-partition basis Online index not available on a per-partition basis for partitioned tables.for partitioned tables.

ALTER INDEX … REBUILD of a corrupted non-ALTER INDEX … REBUILD of a corrupted non-clustered index may use another index (and that clustered index may use another index (and that may be theoretically corrupted as well). may be theoretically corrupted as well). Recommendation: Therefore use DBCC repair rather Recommendation: Therefore use DBCC repair rather

than alter index…rebuildthan alter index…rebuild


TempDB for Index TempDB for Index Operation…Operation…

OperationOperation

TEMPDB space used / Size of the indexTEMPDB space used / Size of the indexSingle column indexSingle column index

8% of row size8% of row size(4 bytes out of 46 (4 bytes out of 46

bytes)bytes)

3 columns index3 columns index25% of row size25% of row size

(10 bytes out of 42 (10 bytes out of 42 bytes)bytes)

All columns indexAll columns index100% of row size100% of row size

(42 bytes out of 42 (42 bytes out of 42 bytes)bytes)

MappinMapping Indexg Index

Sort Sort runsruns TotalTotal





Online clustered Online clustered index createindex create 0.330.33 1.11.1 1.431.43 0.520.52 1.121.12 1.641.64 1.211.21 1.111.11 2.322.32

Online clustered Online clustered index rebuildindex rebuild 0.170.17 00 0.170.17 0.310.31 00 0.310.31 1.011.01 00 1.011.01

Online non-Online non-clustered Index clustered Index createcreate

n/an/a n/an/a n/an/a n/an/a n/an/a n/an/a n/an/a n/an/a n/an/a

Online non-Online non-clustered index clustered index rebuildrebuild

00 00 00 00 00 00 00 00 00

Offline clustered Offline clustered index createindex create 00 1.021.02 1.021.02 00 1.021.02 1.021.02 00 1.011.01 1.011.01

Offline clustered Offline clustered index rebuildindex rebuild 00 00 00 00 00 00 00 00 00

Offline non-Offline non-clustered index clustered index create/rebuildcreate/rebuild

00 1.071.07 1.071.07 00 1.051.05 1.051.05 00 1.021.02 1.021.02* Above table illustrates the amount of TEMPDB space used with respect to the size of the index* Above table illustrates the amount of TEMPDB space used with respect to the size of the index

Online clustered index creation can take up to 2.5x size of heap!Online clustered index creation can take up to 2.5x size of heap!


Index ManagementIndex Management Defrag only necessaryDefrag only necessary

Fragmentation has little performance impact on index-Fragmentation has little performance impact on index-seekseek

Online Index maintenance in VLDBOnline Index maintenance in VLDB Offline operation still much faster than onlineOffline operation still much faster than online On average, Insert/Delete operation slower during online On average, Insert/Delete operation slower during online

index – up to 30%index – up to 30% Non-clustered index with included columns to Non-clustered index with included columns to

replace covering indexreplace covering index

OperationOperation DuratioDurationn

NotesNotes

SQL Server 2000 Offline index SQL Server 2000 Offline index rebuildrebuild

3:343:34

SQL Server 2005 offline index rebuildSQL Server 2005 offline index rebuild 2:362:36 27% faster than SQL 27% faster than SQL Server 2000Server 2000

SQL Server 2005 online index rebuildSQL Server 2005 online index rebuild 10:5210:52

SQL Server 2005 online index rebuild SQL Server 2005 online index rebuild with normal workloadwith normal workload

11:3011:30


Unused IndexUnused Index Unused index cause unnecessary index Unused index cause unnecessary index

maintenance overhead during insert maintenance overhead during insert operationoperation

Determine unused indexDetermine unused indexselect object_name(i.object_id), i.name select object_name(i.object_id), i.name from sys.indexes i, sys.objects o from sys.indexes i, sys.objects o Where objectproperty(o.object_id,'IsUserTable') = Where objectproperty(o.object_id,'IsUserTable') =

11And And i.index_id NOT IN (select s.index_id i.index_id NOT IN (select s.index_id from sys.dm_db_index_usage_stats s from sys.dm_db_index_usage_stats s where s.object_id=i.object_id and where s.object_id=i.object_id and

i.index_id=s.index_id and i.index_id=s.index_id and database_id = database_id = <dbid><dbid> ) )

and o.type = 'U'and o.type = 'U'and o.object_id = i.object_idand o.object_id = i.object_idorder by object_name(i.object_id) ascorder by object_name(i.object_id) asc


Memory PlanningMemory Planning Avoid pagingAvoid paging

Use “Lock Pages in Memory” optionUse “Lock Pages in Memory” option Proper sizingProper sizing

Monitor Buffer/Plan cacheMonitor Buffer/Plan cache System with poor plan reuse may starve System with poor plan reuse may starve

bufferbuffer Use dbcc freeproccacheUse dbcc freeproccache Use dbcc freesystemcacheUse dbcc freesystemcache

SQL Server Max Memory SQL Server Max Memory ==

Total system memoryTotal system memory

–– OS/ApplicationOS/Application

–– <Max worker thread> x <Thread stack <Max worker thread> x <Thread stack size>size>

Thread size: x86 – 512K; X64 – 2MB; IA64 – 4MBThread size: x86 – 512K; X64 – 2MB; IA64 – 4MB


CPU PlanningCPU Planning Degree of ParallelismDegree of Parallelism

OLTP environment favors low MAXDOPOLTP environment favors low MAXDOP OLAP/DW environment favors higher MAXDOPOLAP/DW environment favors higher MAXDOP Dynamically changeable setting MAXDOP=0 for Dynamically changeable setting MAXDOP=0 for

regular business hour, and change to regular business hour, and change to xx during during index maintenanceindex maintenance

Affinity MaskAffinity Mask ““affinity mask” controls CPU 0-31affinity mask” controls CPU 0-31 ““affinity64 mask” controls CPU 32-64affinity64 mask” controls CPU 32-64 When set, SQL Server 2005 scheduler affinitize to When set, SQL Server 2005 scheduler affinitize to

CPUCPU NUMA & application partitioningNUMA & application partitioning

Associating multiple TCP/IP ports to a single nodeAssociating multiple TCP/IP ports to a single node Associating a single TCP/IP port to multiple nodesAssociating a single TCP/IP port to multiple nodes







MemoryMemory CPUCPU

SummarySummary

Agenda Agenda


SummarySummary

VLDB is just many smaller VLDB is just many smaller manageable problems to solve if manageable problems to solve if you partition the big into smaller you partition the big into smaller componentscomponents

SQL Server 2005 has many new SQL Server 2005 has many new features to help VLDBfeatures to help VLDB Partitioning, online utilities, DMV(s), Partitioning, online utilities, DMV(s),

high performance, etc.high performance, etc.

seas – 2006 sql server 2005 vldb management & operation

Documents

vldb challenge

design data

partitioned data

consulting data

data separation

inventory data

sales data

competitive data