performance tuning ssis
DESCRIPTION
Performance Tuning SSIS. Brian Knight, CEO Pragmatic Works [email protected]. HR Departments are no fun. Don’t mention the stalking incident with Clay Aiken What happened in Vegas My prom date with a puppet Most unfortunate incident with a turtle My fear of bounce houses - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/2.jpg)
HR Departments are no fun.
Don’t mention the stalking incident with Clay AikenWhat happened in VegasMy prom date with a puppetMost unfortunate incident with a turtleMy fear of bounce housesHow to sexually harass the HR rep
• What I did to a fish when I was 8
• Any talk about my college years
• The surgery I had last summer
• The stint I had as a traveling gypsy
• Why am I still not allowed back Texas
• How what I did in Vegas truly can’t stay in Vegas
![Page 3: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/3.jpg)
About Brian
Wasn’t very good with girlsEven Kermit the Frog founded a companyAll 15 still awaiting a publisher.Where he writes about his miniature donkey collection.
• SQL Server MVP• Founder of Pragmatic
Works• Author of 15 books• Blogs at BIDN.com
Twitter: @BrianKnight
![Page 4: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/4.jpg)
4
GeoSpatial Data:Semi structured
Legacy data: binary files
Application database
Integration is a seamless, manageable operationSource, prepare, & load data in single, auditable processScale to handle heavy and complex data requirements
SQL Server Integration Services
GeoSpatialComponents
Customsource
Standardsources
Data-cleansingcomponents
Merges
Data miningcomponents
Warehouse
Reports
Mobiledata
Integration Services in Action
Cube
![Page 5: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/5.jpg)
5
Advanced Session
![Page 6: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/6.jpg)
6
Today’s Problems with Integration
Integration todayIncreasing data volumesIncreasingly diverse sources
Requirements reached the Tipping PointLow-impact source extractionEfficient transformationBulk loading techniques
![Page 7: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/7.jpg)
7
Tuning DecisionsChoose the right tool for the jobDon’t be afraid to use T-SQLWill parallelism work?
![Page 8: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/8.jpg)
8
Source OptimizationFlat files – When available, use Fast ParseOLE DB sources – Change network packet sizeUse T-SQL whenever possible in the OLE DB Source
JoiningNULL handlingWhere clauses
![Page 9: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/9.jpg)
SQL ServerSQL Server
Network TrafficConnection Settings
Packet size defaults to 4096Increase to 32767 on large data sets
Database
SSISPackage S
WITCH
LAN LAN
LAN LAN
![Page 10: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/10.jpg)
10
Impact of Compression on ETL
NONE ROW PAGE05
101520253035
0123456
BULK INSERT into a Heap with and without Data Compression
Time to BULK INSERT 50M rows (min)Table Size after Load (GB)
Compression Type
Tim
e (m
inut
es)
Tabl
e Si
ze a
fter L
oad
(GB)
* Not official Microsoft results.
![Page 11: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/11.jpg)
Tuning the Source
Connection manager tuningFlat file tuningOLE DB Source tuning
Demo
![Page 12: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/12.jpg)
12
Transform Components
x x xThe Pipeline presents the buffer to each downstream component
x x xx x xx x xx x xx x x
![Page 13: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/13.jpg)
13
SSIS Data Flow Architecture
Synchronous vs. Non Synchronous
![Page 14: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/14.jpg)
14
Case Study: Patterns
105 seconds 83 seconds
![Page 15: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/15.jpg)
DemoCascading lookup optimizationsCache file lookup
![Page 16: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/16.jpg)
21
Data DestinationsUse “Fast Load” or SQL Server DestinationTable Lock on insert operationsTrace flags for improvementOld principles still apply
![Page 17: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/17.jpg)
Destination Tuning
Demo
![Page 18: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/18.jpg)
23
Managing Resources
Logging events to watch pipeline internalsPipelineExecutionPlan, PipelineExecutionTree, BufferSizeTuning
System Monitor to track I/O issuesBuffers In Use tracks how many buffers are presently being usedBuffers Spooled tracks how many 10 mb buffers have been spooled to disk
![Page 19: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/19.jpg)
Measuring PerformancePerfmon
![Page 20: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/20.jpg)
25
Location
Consider the following configuration…
Where should SSIS run? (Licensing issues aside)
SQL Server 1 SQL Server 2
SSIS Server
![Page 21: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/21.jpg)
26
WSRMWindows System Resource Manager (WSRM) can throttle CPU and memory
Creates a soft throttleCan be scheduled so SSIS gets priority on weekends and nightsOnly activates policy if resources begin to become constrained (about 70%)WSRM is free with Windows Server 2003 Enterprise Edition and included in Windows Server 2008
![Page 22: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/22.jpg)
WSRMCreating a soft schedule cap
Demo
![Page 23: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/23.jpg)
Building a Work Queue System
Create a work queue table.
Create a loop to shift over the work queue constantly checking out work
Spawn x times with a batch file
![Page 24: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/24.jpg)
Demo Results
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
1 Process finishes in 64 seconds
Elap
sed
Tim
e
![Page 25: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/25.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
2 Processes finish in 36 seconds
Elap
sed
Tim
e
Demo Results
![Page 26: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/26.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
4 Processes finish in 28 seconds
Elap
sed
Tim
e
Demo Results
![Page 27: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/27.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
8 Processes finish in 27 seconds
Elap
sed
Tim
e
Demo Results
![Page 28: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/28.jpg)
Parallel Load
Demo
![Page 29: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/29.jpg)
SummaryPlanning
Don’t underestimate the power of the whiteboard!Use the right tool for the right job
Leverage the power of the enginePatterns and Practices
Understand best practicesBut don’t be afraid to experiment
![Page 30: Performance Tuning SSIS](https://reader035.vdocument.in/reader035/viewer/2022062305/56815eec550346895dcda390/html5/thumbnails/30.jpg)
35
The End Already?
Questions
http://www.bidn.com/people/brianknight
@BrianKnight
http://www.youtube.com/pragmaticworks