2 overview of ssis performance troubleshooting methods performance tips
TRANSCRIPT
![Page 1: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/1.jpg)
Maximizing SSIS Package Performance
Tim Mitchell
![Page 2: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/2.jpg)
2
Today’s Agenda
• Overview of SSIS performance• Troubleshooting methods• Performance tips
![Page 3: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/3.jpg)
3
Tim Mitchell
• Business intelligence consultant• Partner, Linchpin People• SQL Server MVP• TimMitchell.net / @Tim_Mitchell• [email protected]
![Page 4: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/4.jpg)
4
Housekeeping
• Questions• Slide deck
4
![Page 5: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/5.jpg)
5
Performance Overview
![Page 6: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/6.jpg)
6
Performance Overview
Two most common questions about SSIS package executions:• Did it complete successfully?• How long did it run?
![Page 7: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/7.jpg)
7
Performance Overview
Why is ETL performance important?• Getting data to the right people in a
timely manner• Load/maintenance window• Potentially overlapping ETL cycles• The most important concern: bragging
rights!
![Page 8: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/8.jpg)
8
Troubleshooting SSIS Performance
![Page 9: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/9.jpg)
9
Troubleshooting SSIS Performance
Key questions:• Is performance really a concern?• Is it really an SSIS issue?• Which task(s) or component(s) are
causing the bottleneck?
![Page 10: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/10.jpg)
10
Troubleshooting SSIS Performance
Is this really an SSIS issue?• Test independently, outside SSIS• Compare results to SSIS
![Page 11: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/11.jpg)
11
Troubleshooting SSIS Performance
Where in SSIS is the bottleneck?• Logging is critical• Package logging (legacy logging)• Catalog logging (SQL 2012 only)
![Page 12: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/12.jpg)
12
Troubleshooting SSIS Performance
SELECT execution_id, package_name, task_name, subcomponent_name
, phase, MIN(start_time) [Start_Time], MAX(end_time) [End_Time]FROM catalog.execution_component_phasesWHERE execution_id =
(SELECT MAX(execution_id) from [catalog].[execution_data_statistics])GROUP BY execution_id, package_name, task_name, subcomponent_name, phaseORDER BY 6
![Page 13: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/13.jpg)
13
Troubleshooting SSIS Performance
Brute force troubleshooting: Isolation by elimination• Disable tasks• Remove components
![Page 14: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/14.jpg)
14
Troubleshooting SSIS Performance
Monitor system metrics• Disk IO• Memory• CPU• Network
![Page 15: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/15.jpg)
15
Performance Tips
![Page 16: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/16.jpg)
16
Tip #1: Tune your sources and destinations
• Many performance problems in SSIS aren’t SSIS problems
• Sources and destination issues are often to blame
![Page 17: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/17.jpg)
17
Tip #1: Tune your sources and destinations
• Improper data retrieval queries• Index issues• Network speed/latency• Disk I/O
![Page 18: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/18.jpg)
18
Tip #2: Using OPTION (FAST <n>)
• Directs the query to return the first <n> rows as quickly as possible
SELECT FirstName, LastName, Address, City, State, ZipFROM dbo.PeopleOPTION (FAST 10000)
• Not intended to improve the overall performance of the query
![Page 19: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/19.jpg)
19
Tip #2: Using OPTION (FAST <n>)
• Useful for packages that spend a lot of time processing data in data flow
Query time from SQL Server source
Processing time in SSIS package
Load time to destination
Without OPTION (FAST <n>)
![Page 20: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/20.jpg)
20
Tip #2: Using OPTION (FAST <n>)
• Useful for packages that spend a lot of time processing data in data flow
Processing time in SSIS package
Load time to destination
Query time from SQL Server source Using OPTION (FAST <n>)
![Page 21: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/21.jpg)
21
Tip #2: Using OPTION (FAST <n>)
![Page 22: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/22.jpg)
22
Tip #3: Blocking transformations
• Know the blocking properties of transformations• Nonblocking• Partially blocking• Fully blocking
![Page 23: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/23.jpg)
23
Tip #3: Blocking transformations
• Nonblocking – no holding buffers• Row count• Derived column transformation• Conditional split
![Page 24: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/24.jpg)
24
Tip #3: Blocking transformations
• Partially blocking – some buffers can be held• Merge Join• Lookup• Union All
![Page 25: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/25.jpg)
25
Tip #3: Blocking transformations
• Fully blocking – everything stops until all data is received• Sort• Aggregate• Fuzzy grouping/lookup
![Page 26: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/26.jpg)
26
Tip #3: Blocking transformations
• Partially or fully blocking transforms are not evil! Pick the right tool for every job.
![Page 27: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/27.jpg)
27
Tip #3: Blocking transformations
Demo – Blocking Transformations
![Page 28: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/28.jpg)
28
Tip #4: Slow transformations
• Some transformations are often slow by nature• Slowly Changing Dimension wizard• OleDB Command
![Page 29: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/29.jpg)
29
Tip #4: Slow transformations
• Useful for certain scenarios, but should not be considered go-to tools for transforming data
![Page 30: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/30.jpg)
30
Tip #5: Avoid the table list
• Table list = SELECT * FROM…• Can result in unnecessary columns• A narrow buffer is happy buffer
![Page 31: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/31.jpg)
31
Tip #5: Avoid the table list
• Use the query window to select from table, specifying only the required columns
• Writing the query can be a reminder to apply filtering via WHERE clause, if appropriate
![Page 32: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/32.jpg)
32
Tip #6: RunInOptimizedMode
• Data flow setting to prevent the allocation of memory from unused columns
• Note that you’ll still get a warning from SSIS engine
![Page 33: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/33.jpg)
33
Tip #7: Transform data in source
• When dealing with relational data, consider transforming in source
• Let the database engine do what it already does well
![Page 34: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/34.jpg)
34
Tip #7: Transform data in source
• Sorting• Lookups/joins• Aggregates
![Page 35: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/35.jpg)
35
Tip #8: Concurrent executables
• Package-level setting to specify how many executables can be running at once
• Default = -1• Number of logical processors + 2
![Page 36: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/36.jpg)
36
Tip #8: Concurrent executables
• For machines with few logical processors or potentially many concurrent executables, consider increasing this value
![Page 37: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/37.jpg)
37
Tip #8: Concurrent executables
Demo – Concurrent Executables
![Page 38: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/38.jpg)
38
Tip #9: Go parallel
• Many operations in SSIS are done serially
• For nondependent operations, consider allowing processes to run in parallel
![Page 39: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/39.jpg)
39
Tip #9: Go parallel
• Dependent on machine configuration, network environment, etc.
• Can actually *hurt* performance• Testing, testing, testing!
![Page 40: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/40.jpg)
40
Tip #10: Don’t be afraid to rebel
• Sometimes a non-SSIS solution will perform better than SSIS
• If all you have is a hammer…
![Page 41: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/41.jpg)
41
Tip #10: Don’t be afraid to rebel
• Some operations are better suited for T-SQL or other tools:• MERGE upsert• INSERT…SELECT• Third party components• External applications (via Execute
Process Task)
![Page 42: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/42.jpg)
42
Tip #11: Watch your spools
• Buffers spooled = writing to physical disk
• Usually indicates memory pressure• Keep an eye on PerfMon counters for
SSIS: Buffers Spooled
![Page 43: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/43.jpg)
43
Tip #11: Watch your spools
![Page 44: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/44.jpg)
44
Tip #11: Watch your spools
• Any value above zero should be investigated• Keep it in memory!
![Page 45: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/45.jpg)
45
Tip #12: Buffer sizing
• Data flow task values:• DefaultBufferSize• DefaultBufferMaxRows
• Ideally, use a small number of large buffers
• Generally, max number of active buffers = 5
![Page 46: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/46.jpg)
46
Tip #12: Buffer sizing
• Buffer size calculation:• Row size * est num of rows /
DefaultMaxBufferRows
![Page 47: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/47.jpg)
47
Tip #13: MaximumInsertCommitSize
• OleDbDestination setting
• Controls how buffers are committed to the destination database
![Page 48: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/48.jpg)
48
Tip #13: MaximumInsertCommitSize
MICS > buffer size
Setting is ignored. One commit is issued for every buffer.
MICS = 0 The entire batch is committed in one big batch.
MICS < buffer size
Commit is issued every time MICS rows are sent.Commits are also issued at the end of each buffer.
Source: Data Loading Performance Guide http://technet.microsoft.com/en-us/library/dd425070(v=sql.100).aspx
![Page 49: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/49.jpg)
49
Tip #13: MaximumInsertCommitSize
• Note that this does NOT impact the size of the buffer in SSIS
![Page 50: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/50.jpg)
50
Tip #13: MaximumInsertCommitSize
• In most cases, a small number of large commits is preferable to a large number of small commits
![Page 51: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/51.jpg)
51
Tip #13: MaximumInsertCommitSize
• Larger commit sizes = fewer commits• Good:• Potentially less database overhead• Potentially less index fragmentation
• Bad:• Potential for log file or TempDB
pressure/growth
![Page 52: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/52.jpg)
52
Tip #14: Prepare for paging buffers
• Plan for no paging of buffers to disk. But….
• Build for it if (when?) it happens• BLOBTempStoragePath• BufferTempStoragePath
• Fast disks if possible
![Page 53: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/53.jpg)
53
Tip #15: Manage your lookups
• Lookup cache modes• Full (default)• Partial• None
![Page 54: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/54.jpg)
54
Tip #15: Manage your lookups
• Full cache:• Small- to medium-size lookup table• Large set of data to be validated
against the lookup table• Expect multiple “hits” per result
![Page 55: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/55.jpg)
55
Tip #15: Manage your lookups
• Partial cache:• Large lookup table AND reasonable
number of rows from the main source• Expect multiple “hits” per result
![Page 56: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/56.jpg)
56
Tip #15: Manage your lookups
• No cache:• Small lookup table• Expect only one “hit” per result
![Page 57: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/57.jpg)
57
Tip #15: Manage your lookups
Demo – Lookups and caching
![Page 58: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/58.jpg)
58
Tip #16: Share the road
• Strategically schedule packages to avoid contention with other packages, external processes, etc.• Consider a workpile pattern in SSIS
![Page 59: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/59.jpg)
59
Tip #16: Share the road
• Resource contention is a key player in SSIS performance issues• Other SSIS or ETL processes• External processes
![Page 60: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/60.jpg)
60
Tip #17: Stage your data
• SQL-to-SQL operations perform well• Can’t do direct SQL to SQL with
nonrelational or external data
![Page 61: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/61.jpg)
61
Tip #17: Stage your data
• Staging the data can allow for faster, direct SQL operations• Remember: Let the database engine
do what it does well.• Updates in particular
![Page 62: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/62.jpg)
62
Tip #17: Stage your data
Demo – Staged Update
![Page 63: 2 Overview of SSIS performance Troubleshooting methods Performance tips](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e4d5503460f94b43ae0/html5/thumbnails/63.jpg)
63
Additional Resources
• Rob Farley demonstrates FAST hint: http://bit.ly/eYNlMg
• Microsoft data loading performance guide: http://bit.ly/17xjgbw