Connectivity Options for Data Integration with Microsoft SQL Server Integration Services (SSIS)

Cedric LabuschagneSolutions ArchitectBI-LateralBIN301

Agenda

What is new in SSIS 2008?Connectivity componentsConnectivity CategoriesNew SSIS connectors

ERP

Thinking Bigger About Business Intelligence

Majority of the content for decision making are in multiple heterogeneous sources (e.g. ERP systems, databases, flat files)

Need to integrate & consolidate data from all these sources, and make quality data available to users

Connectivity is the key to address these challenges

FlatFile

ERP Systems

Excel

XML

Databases

What is new in SSIS 2008?

Improved and new task componentsNew Lookup Capabilities (caching enhanced)Data Profiling Task (data quality checking)C# Scripting. VSA replaced by VSTAScalable Data Flow (thread pooling)

Enhanced and new connectivity supportADO.NET Source and DestinationNew connectors for Oracle, Teradata and SAP BI

SSIS Connectivity ComponentsSSIS Package

Data Flow Task

FTP FLATFILE ADO.NETOLEDB

FTP ExecuteSQL Task FlatFileLookup ADO.NET

ConnectionManagerTask Data Source/

DestinationDataTransformation

Note: Execute SQL Task can also make use of ADO.Net, ODBC Connection Managers

Component InteractionSSIS

Data Storage

AcquireConnection()

Connection Object

Write() Read()

Taskor

Component

Connection Manager

Data Flow Connectivity OptionsData Flow Task

OLEDBConnector

ADO.NETConnector

CustomConnector

OLEDBProvider

NativeADO.NETAdapter

ADO.NETODBC Data Source

SpecificInterface/Protocol

Data Source Specific Interface/ Protocol

ODBC

Data Storage

Data Access APIs and SSIS

"When developing in BIDS, 32-bit data providers are always used at design time, if 64-bit providers will be used in production packages, the equivalent 32-bit provider must be available at design time."

API Pros Cons

OLEDB • Comprehensive support with SSIS • 64-bit OLE DB providers less prevalent.

ADO.NET• Good support with SSIS• Direct support with ADO.Net Source & ADO.Net Destination

• Somewhat limited availability of out of the box ADO.NET providers

ODBC • ODBC drivers for possible data sources widely available

• Requires the use of a bridge - introduces performance and reliability issues

Custom-API • Designed for performance of native utilities

•Limited availability for most databases • Inconsistent support with SSIS - varies according to architecture

SSIS Enterprise Data Connectivity Versions Suggested Connector SSIS Source &

DestinationSSIS Connection Manager

SQL Server All SQL Data Access Components ADO.NETOLEDB

ADO.NETOLEDB

Oracle 9iR2, 10, 11 Microsoft Connector for Oracle Oracle MSORA

Teradata V2R6.0/1/2, 12 Microsoft Connector for Teradata Teradata MSTERA

SAP BI 7 Microsoft Connector for SAP BI SAP BI SAPBI

DB2 8.1, 9.x Microsoft OLEDB Provider for DB2 OLEDB OLEDB

SAP R/3 4.6c, 4.7, 5, 6 BizTalk Adapter 3.0 for mySAP ADO.NET ADO.NET

Siebel 7.5, 7.7, 7.8, 8 BizTalk Adapter 3.0 for Siebel ADO.NET ADO.NET

Connectivity CategoriesDatabase Systems (RDBMS)• Microsoft SQL Server • Oracle • Sybase • IBM DB2 • Teradata • MS Access

Queue Systems & Protocols• MSMQ (Message Queue Task) • (s)FTP (FTP Task) ** • HTTP/HTTPs (Web Service Task) • IBM MQ Series* • Tibco Rendezveus*

Unstructured ,Semi-structured Data• Microsoft Excel, CSV • Text • XML • EDI (Electronic Data Interchange)

Application Systems • Siebel • Microsoft Excel • Hyperion (Essbase, Planning,

Financial Management)• SAP R/3 & SAPBI

* not supported out-of-the-box, however, custom components can do this.** sFTP is not supported out-of-the-box

Microsoft Connector for SAP BI

Microsoft Connector for SAP BI

Provider Microsoft

Version SAP NetWeaver BI Version 7

Pre-requisites SQL Server Enterprise LicenseSAP librfc32 librarySAP OpenHub license

Platform x86, x64, ia64

Microsoft Connector for SAP BIData Flow Task

SAP BIConnection Mgr

Remote Function Calls (RFC’s) *

SAP BI

SAP BIData Source

SAP BIData Destination

OpenHub BAPI

*standard programming interface used by SAP

SQL Server

Application Scenario 1Extracting data from SAP BI into SQL Server

SAP BI

OHSDestination

SSIS

SSIS Package

OLE dBDestination

SAP BISource

Ope

n H

ub

Info Cube

ProcessChainTriggered

SAP BIConnection

Manager

SAPBI Connectordemo

Microsoft Connector for SAP BI Extracting data from SAP BI to SQL Server

1. SAP BI Configuration1. Set up the RFC Destination2. Configure and create the Open Hub Destination3. Create the Data Transfer Process (DTP) and transformation4. Configure the process chain

2. SSIS Package1. Setup the SAP BI Source2. Setup the connection manager for SAP BI3. Setup the Destination (OLE DB, SQL Destination)4. Construct Workflow

3. Run the SSIS package

SAP BI

Stag

ing

BAPI

Application Scenario 2Loading non-SAP data into SAP BI

SSIS

SSIS Package

SAP BIConnection

Manager

OLE DBSource

SAP BIDestinationSQL

Server XML

FlatFile

InfoSource/InfoPackage

Microsoft Connector for Oracle (by Attunity)

Microsoft Connector for Oracle

Provider AttunitySupported by Microsoft

Version Oracle 9iR2, 10, and 11

Pre-requisites SQL Server Enterprise License*Oracle client components

Platform SSIS: x86, x64, ia64

* 32 bit @design time, 64 bit @ runtime)

Microsoft Connector for OracleData Flow Task

OracleConnection Mgr

OCI

Oracle dB

OracleData Source

OCIArray Binding

OracleData Destination

OCIDirect Path

OCI

Performance of Oracle Connectors

Load Extract

27680

37362

1646

29756

Microsoft Connector for Oracle (by Attunity)Microsoft OLE DB Provider for Oracle

Tota

l Thr

ough

put (

Row

s/Se

c)

x16 x 1.25

Performance of Microsoft Connector for Oracle

Load Load - 10K Extract Extract - 10K

2901933747 36722 39466

26413

50158

8425491895

Mixed Types String Only

Tota

l Thr

ough

put (

Row

s/Se

c)

Microsoft Connector for OracleCustom Properties

Component Access Mode Property (Default)Oracle Source Table Name Batch Size (100)

Prefetch Count (0)SQLOracle Destination Table Name Batch Size (100)

Transaction SizeNo Logging (False)

Table Name – Fast Load Parallel (False)No Logging (False)Buffer Size (64KB)

Oracle Connectordemo

Microsoft Connector for Teradata (by Attunity)

Microsoft Connector for Teradata

Provider AttunitySupported by Microsoft

Version Teradata V2R6.0/1/2, 12

Pre-requisites SQL Server Enterprise LicenseTeradata ODBC driverTPT 12 for high-performance

Platform x86, x64

Microsoft Connector for TeradataData Flow Task

TeradataConnection Mgr

TPT12

Teradata DW

TeradataData Source

TeradataData Destination

TPTFastLoad

TPTTPUMP

TPTFast

Export

Microsoft Connector for TeradataCustom Properties

Component Access Mode Property (Default)Teradata Source Table Name – TPT Export BlockSize (64330)

DataEncryption (False)Max Sessions (1)Min Sessions (1)TenacityHours(4)TenacitySleep(6)

SQL Command – TPT Export

Teradata Destination Table Name – TPT Stream BlockSize (64330)DataEncryption(False)MaxErrors (0)MaxSessions (1)MinSessions(1)TenacityHours(4)TenacitySleep(6)

Table Name – TPT Load

TeraData Connectordemo

Recap

SSIS connectivity Provides a robust connectivity architectureProvides an extensible connectivity architectureProvides rich connectivity options to enterprise data via :

Built-in connectorsAdditional connectors (available from Microsoft downloads)*Third party connectors

Rich ecosystem for SSIS connectors

*[codeplex or custom dev]

question & answer

www.microsoft.com/teched

International Content & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources Tech·Ed Africa 2009 sessions will be made available for download the week after the event from: www.tech-ed.co.za

SSIS Community ResourcesSSIS Community wiki

http://wiki.sqlis.com/default.aspxSSIS Codeplex

http://sqlsrvintegrationsrv.codeplex.com/SSIS Junkies :

http://blogs.conchango.com/jamiethomson/Attunity Support Forums :

http://www.attunity.com/forums/microsoft-ssis-oracle-connectorhttp://www.attunity.com/forums/micorosft-ssis-teradata-connector

Additional Community ResourcesTechNet Community for IT Professionalshttp://technet.microsoft.com/en-us/sqlserver/bb671048.aspxDeveloper Center http://msdn.microsoft.com/en-us/sqlserver/bb671064.aspxSQL Server 2008 Learning Portalhttp://www.microsoft.com/learning/sql/2008/default.mspx

Additional Resources

External ResourcesSSIS ETL Record – Loaded 1TB in 30 Minutes! http://msdn.microsoft.com/en-us/library/dd537533.aspxConnectivity Options for Microsoft SQL Server 2008 Integration Serviceshttp://msdn.microsoft.com/en-us/library/dd299429.aspxMicrosoft Connectors for Oracle and Teradata by Attunity http://www.microsoft.com/downloads/details.aspx?FamilyId=D9CB21FE-32E9-4D34-A381-6F9231D84F1EUsing SQL Server 2008 Integration Services with SAP BI 7.0

http://msdn.microsoft.com/en-us/library/dd299430.aspx

SQL Server 2008 Business Value Calculator: www.moresqlserver.com

• Team Forum: http://social.msdn.microsoft.com/forums/en-US/sqlintegrationservices/threads/

Complete a session evaluation and enter to win!

10 pairs of MP3 sunglasses to be won

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.