SQL Server 2005Integration ServicesSQL Server 2005Integration Services

Dave GloverDave GloverMicrosoft AustraliaMicrosoft Australiahttp://blogs.msdn.com/dgloverhttp://blogs.msdn.com/dglover

AgendaAgenda

Cleanse Data InputSplit an Output ChannelScript to Branch ControlLookup Fuzzy ValuesLoop through FolderConfigure and Deploy

OverviewOverview

Part of SQL 2005 Std and Ent Editions:Integration Services with Basic Transforms. Provides graphical extract, transform, and load (ETL) capabilities Ent only: Integration Services Advanced Transforms Includes data mining, text mining, and data cleansing

Collaborative DevelopmentSeparate Management ToolsData Quality and profilingScalable

SSIS Runtime

WMIWMI

Sources

Flat FileFlat File

DestsOLED

B C

onnectio

n F

TP

HTTP

FIL

E O

DB

C M

anagers C

US

TO

M

FLA

TFIL

E

CustomCustom

XMLXML

SQLSQLServerServer

Flat FileFlat File

FTPFTP

FileFile

OLED

B C

onnectio

n F

TP

HTTP

FIL

E O

DB

C M

anagers C

US

TO

M

FLA

TFIL

EWEBWEBFTPFTP

DBDB

XML TaskFILE

FTP Task FTP

Web Services Task

HTTP

Data Flow Task

OD

BC

…

FILE

…

CustomCustom

Containers

WMI Event Task

WM

I

Event Handlers

Send Mail Task Send Mail Task

Variables Lo

g P

rov

ide

rs

Task HostTask Host

Custom TaskCustom Task

VariablesVariables Lo

g P

rov

ide

rsXMLXML

Cleansing Data InputCleansing Data Input

Connection-Related ObjectsConnection ManagerFlow TypesData AdapterMetadata ValidationData Flow Components

Flow TypesFlow Types

Two different types of flowControl Flow = Runtime = TasksData Flow = Pipeline = Transforms

Managed in DesignerUsed to be single view in SQL 2000Separate views in SQL 2005

Control Flow handles tasks and precedenceData Flow handles transformations - zoomed contents of Data Flow task

Data Flow ComponentsData Flow Components

Components

PathsData route between one component and the nextIncludes metadata about columns moving aroundLineage Identifier tracks item transformations

PipelineComponents connected by a path

Source Transform Destination

Splitting an Output ChannelSplitting an Output Channel

Distributors and CollectorsPrecedence Constraints

Distributors and CollectorsDistributors and Collectors

Distributor transformationsMulticastConditional Split

Collector transformationsUnion AllMergeMerge Join SrcSrc

TfmTfm

DstDstDD CCTfmTfm

TfmTfm

Precedence ConstraintsPrecedence Constraints

Connect one task to anotherGive sequential relationship to tasks

Success/Failure/Completion workflowEstablish concurrency

Connected sequence of tasks is a task listIndependent Task Lists execute concurrentlyTasks within a list execute sequentially

DemoDemo

Cleansing Data &Splitting an Output Channel

•Script and Branch Control•Script and Branch Control

Package VariablesScript TaskComplex PrecedenceContainers

Package VariablesPackage Variables

ScopeEach container can have variablesDefine namespace for user variablesContainers can access variables from higher levels

Accessible fromExpressions (such as loops and constraints) – Use @Parameters in Execute SQL taskParent Package (as part of configuration)Script

Script TaskScript Task

Currently VB .NetCan read or modify properties throughout the packageCan’t access inner workings of tasks or transformsCan’t modify pipeline metadata (e.g., number of columns piped)

ContainersContainers

Container providesGrouping of task lists (list of one is allowed)Transaction scopeVariable scope

A package is a containerAdd your own tasksInsert your own containersLoops are containers, too

Fuzzy Lookup TransformationFuzzy Lookup Transformation

Proximity algorithm to find matchesBuilds index – Index can persistCreates metrics

SimilarityConfidence

Uses a separate connection for reference table

LoopsLoopsFOR LOOP

Loops while expression is TRUEManually add loop counter

Init: @N = 1Eval: @N <= 25Increment: @N =@N + 1

Execute tasks in container on each iterationMore control—and more complex than For Each

FOR EACH LOOPLoops over set of objects

FilesXML nodesDatabase objects

Set variable (e.g. file name) for each iterationExecute tasks in container on each iteration

DemoDemo

Script and Branch Control

Configure and DeployConfigure and Deploy

ConfigurationsXML CustomizabilityDeploymentExecution

Package Lifecycle SupportPackage Lifecycle Support

Data VisualizersDebugging

Stop execution during packageStop control flow before and afterStop transformation during (visualizer)

LoggingEvent HandlersLog Providers

ConfigurationsConfigurations

Take something from the systemEnvironment variables, registry, XML option fileApply it to some part of your packageRun the package with the new setting

Useful for multiple “similar” jobsSimilar to Dynamic Properties from SQL 2000Facilitate reusability, different environments

Configurable at runtime or during execution

DeploymentDeployment

Same server deploymentDeploy from BI Development Studio

Multiple server deploymentDeployment UtilityPackage configuration files with packageDeploy to SQL Server (msdb) or file (dtsx)

ExecutionExecution

Command-line executionDTEXEC

User Interface executionDTEXECUICan generate command line for DTEXEC

SchedulingSQL Server Agent

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Report Builder OverviewReport Builder Overview

Report Builder ArchitectureReport Builder Architecture

SQL Server CatalogSQL Server CatalogSQL Server CatalogSQL Server Catalog

Web Service / URL interfaceWeb Service / URL interfaceWeb Service / URL interfaceWeb Service / URL interface

Report ServerReport Server

DeliveryDeliveryRenderingRendering

QueryQuery SecuritySecurity

Report BuilderReport BuilderClientClient

Report BuilderReport BuilderClientClient

ModelModelDesignerDesigner

ModelModelDesignerDesigner

Data Sources(SQL Server,

Analysis Services)

Drill through report generationDrill through report generation

Query generationQuery generation Model SecurityModel Security

Report ProcessingReport Processing

ReportReportDesignerDesigner

ReportReportDesignerDesigner

SQL Mgmt SQL Mgmt StudioStudio

SQL Mgmt SQL Mgmt StudioStudio

ReportReportManagerManagerReportReport

ManagerManager

Why Report Builder?Why Report Builder?

Report Report ConsumersConsumers

Business Business UsersUsers

Report Viewer

Report Builder

Power UsersPower UsersDevelopersDevelopers

Report Designer

What is Report Builder?What is Report Builder?A new ad-hoc report design tool for SQL Server Reporting Services Targeted at business users who want to find and share answers to interesting questionsDriven from a business model of the data so users do not need to understand the underlying data structuresNot a full analytical client or replacement for Pivot TablesFully integrated with Reporting Services and delivered in SQL Server 2005

Report BuilderReport Builder

Report Builder vs. Report DesignerReport Builder vs. Report DesignerReport Builder Report Designer

Targeted at business users Targeted at IT pros and developers

Ad hoc reports Managed reports

Auto-generates queries using semantic layer on top of the source

Native queries (SQL, OLE DB, XML/A, ODBC, Oracle)

Reports built on templates Free-form (nested, banded) reports

Click-once application, easy to deploy and manage

Integrated into Visual Studio

Cannot import Report Designer reports

Can work with reports built in Report Builder

Wrap-upWrap-upReport Builder is…

a new ad hoc report design tool for SQL Server Reporting Servicestargeted at business users who want to find and share answers to interesting questionsdriven from a business model of the data so users do not need to understand the underlying data structuresis fully integrated with SQL Server Reporting Services and delivered in SQL Server 2005

© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.