© 2006 IBM Corporation

Introduction to Federation Server

Alex LeeIBM Information Integration SolutionsManager of Technical Presales Asia Pacific

2

WebSphere Federation Server

Federation overview

Tooling support

Case studies

Summary and references

3

What if you could…

Using standard SQL and any tool that supports JDBC/ODBC…

while looking to the user like a single database

BI toolsBusinessAnalysis

MgmtReports

Access data anywhere in your enterprise

No matter where it resides

Regardless of what format it is in

Regardless of vendor

Without creating new databases and without disruptive changes to existing ones…

4

Then you could …Produce information needed by the organization faster

– Melbourne Health – built the worlds first solution to access public medical history –discoveries were made days after implementation versus months with the prior process.

Improve the productivity of your people

– Taikang Life - saved 90% in people costs to compile real time reports (1 instead of 10 people)

Reduce business process costs– Neckermann – reduced labor costs

equivalent to 5 full-time employees per year– Pioneer Display – production efficiency

increased over 25%

BI toolsBusinessAnalysis

MgmtReports

5

And without…

BI toolsBusinessAnalysis

MgmtReports

Building new databases for data you already have stored in multiple places

– Acquiring hardware and software infrastructure to support them

– Keeping them up to date– Keeping them secure– Assuring their reliability

and availability for the next 5-7 years

6

What is Federation?

Federation is an integration pattern that allows a collection of resources to be viewed and manipulated as if they were a single resource while retaining their autonomy and integrity. It is the technology on which EII is based.

7

IBM WebSphere Information ServerDelivering information you can trust

Parallel ProcessingRich Connectivity to Applications, Data, and Content

Information Server

Understand Cleanse Transform Federate

QualityStageInformation Analyzer Federation ServerDataStage

Information Services Director

Metadata Server

8

Data FederationTransparent

Appears to be one sourceIndependent of how and where data is storedApplications continue to work despite of any change in how data is stored

HeterogeneousAccesses data from diverse sourcesRelational, Structured, XML, messages, Web, …

Extensible Bring together almost any data source.Wrapper Development Toolkit

High FunctionFull query support against all dataCapabilities of sources as well

AutonomousNon-disruptive to data sources, existing applications, systems.

High PerformanceOptimization of distributed queries

9

Relationaldatabases

Web,Non-Relational

Sources

CollaborationSystems

XMLWeb services

Packagedapplications

SQL, SQL/XML

Content Repositories

and Imaging Systems

Workflow systems

Content

Mainframefiles

Mainframedatabases

SQL

Virtualized Information AccessAccess diverse and distributed information as if it were in one system

Single sign on – Unified views – Common language – Web services or Java API Query and update – Optimized access

Classic Federation Server for z/OS

Federation Server

II Content Edition

10

VSAMSequential

IMS AdabasCA-DatacomCA-IDMS

Federated Sources

DB2InformixOracleSybase Teradata Microsoft SQL Server ODBC

OLE DBExcelFlat filesLife sciencesCustom-built

DB2 CMFamilyDomino.doc DocumentumFileNetOpen TextStellentInterwovenHummingbird

WebSphereFileNet

Lotus NotesMicrosoft Index ServerIBM Lotus Extended Search

SametimeQuickPlaceMicrosoft Exchange

WebSphere BI Adaptors

SAPPeopleSoftSiebel

Plus partner tools and custom-built connectors extend access to more sources

Content& Imaging

Workflow systems

Relationaldatabases

WebOther

XMLWeb services

Packagedapplications

Mainframefiles

Mainframedatabases

SQL

ContentSQLFederation Server

Classic Federation Server for z/OS

II Content Edition

CollaborationSystems

11

Incorporate data sources using wrappers

Access to a particular class of data sources or protocols

Contains information about data source characteristics

High-function relational wrappers from IBM

Read/Write access

Clean, simple interface for nonrelational wrappers

Written by IBM, third parties, customers

Read Only

Data Federation Approach

Server2Server1

Nicknam

e

Nicknam

e

Nicknam

e

(OptionalLocal Data)

(remote data source 1)

(remote data source 2)

WrapperB

Server3

Nicknam

e

Wrapper A

Table

Federated views

(remote data source 3)

12

Powerful query processing engine in federated server

Decomposes, rewrites and distributes queries

Cost-based optimizer chooses query plan with pushdown as appropriate

Query execution engine drives wrappers, combines results

Compensates for missing function in data source

Invokes functions at remote sources as needed

Data Federation ApproachN

ickname

Nicknam

e

Table

Rel. WrapperClient library

NR. WrapperClient library

Local + Remote Execution Plans

Federated Server

Non-SQL

DB2 cost-based optimizer

13

Agenda

Federation overview

Tooling support

Case Studies

Summary and References

14

Tools for modeling

Visualize and define mappings between remote schema and federated schema

Generate federated schema based on transformations and joins– Nicknames

– Views

Simplify creation of virtual schemas

15

Control Center– Tools to configure and

administer standard wrappers

– Plug-in architecture allows custom wrappers to be administered

Administration Tools

16

Tools help manage the complexity

Configuration wizard– Guides you through federation

configuration process

Discovery– Server discovery: Automatically

discovers and configures external servers

– Nickname discovery

Deploy– Capture configuration to a script,

save and deploy

– Facilitates cloning system configuration for horizontal scaling...

17

WFS tools help manage the complexity

Health Monitoring– Monitor health of servers, nicknames that affect

configuration

Statistics refresh– Refresh nickname statistics on demand or by

scheduled task

Snapshot Monitoring– Snapshots are useful for determining the status of

a database system.

Event Monitoring – Collect information about the database and any

connected applications when specified events occur.

18

Agenda

Federation overview

Tooling support

Case Studies

Summary and References

19

When to use Federation

Too big - Data from multiple sources is just too big to integrate on a permanent basis

Too ad hoc - Data is too varied and unpredictable to make an ETL process worthwhile

Too proprietary - Data is owned by disparate entities/organizations that do not want to support ongoing ETL processes

Too recent - Data from multiple sources is required that must be current or must not be updated while being read

Application or tool does not support native access to the sources being accessed

20

Providing on-demand relational access to multiple types of dataRequirements

Warehouse users need access to up-to-the-minute data from an external sourceExternal data may not be integrated into the warehouse until later (or never) because it is not practical or possible/allowed

SolutionAccess remote data via Federation Server nicknames and combine with DW dataAlso enables prototyping of ETL development

Federation Server

External Web services

Client

Data Warehouse

ODSFlat files or spreadsheets

21

Enabling transparent drill-through to detail data from summary data

Requirements

Users query a summary warehouse that is fed from a detail warehouse by an ETL processMost queries satisfied from summary alone. Some need to retrieve detail data after initial filtering by the summaryBe able to retrieve detail on-demand within the context of the summary query without making a new connection

Solution

Extract, transform and load relevant source data into summary and detail databases using bulk data movementUsers query summary data using existing applicationDetail data visible on-demand from summary warehouse using federation

Federation Server

DataStageClient

DataStage

Summary Database

Bulk data

Bulk data

requestsDetail Database

requestsdatadata

22

Unified view of regionally distributed data with same data model

RequirementsSeveral regional databases with similar logical data models, but unique dataApplication needs to see the data as one large database with a single schemaImpractical to physically consolidate data

SolutionAccess relevant remote tables via Federation Server nicknamesConnect matching nicknames from different sources via a UNION ALL viewCan optionally cache common data at the federated server or create local aggregates

Client

Federation Server

Seattle

Phoenix

San Jose

23

Placement, Consolidation, and Access Choices

ETL or replication preferred: – Centralized data needed for access performance or availability

• Complex, multi-dimensional queries

– Point-in-time consistency needed e.g. close of business

– Complex transformation needed for semantically consistent data

Federation preferred:– Access performance and load on sources traded for overall lower cost

• Queries returning small result sets among federated systems• Large volume data that is infrequently accessed• Data that changes rapidly

– Business requirements demand current data

– Data security, licensing or regulations restrict data movement

– Unique functions must be accessed at data source

– Data semantics consistent and managed across domains

– Read/write access is required

24

Agenda

Federation overview

Tooling support

Case Studies

Summary and References

25

Value of Federation

Speed time to market for new applications– Simplify and enrich portal development

– Reduce hand-coding by half

– Reduce skills requirements

– Use familiar SQL programming model and existing tools

– Build on a standards-based, strategic integration platform

Enhance value and insight from existing assets and applications– Work within your existing infrastructure

– Extend existing warehouses

– Combine existing data and content assets in new ways

– Facilitate cross-divisional reporting

Increase control over IT costs– Reduce need to rip and replace

– Reduce need to manage redundant data

26

ReferencesFor product information on WebSphere Federation Server

http://www.ibm.com/software/data/integration/federation_serverFor the system and data source requirements:

http://www.ibm.com/software/data/integration/federation_server/requirements.html

WebSphere Federation Server V9.1 infocenter:http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp

Some whitepapers on federation technology:“IBM Federated Database Technology”:

http://www.ibm.com/developerworks/db2/library/techarticle/0203haas/0203haas.htmlTwo-part series on using data federation technology:

http://www.ibm.com/developerworks/db2/library/techarticle/dm-0506lin/http://www.ibm.com/developerworks/db2/library/techarticle/dm-0507lin/

“Maximizing the performance of WebSphere Information Integrator with MQTs”:http://www.ibm.com/developerworks/db2/library/techarticle/dm-0605lin/

“Use federated procedures in WebSphere Federation Server”:http://www.ibm.com/developerworks/db2/library/techarticle/dm-0605bhatia

27