david besemer, cto on demand data integration with data virtualization

David Besemer, CTO

On Demand Data Integration

with Data Virtualization

2

Agenda

State of Enterprise Information

The Case for Data Virtualization

How Data Virtualization Works

Data Virtualization Adoption Patterns

© 2011 Composite Software, Inc. / Composite Proprietary and Confidential

3

The State Of Enterprise Information

More demanding business users Competition drives faster time-to-information Younger staff want more “do-it-yourself” “IT’s challenges are not my problem.”

Information overload Exponential data volume growth Omnipresent delivery

“Over the top” IT complexity New sources, uses, and enabling technology Layered on byzantine IT infrastructures


4

Data Management Trends

Changing role of the Data Warehouse Data warehouse no longer viewed as only focal point for

all data integration

Lower latencies required Information needs moving toward real time

Rising “fit-for-purpose” storage and processing Appliances, MPP, NoSQL

Data Quality being addressed at every layer Source, Consolidation, Virtual, and Visualization

Clouds are approaching… Most enterprises looking to leverage cloud computing


5

Agenda






6

The Challenge

Big DataFiles Packaged Applications

Web Services

RDBMS

BI, CPM, andReporting

Custom and Composite Apps

Portals andDashboards

SOAInitiatives

SourceData

Siloed & Rigid

ConstantChange

BusinessSolutions

Data IntegrationChallenge


7

Traditional Physical Data Consolidation

Big DataFiles RDBMS Web Services

Packaged Applications

Enterprise DataWarehouse

PhysicalData Marts

Physical OperationalData Stores

PhysicalIntermediate Stores & ETL Middleware

SourceData

BusinessSolutions




SOAInitiatives


8







SOAInitiatives


PhysicalData Marts



SourceData

BusinessSolutions

More silos & complexity Slows future IT progress

Physical consolidation Forces the business to wait

longer for solutions

Wait, wait, wait!

Uncontrolled data replication Reduced data quality Significant hidden costs

$$$

$ $$$

$$$

$$

$$$

$

Batch integration Delay real-time information

Customer X

Invoice

UNPAID

Customer X

Invoice

PAID IN FULL

Batch Data

On-Demand Data

OLD


Data Integration Architectures and Patterns: Build a Portfolio to Address the Range of Needs

Physical Movement and Consolidation

(e.g., ETL)

Abstraction/Virtual Consolidation

(Data Federation)

Change-Capture and Propagation (Replication or

Messaging)

Common Metadata (Location, Format, Structure, Quality, Meaning)

Common Connectivity (Full range of source/target types)

BI Tools/Apps. Master Data Mgmt. Operational Apps. Interenterprise

Leading organizations support multiple styles of data integration and delivery to address a range of business requirements — breadth enables leverage and agility.

Com

mon D

esign, Adm

in., Go

vernance

10© 2010 Composite Software, Inc. / Composite Proprietary and Confidential

Physical Movement and Consolidation

(ETL, CDC)

Abstraction / Virtual Consolidation

(Data Federation)

Middle-ware ETL CDC Data Virtualization EAI / ESB

Purpose

Attribute

How Data Virtualization Differs

Synchronization and Propagation

(Messaging)

DB DB

ScheduledEventDriven

Application ApplicationDB Application

On DemandEventDriven

DB DB


11







SOAInitiatives

PhysicalData Marts





12

Data Virtualization Increases Agility






EnterpriseSearch

PhysicalData Marts


VirtualData Marts

Virtual OperationalData Stores


DataVirtualization



13

Shared Data Services & Relational Views Further Extend Flexibility and Agility



PhysicalData Marts


Virtual Data Layer

VirtualData Marts


Web Data Services& Relational Views


CompositeInformation

Server





SOAInitiatives


14

A Complete Data Integration Architecture






SOAInitiatives

Physical Data Consolidation Layer

Virtual Data Layer

VirtualData Marts


Shareable Data Services& Relational Views

PhysicalData Marts




15

Forrester Data Management Reference Architecture


16

Agenda






17

How Data Virtualization Works – Example Scenario

1) I need to build an application that looks like this…

2) The view or data service needs to look like this…

3) And the data comes from these sources, in these formats…


18

Composite Information Server

Studio

Data Discovery and Design

Design Steps1. Discover data and relationships

2. Model individual view/service

3. Validate view/service

4. Modify as required

BenefitsFaster time to solution

Easy to learn and use

Extensible / reusable objects

Discovery


19


Data Virtualization Production

Production Steps1. Application invokes

request

2. Optimized query (single statement) executes

3. Deliver data in proper form

BenefitsUp-to-the-minute data

High performance

Less replication required

Optimizer


20


Data Virtualization Production with Caching

Production Steps1. Cache essential data

2. Application invokes request

3. Optimized query (leveraging cached data) executes

4. Deliver data in proper form

BenefitsRemoves network constraints

7-24 availability

Optimal performance

CacheOptimizer


21

Agenda






22

DataFederation DW Extension

Cloud DataIntegration


Data VirtualizationLayer

Big DataIntegration


23

DataFederation

Data Federation for Business Intelligence

“My application requires data from multiple incompatible sources.”

Project Manager


24

“My data warehouse does not contain all the data required for the reports we need to build.”

Data WarehouseExtension

Data Warehouse Extension for 360o View

Data WarehouseOwner


25


Data Virtualization Layer for Business & IT Agility

“How do I build an agile data layer for easy data access and delivery.”

IT Director


26


Cloud Data Integration for IT Extensibility

“I need to integrate data between on-premise systems and applications running in the cloud.”

CIO


27

“More and more of my data now lives in MPP and Hadoop sources. How do I combine big data with traditional data for analysis?

Big DataIntegration

Big Data for Analytics

Business Analyst


28

DataFederation DW Extension




Big DataIntegration

Semantic Abstraction Federated Query Loose Coupling Caching Location Independence

= DataVirtualization


29

Composite Software Contact

For more information please contact:

Pamela Sotnick Director, Federal AccountsMobile [email protected]

Katy MannDirector, Federal AccountsMobile 301.452.7042 [email protected]

David [email protected]

mailto:[email protected]

Questions

david besemer, cto on demand data integration with data virtualization

Documents

data warehouse data

data virtualization

demand data integration

nosql data quality

data management trends

composite software

composite proprietary

composite apps portals