the growing practice of operational data integration...
TRANSCRIPT
Philip Russom
Senior Manager, TDWI Research
April 14, 2010
The Growing Practice of
Operational Data Integration
2
Sponsor:
3
Speakers:
Philip Russom
Senior Manager,
TDWI Research
Gavin Day
VP of Operations and
Information Technology,
DataFlux
4
Agenda
• Defining Operational Data Integration (OpDI)
– Three Practices of OpDI
• Growth of OpDI
• OpDI Project Types
• Organizational Issues for OpDI
• Software Automation for OpDI
• Recommendations
5
Definitions of Operational Data Integration
• OpDI is manifested in diverse project types, so it’s hard to define OpDI with one, simple description.
• Sometimes defined negatively from BI perspective as:
– “non-data-warehouse data integration work”
• POINT: OpDI rarely has much connection to BI/DW
• More positive definitions based on what OpDI does:
– “the exchange of data among operational applications, whether in one enterprise or across multiple ones”
• Applies to some forms of OpDI, like data synchronization and business-to-business data exchange
– “data and database maintenance work that relocates datasets or database management systems”
• Applies to OpDI forms, like data migrations, consolidations, collocations, and database mgt system upgrades.
6
Data
Integration
Practices
Business Initiatives and
Technology Implementations
Customer Data
Integration (CDI)
Product Information
Mgt (PIM)
Master Data
Mgt (MDM)
Data
Warehousing
(DW)
Business
Intelligence
(BI)
Data Migration
Business-to-Business (B2B)
Data Exchange
Data Synchronization
Analytic
Data Integration
(AnDI)
Hybrid
Data Integration
(HyDI)
Operational
Data Integration
(OpDI)
Three Broad Practice Areas of Data Integration
Focus of
today’s
Webinar
7
The Three Main Practice Areas within OpDI
1. Data Migration
– Four types of database maintenance projects:
• Database migrations, consolidations, and collocations,
plus database management system upgrades
2. Data Synchronization (or data sync)
– Usually about syncing customer data across multiple customer-
facing apps, e.g., CRM, SFA, call center, etc.
– Also enables active-active and multi-master database
configurations for database high availability
3. Business-to-Business (B2B) Data Exchange
– Exchanging documents and files across diverse organizations
involved with supply chains, monetary exchanges, procurement,
fulfillment, BPO outsourcing, etc.
8
OpDI is a Fast-
Growing Practice
• Over time, OpDI has grown as a percentage of overall DI work, relative to AnDI:
– 81% AnDI and 19% OpDI in 2004
– 51% AnDI and 49% OpDI in 2008
– Rounded: 80/20 to 50/50 split
• OpDI growth is a dramatic change that only took four years.
– OpDI needs more people and budget to keep pace with growth.
• Both AnDI and OpDI are growing.– OpDI is growing faster than AnDI.
– Both practices need resources.
• DI work is getting more varied and projects are more numerous.
– DI projects for both BI/DW & op.apps are getting bigger.
51%
63%
75%
61%
75%
81%
49%
37%
25%
39%
25%
19%
December 2008
(Source: TDWI Report Survey)
August 2008
(Source: TDWI Tech Survey)
November 2007
(Source: TDWI Tech Survey)
February 2007
(Source: TDWI Tech Survey)
February 2006
(Source: TDWI Tech Survey)
November 2004
(Source: Forrester Wave Report)
Analy tic
Data
Integration
(AnDI)
Operational
Data
Integration
(OpDI)
“With data integration usage in your
organization, what is the approximate
percentage split between
analytic DI versus operational DI?”
9
Drivers for OpDI Growth, per Practice
• Data Migration
– As number of databases increases, maintenance does, too
– Databases proliferate and age, so need migration/consolidation
– Mergers/acquisitions & reorganizations force migrations
• Data Synchronization
– As customer-facing apps increase, data sync does, too
– 360-degree views are becoming the norm
• Business-to-Business (B2B) Data Exchange
– For firms with active supply chain, B2B data exchange is critical
– As business goes more global, as global firms integrate regional business units more, B2B data exchange increases(to reach internal units, plus external partners & customers)
– Many old solutions; firms are forced to update/replace them
10
Why care about OpDI Now?
• OpDI is a growing practice.
– It’s an increasing percent of DI workload.
• OpDI solutions need improvement or replacement.
– Many are hand-coded legacies; replace them with vendor tools.
• OpDI solutions tend to be feature poor.
– Augment them with missing functions for data quality, master data,
scalability, architecture, Web services, development tools.
• OpDI and AnDI have different goals and sponsors.
– Don’t assume you can do both with same team, tools, budget.
• OpDI addresses real-world problems, especially with change.
– Data migration is essential to enterprise transformation.
• OpDI supports mission-critical applications.
– Enterprise comes to a halt without data sync & B2B data exchange.
11
Data Migration Project Types*• Database Migration
– From database platform A to platform B.
• It’s more like new dev than migration.
• Database Mgt System Upgrade– Version A of Brand X upgrades
to version B of Brand X.
• Database Consolidation– Multiple, similar databases consolidate
into a single one, with a single data model.
• Database Collocation
– Multiple databases are copies into a single DBMS instance (typically on a singlehardware server), without merging data models.
* Synonyms = implementations, initiatives, jobs, tasks, assignments…
Many-to-One
Redundant
Databases
One Data
Model
Many-to-One
Redundant
Databases
Multiple
Data Models
One-to-One
Brand A:
Old and/or non-
Standard
Brand B:
New and/or
Standard
One-to-One
Version A
of Brand X
Version B
of Brand X
12
Data Migration Projects• These projects are in the context of:
– General database maintenance or data lifecycle mgt
– In support of applications maintenance• E.g., when application instances are consolidate or apps are migrated
to a new platform, application databases are, too.
– Reduce the number of databases to reduce cost
– Migrate data to more modern and/or standard platforms
• Data migration projects kill off systems– Far more intrusive than data sync
• Can be a multi-phase project that runs for years– Coordination among data & app owners, sponsors, business
managers, end-users, multiple IT teams, etc.
• Multiple projects may be combined:– E.g., before consolidating several databases, migrate some of
them to common platform & upgrade others to a desirable version
– Collocate several databases before consolidating them into one; typical of data mart consolidation projects
13
Data Synchronizationas a form of Operational Data Integration
• Most common data sync solutions:
– Sync customer data across customer-facing apps
– Sync operational data across ERP instances and other apps
– Sync primary and secondary databases for high availability
• Data Synchronization and Replication overlap somewhat
– The two use similar technologies and get similar results
– Most data sync solutions are built with replication technology
– Other tech’s can also implement or contribute to data sync
• Including EAI, ETL, and hand coding
– Data sync solutions involve advanced capabilities rarely seen in
simple replication solutions. (See next slide.)
Many-to-Many
One-to-One
14
Advanced Capabilities of
Data Synchronization
• Multi-directional data flow.
– Usually bidirectional, not just one-way
– May also be multi-directional
• Conflict resolution.
– Data value conflicts arise when syncing
– Supports rules for resolving conflicts
• Heterogeneous sources and targets.
– Handles many database brands, app & file types
– Supports appropriate interfaces for all these
• Data transformation.
– Normalizing, merging data from numerous
heterogeneous data models
15
B2B Data ExchangeAs a form of Operational Data Integration
• What it does:
– Documents or files containing data are exchanged between two or more organizations
– Examples: flat files, XML files, EDI documents, spreadsheets
• Relationship of organizations exchanging data varies:
– Exchange data between business units of the same company
– Among companies in a B2B relationship
• Most B2B relationships involve exchanging hard goods:
– Manufacturing, retail, consumer packaged goods, etc.
• Other relationships involve exchanging information or money:
– Financial services, insurance, healthcare, government, etc.
• B2B Data Exchange enables communication among organizations that cannot communicate data directly:
– Two organizations have very different data models
– So data exchange usually involves third, intermediary model
16
Applications
• Operational
• Transactional
Applications
• Operational
• Transactional
Data
Warehouse
Data Integration
• Get enterprise data
• Transform it into
exchange format
Data Quality
• Standardize
• Verify
Data Integration
• Get exchange data
• Transform it into
enterprise models
Data Quality
• Verify, re-standardize
• Process exceptions
Miscellaneous
Data Sources
Intermediate
Data Models
• XML-based
open standard
• Other standards
• Proprietary data
model to which
partners agree
Business A Business BB2B Data Exchange
B2B OpDI Hinges on Intermediary Data Models
2
Know and support the
“lingua franca” that all
organizations can speak.
Transport datasets with
security and scalability.
1
DI routines transform data
from enterprise models
into the appropriate
exchange model.
3
DI routines transform data
from exchange models
into the appropriate
enterprise models.
DataFlux: OpDI in Action
• Foundation in data governance – Ability to create, enforce and monitor business rules across your
organization
– Visibility and transparency to ensure your data reflects your business
• Business and IT collaboration– Business builds the rules; IT manages and applies the rules
– IT and business goals and process alignment
• Build once, use repeatedly– Utilize web services and other industry standards to simplify the
integration and synchronization process
– Deliver a foundation for repeatable data integration programs (and realize rapid time-to-value)
The Tools to Succeed
Data Migration
19
A US-based manufacturer of document management products needed a more unified view of its expanding European operations
Goal Create a single instance of an enterprise resource planning (ERP) application to better manage operations worldwide.
Challenge The company’s European group had grown through a mixture of organic growth and M&A. As a result, the operations group was using 20+ ERP systems across the European region – each localized to a different country.
Solution Performed a pilot data integration project to create a single set of business rules for migrating data, then expanded that to bring other geographies online.
Results The European operations finally had a way to look across the continent to make decisions. Logistics were improved due to shared resources, and the company was able to capitalize on new markets more rapidly.
Leading provider of automotive information seeks to provide more accurate data on transactions from millions of consumers.
Goal Build a data quality and data integration backbone that validates data as it enters operational systems.
Challenge With data coming from a variety of internal and external data sources, the company struggled with:
• Inconsistent data that was not standardized across systems
• Millions of records that could not easily be merged or consolidated effectively
Solution Used real-time data integration technologies (with web services on an EAI backbone) to validate data entered into systems prior to moving it to a system of record.
Results Data is validated as it enters the target systems – providing consistent, accurate and integrated data in a timely fashion.
Data Synchronization
Clinical research organization (CRO) with hundreds of clinical trials underway – at any time – across the globe.
Goal Understand information on patients, doctors and projects across systems.
Challenge Data on clinical trials was kept in different applications and databases. This organization wanted to make use of that data, but each database had its own standards and nomenclatures, making integration difficult.
Solution Used a combination of batch and real-time data integration technologies to deliver a unified master data hub. The project started with data profiling, moved to data quality/ governance, and finished with master data management that synchronizes enterprise data.
Results Delivered a more unified view of trials, which helps the organization make smarter decisions about future projects.
Data Exchange
Data Management Methodology
22
23
Staffing OpDI
Source: TDWI Report Survey of late 2008, 336 respondents.
“For DI outside data warehousing, who
does the work most often?”
“How many full-time people were
assigned to your most recent OpDI
project?”
Source: TDWI Report Survey of late 2008, 252 respondents.
“Which of the following
best describes the frequency
of OpDI work?”
24
Tools and Techniques for OpDI
Source: TDWI Report Survey of
late 2008, 252 respondents.
“In addition to the preferred technology
you selected in the last question, which
of the following techniques and tool
types were used?”
“In your organization, what is the
preferred technology for most
operational DI projects?”
25
Recommendations
• Expect both OpDI and AnDI to grow; OpDI needs more resources.
• Revamp your OpDI solutions for future success.
– Many are hand-coded legacies that need replacing
– Others lack modern functions for quality, master data, services, cloud
• Apply multiple OpDI project types in tandem.
– Collocate or upgrade databases before migrating/consolidating them.
– Sync old and new databases during a migration/consolidation.
• Foster data standards; these are key to B2B data exchange.
• Avoid staffing OpDI from the data warehouse team.
– They have great skills, but you delay important BI work.
• Consider a data integration competency center.
– It provides staff, shared services, other resources for all DI practices.
• When OpDI work is intermittent:
– Outsource it. Instead of permanent staff, hire consultants
– Look for tools licensed via short-term contract, open source, SaaS, or Cloud
26
One last recommendation…
Get a free copy of
TDWI Report on
Operational Data
Integration
• Download the report
in a PDF file at:
www.tdwi.org/research/
reportseries
• It goes into more detail
than today’s Webinar,
so read it.
27
Questions??
28
Contact Information
If you have further questions or comments:
Philip Russom, TDWI
Gavin Day, DataFlux