copyright 2001 advanced strategies, inc. 1 data bridging an overview prepared for digit 07-11- 2001...
TRANSCRIPT
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com1
Data Bridging
An Overview
Prepared for DIGIT 07-11- 2001
By Advanced Strategies, Inc.
Copyright 2001 Advanced Strategies, Inc. www.advstr.com2
What is Data Bridging?
A process that connects two or more fully specified data stores for a limited time or on an ongoing basis
D ataS tore
D ata B ridge D ataS tore
<S ource> <T arge t>
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com3
What is Your Data Bridging Experience?
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com4
Foundation Concepts
Copyright 2001 Advanced Strategies, Inc. www.advstr.com5
Why Do We Say Data Stores and Not Data Bases? A Data Store is any facility that is capable
of retaining data over time The data stores can be of any type:
– Flat files, Relational Databases, …– Manual or automated
Bridges can be theoretically built among any collection of data stores
Copyright 2001 Advanced Strategies, Inc. www.advstr.com6
Bridges are used:
In Data Conversions & Initial Data Loading As Ongoing Interfaces For sourcing Data Warehouses or Data
Marts ...
Copyright 2001 Advanced Strategies, Inc. www.advstr.com7
Bridging is Not Simple andIs Often Underestimated.
D irty D a ta
U n its
C lean bu tIncom p le te
M illions
Source(s) Target(s)Bridge
B ridg le t
Trigger
Data Bridge Basics
8/31/99 1:40:38 PM rhbBridg ingO vrV iew
C opyrigh t 2001 A dvanced S tra teg ies ,Inc .
w w w .A dvS tr.com
Copyright 2001 Advanced Strategies, Inc. www.advstr.com8
Key Concepts
D ata S tore
Bridges Can Be Batch
Bridges involve:Occurrences of Data Components
Bridges Can Be Real-Time
Copyright 2001 Advanced Strategies, Inc. www.advstr.com9
Key Concepts Continued
The target of one bridge can be the source of another
D ataS tore
D ata B ridge D ataS tore
<S ource> <T arge t>
D a ta B ridge D ataS tore
<S ource> <T arge t>
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com10
The Bridging Process
Copyright 2001 Advanced Strategies, Inc. www.advstr.com11
The Data Bridging Process
A bridge involves an integrated collection of activities
Each activity performs a specific function
Copyright 2001 Advanced Strategies, Inc. www.advstr.com12
The functions can be performed singularly across an entire set of data occurrences;
or collectively on a given set of data occurrences
The Data Bridging Process
Copyright 2001 Advanced Strategies, Inc. www.advstr.com13
What are the Minimum Processes in a Bridge?
Suppose there were multiple choices for possible source stores
SalesD ata
W arehouseLoadU nload
O rderSystem (C orpora te
O ffice)
(3 R egiona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com14
Adding Source Selection
Suppose we are not confident that the Source Data is pristine (enough)
S alesD ata
W arehouseS ource LoadU nload
O rderS ystem (C orpora te
O ffice)
(3 R eg iona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com15
Adding Source Data Audit
Suppose the Source Data is predictably flawed.
S alesD ata
W arehouseS ource LoadA ud it U n load
O rderS ystem (C orpora te
O ffice)
(3 R eg iona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com16
Adding Source Data Scrubbing
Suppose the Source Data is not in the units, size or format we need
Suppose the Target Data needs to be assembled from multiple Sources, ...
S alesD ata
W arehouseS ource LoadS crubA ud it U n load
O rderS ystem (C orpora te
O ffice)
(3 R eg iona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com17
Adding Transformation
Suppose we need to establish an audit trail for the data; or be able to provide the user with information about the source and/or quality of the Target Data.
SalesD ata
W arehouseSource LoadScrub TransformAudit U nload
O rderSystem (C orpora te
O ffice)
(3 R egiona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com18
Adding Logging
Suppose we need to assure ourselves and our users that the process has gone as expected and that the Target Data can be trusted
SalesD ata
W arehouseSource LoadScrub TransformAudit U nload
Log
O rderSystem (C orpora te
O ffice)
(3 R egiona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com19
Adding Validation
SalesD ata
W arehouseSource LoadScrub TransformAudit U nload
Log Valida te
O rderSystem (C orpora te
O ffice)
(3 R egiona lO ffices)
Copyright 2001 Advanced Strategies, Inc. www.advstr.com20
Sourcing Auditing Unloading Loading Scrubbing Transforming Logging Validating
Here are the possible activities in our bridging process;
Not all are always needed, but all should be carefully considered
Copyright 2001 Advanced Strategies, Inc. www.advstr.com21
The Data Bridging Process - The Framework
Sets o fD ata S tores and
C om ponents Aud it thedata
com ponent
Identifysources o f
data
AuditR ules R ejected
D ataC om ponent
U nload thedata
com ponent
Approved,H igher-C onfidenceD ata C om ponent
Approved,Low er-C onfidenceD ata C om ponent
U nloadedD ata
C om ponent
Scrub thedata
com ponent
ScrubbedD ata
C om ponent
Transformthe data
com ponent
Transform ationR ules
Transform edD ata
C om ponent
Load thedata
com ponent
Source toTarget
M appingU n-Transform ableD ata C om ponent
U n-Scrubab leD ata C om ponent
Log
Log sta tus,im age, e tc. Va lida te
resu ltingdata
com ponent
ScrubR ules
S ta tus, Im age, e tc.
D ata C om ponent
C orrectD ata C om ponent
E rroneousD ata C om ponent
Va lida tion R u les
SourceD ata S tore(s)
(Se lected Sources)
(C and idateSources)
TargetD ata S tore(s)
SourcingR ules U nload
R ules
LoadR ules
Logging R u les
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com22
A Bridging MethodologyA Few Notes
Copyright 2001 Advanced Strategies, Inc. www.advstr.com23
How to Approach a Bridging Effort? As a Fully Formed Subproject: think “Path
To A Solution” Ideally, initiated after the target data stores
have been designed
Copyright 2001 Advanced Strategies, Inc. www.advstr.com24
Abbreviated Path To a Solution for Bridges Define the bridge, especially the values Analyze the Source, Target and Specify
Mapping, Bridging Rules, ... Design the Bridge Realize the Bridge Implement & QA the Bridge Use & Maintain the Bridge
Copyright 2001 Advanced Strategies, Inc. www.advstr.com25
Bridges Involve All Aspects of a Business Object/Data - what are the data and
relationship occurrences we are moving Process - what are the bridging activities Event - what triggers the bridging activities Location - where is the data and how do we
transport it
Copyright 2001 Advanced Strategies, Inc. www.advstr.com26
Socio-Political - do we need permission to access the data, are there data practices mandates we must consider, can the data be misinterpreted or misused
Enabling-Medium - are there platform conversions required; technology bridging
Bridges Involve All Aspects -- Continued:
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com27
Summary
Copyright 2001 Advanced Strategies, Inc. www.advstr.com28
The Big Picture: A Summary
Copyright 2001 Advanced Strategies, Inc. www.advstr.com29
Data Bridging is not normally trivial or cheap
Bridging can make or break many types of projects; it should never be an after thought
A rigorous process can help make it manageable
Data Bridging can be treated as a series of subprojects
Conclusion
Copyright 2001 Advanced Strategies, Inc.
www.advstr.com30
Questions?