oracle data integrator
Post on 08-Dec-2015
51 Views
Preview:
DESCRIPTION
TRANSCRIPT
Oracle Data Integrator A Success Story
Jason Jones Key Performance Ideas, Inc.
About Jason Jones
l Oracle Essbase Certified Developer – experience since 2005
l Oracle Data Integrator since 2009 l Extensive experience in developing, tuning and
enhancing Essbase, Hyperion Planning, and ODI
l Programming expertise: ● Developing software in Java ● Mobile solutions for the iOS platform (Objective-C) ● Relational databases (SQL Server, Oracle) 2
Agenda
l Company Situation l What is Oracle Data Integrator? l Problems & Pain Points l Solutions & Opportunities l ODI Functionality & Benefits l Lessons Learned l How to Get Started l Q&A
3
Company Situation
l Large health services company l Significant Oracle database implementation l In process of implementing Essbase l Transitioning to new database servers l Data movement processes have grown
organically over the years
What is Oracle Data Integrator?
l History ● Originally developed by Sunopsis, acquired in 2006 ● Originated Extract Load Transform (ELT) as
alternative to ETL ● Data movement between heterogeneous data
sources l Future
● Strategic product for data integration ● Significant development resources
ETL vs. E-LT
Source ETL Server Target
Target
ETL Server
Traditional Architecture
E-LT Architecture
Source
Problems & Pain Points
l SQL procedures being used as de facto ETL solution l Time-consuming to develop l Hard to maintain l Little error visibility l Difficult to integrate with other data sources l Have to maintain in multiple environments l Manual logging facility l Lacking documentation/comments l Difficult to troubleshoot
Solutions & Opportunities
l Easier development and maintenance l Work with different relational database
technologies l Load to/from text files, Essbase/Planning l Email alerts l Easily switch between physical environments l Process logging, easily pinpoint errors
Essbase
Typical Data Flow
ODI Server
Data Movement OLTP
(Oracle)
Operational Data Store
(Oracle)
Knowledge Modules
l RKM: Import existing table/data structure l LKM: Load data from tables l IKM: Insert/update data in target table l JKM: Process new rows of data only l CKM: Validate data being moved
Models
• Create models for relational tables, text files, and Essbase dimensions to load data to and from
• Easily create models automatically based on existing tables with RKM
• Serve as the source and target of interfaces
Enhanced Functionality
• Enforce data integrity! • Exist as logical units irrespective of physical environment
Benefits
• Define requirements for data entering target tables • Reuse for multiple interfaces
Health Services Organization Value Achieved
Model Datastore
12
Interfaces
• Moves data between models • Source is typically multiple models (tables) • Destination is always one and only one model (table) • Declarative!
Enhanced Functionality
• Easy to create, modify, and maintain • Work across different technologies • Document business rules • Leverage power of existing database server to transform data • “Auto mapping” speeds development time
Benefits
• Visual replacement for many SQL procedures • Quickly define criteria for updating tables • Choose location of data transformation
Health Services Organization Value Achieved
Interface Screen
14
Interface Flow
• Validate data being processed • Automatically recycle data • Choose loading/integrating strategy
Enhanced Functionality
• Very easily configure complex actions that would normally be tons of code
• Select Knowledge Modules (strategy used to load, integrate, validate)
Benefits
• Turn on data validation, error recycling with mouse click
Health Services Organization Value Achieved
Interface Flow Screen
16
Procedures
• Useful for performing one-off actions if they can’t be put into interface
• e.g. call a stored procedure
Enhanced Functionality
• Perform action that doesn’t fit into an interface
Benefits
• Reference existing data transformation process without having to rebuild it
Health Services Organization Value Achieved
Interface Overview
18
Procedure Steps
19
Procedure Step Definition
20
Packages
• Chain together multiple interfaces, procedures, and other steps • Allow for error control flow
Enhanced Functionality
• Gracefully handle errors • Restart process automatically, if desired • Send an email alert!
Benefits
• Treat execution of multiple interfaces as one job • Add email step in case of error (or success)
Health Services Organization Value Achieved
Package Flow
22
Scenarios
• “Freezes” an interface, procedure, or package in place • Changes can be made to procedure/interface/package that won’t
affect existing functionality • Scenarios are unit of work that can be scheduled and called from
command-line
Enhanced Functionality
• Avoid breaking existing processes when need arises to change/augment functionality in interface
Benefits
• Call ODI functionality from command-line when needed • Deploy functionality and not be scared to make changes to it
Health Services Organization Value Achieved
Generated Scenarios
24
Operator
• Easily view status of all jobs • Scenarios, packages, interfaces, procedures, load plans • Replaced need to have manual logging statements
Enhanced Functionality
• See exactly where and why a process failed
Benefits
• Insight into currently executing and already executed jobs • Drill down to exact cause and reason of error
Health Services Organization Value Achieved
Operator Overview
26
Operator Step Detail
27
Operator Step Generated Code
28
Scheduler
• More robust than Windows Task scheduler • Easy to set schedule for jobs to run • Can call jobs from command-line but use scheduler if possible!
Enhanced Functionality
• Easier to use than setting up Windows Task Scheduler to run a batch file to run a scenario…
Benefits
• Directly schedule ODI job to run without having to setup batch file • Run ODI jobs without needed additional deployment step
Health Services Organization Value Achieved
Scheduler
30
Journalized Data
• Pattern for only consuming updated/inserted rows of data • Easy to implement • Single checkbox in interface for using journalized data
Enhanced Functionality
• Avoid processing all data, block of data by day • Avoid maintaining timing variables
Benefits
• Get away from “day of data” processing paradigm • Move data from one system to another more often
Health Services Organization Value Achieved
Journals
32
Journalized Data in Interface
33
Topologies
• Logical “Customer” system • Physical Customer system in Development, QA, Production
Enhanced Functionality
• Use same logical job for both development and production environments
Benefits
• Save significant effort not having to copy/deploy code only differing by environment
• Pick and choose test/production systems
Health Services Organization Value Achieved
Lessons Learned
l Critical to leverage someone who has done this to lay foundation! Leverage an experienced individual to set critical first steps up correctly
l Start with simple ETL job and build a roadmap to larger data movement needs
l First step/phase can be large, subsequent jobs much easier l Don’t re-implement functionality, build idiomatic ODI jobs l Always try to use iterative development model l Reusability, consistency, maintainability l Implementation duration l What to look for
How To Get Started
l ODI is now Oracle’s data movement tool standard l Serious thought must be made to implement platform -
initial step is typically an architectural discussion l Utilize experts to help build a roadmap and identify new
idiomatic ODI functionality l Initial functionality development
● Software install, physical topologies, logical topologies, models reverse engineered
● Interfaces, procedures, packages, scenarios, solutions
37
WALKTHROUGH OF SQL TO SQL INTERFACE STEPS
Drop Work Table
drop table ZODI.C$_0DIM_PETS
38
Lock Journalized Table
update ZODI.J$PETS set JRN_CONSUMED = '1' where (1=1) And JRN_SUBSCRIBER = ‘LAB_RESULTS_TRANSFER_PETS’
39
Create View/Table on Source
create or replace view ZODI.C$_0DIM_PETS (
C1_PET_ID, … ) as select * from ( select
PETS.PET_ID C1_PET_ID, … from ZODI.JV$PETS PETS where (1=1) And JRN_SUBSCRIBER = ’LAB_RESULTS_TRANSFER_PETS’
40
Drop Synonym on Target
drop synonym ESSBASE.C$_0DIM_PETS
41
Create Synonym on Target
create synonym ESSBASE.C$_0DIM_PETS for ZODI.C$_0DIM_PETS@ZOASIS
42
Drop Flow Table
drop table ESSBASE.I$_DIM_PETS
43
Create Flow Table
create table ESSBASE.I$_DIM_PETS (
PET_ID NUMBER(38) NULL, PET_NAME VARCHAR2(32) NULL, … JRN_FLAG VARCHAR2(1) NULL, JRN_DATE DATE NULL, DIM_PET_ID NUMBER NULL ,IND_UPDATE char(1)
) NOLOGGING
44
Insert Flow Into Table
insert /*+ APPEND */ into ESSBASE.I$_DIM_PETS ( PET_ID, ………… ,IND_UPDATE )
select C1_PET_ID,
from ESSBASE.C$_0DIM_PETS where (1=1)
45
Analyze Integration Table
begin dbms_stats.gather_table_stats(
ownname => 'ESSBASE', tabname => 'I$_DIM_PETS', estimate_percent =>
dbms_stats.auto_sample_size ); end;
46
Synchronize Deletions from Journal Table
delete from ZSTAGING.DIM_PETS where exists (
select 'X' from ESSBASE.I$_DIM_PETS I where ZSTAGING.DIM_PETS.PET_ID =
I.PET_ID and IND_UPDATE = 'D' )
47
Create Index on Flow Table
create index ESSBASE.I$_DIM_PETS_IDX on ESSBASE.I$_DIM_PETS (PET_ID) NOLOGGING
48
Merge Rows
merge into ZSTAGING.DIM_PETS T using ESSBASE.I$_DIM_PETS S on (
T.PET_ID=S.PET_ID )
when matched then update set
T.PET_NAME = S.PET_NAME, … when not matched then insert
( T.PET_ID,…
49
Commit Transaction
/*commit*/
50
Drop Flow Table
drop table ESSBASE.I$_DIM_PETS
51
Cleanup Journalized Table
delete from ZODI.J$PETS where JRN_CONSUMED = '1' And JRN_SUBSCRIBER = ‘LAB_RESULTS_TRANSFER_PETS' /* AND JRN_DATE < sysdate */
52
Drop Synonym on Target
drop synonym ESSBASE.C$_0DIM_PETS
53
Drop View/Table on Source
drop view ZODI.C$_0DIM_PETS
54
Jason Jones Direct: 206.427.1373 Email: jjones@KeyPerformanceIdeas.com
Thank You!
top related