milomir vojvodic oracle dis
TRANSCRIPT
Oracle GoldenGate and Oracle Data IntegratorMilomir Vojvodic
Oracle Data Integration Solutions
Oracle Applications
Custom Applications
MDM Applications
Business Intelligence
Activity Monitoring
SOA Platforms
Comprehensive Data Integration SolutionSOA AbstractionProcess Manager Service Bus Data Services
Data Federation
Oracle Data Integrator E-LT/ETL
Oracle GoldenGate
Oracle Data Quality Profiling Cleansing Parsing
Real-Time Data Integration
Storage
Data Warehouse/ Data Mart
OLTP System
OLAP Cube
Flat Files
Web 2.0
Web and Event Services, SOA
DIS products vs. out of the box
Oracle OOTB
DIS Oracle strategic productsOracle GoldenGate
Difference?- Heterogeneous - Scalability, performance, reliability - Best Productivity - Makes HA/DR investments work - Best MAA solution for 11g - Provides reporting benefits
Streams
Data Guard
Active Data Guard
OWB
Oracle Data Integrator
- Increased ROI - Broader support/heterogenity - Best productivity & scalability
Maintenance Mode
Oracle Excellent Product ViabilityWeight -ingIBM Oracle
Informatica
iWay
SAP
SAS
Micro -soft
Bulk data movement
55%
5.0
4.5
5.0
4.1
4.6
4.3
4.3
Federated views Message-oriented movement Data replication and synchronization Total Rating Weighted Rating
15%
3.9
3.0
3.9
3.7
4.2
3.4
2.2
10%
4.5
3.7
3.3
4.1
3.2
2.0
3.7
20%
4.6 18 4.705
5.0 16.2 4.295
4.5 16.7 4.565
2.9 14.8 3.8
4.0 16 4.28
3.0 12.7 3.675
2.9 13.1 3.645
Source: Gartner Critical Capabilities for Data Integration Tools: Common Data Delivery Styles, Dec 2010 (showing top 7 of 11 vendors in chart above)
Pre-Built SolutionsSOA SuiteBAM & CEPODI Embedded Pre-built Content
ID MgmtBPEL & OSBODI Integrated
Business IntelligenceBI AppsODI Embedded Pre-built Content
Other Apps (+more)Agile PLMODI Embedded Pre-built Content
Identity AnalyticsODI Embedded Pre-built Content
BIEE SuiteODI Integrated
RetekODI Embedded Pre-built Content
Enterprise Performance Management and FinancialsHFMODI Embedded
PlanningODI Embedded Pre-built Content
Essbase
MDM/DRM
Spend AnalyzerODI Embedded Pre-built Content
Governance & RiskODI Embedded Pre-built Content
Siebel CRMODI Integrated GoldenGate Integrated New
ODI Integrated
ODI Integrated
Oracle Fusion ERP ApplicationsFusion Pillar ReplicationOGG Embedded Pre-built Content
Fusion AIANew ODI Embedded Pre-built Content
Fusion HCMODI Embedded Pre-built Content
Fusion MDMODI Embedded Pre-built Content
Fusion CRMODI Embedded Pre-built Content
Fusion FinancialsODI Embedded Pre-built Content
Fusion MigrationODI Embedded Pre-built Content
New
Oracle Golden GateMilomir Vojvodic
What is GG and how it works?Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs.
Trail files: Stages and queues data for routing. Pump: Distribute data for routing to multiple targets. Route: Data is compressed, encrypted for routing to targets. Delivery: Applies data with transaction integrity, transforming the data as required.
Pump Capture Source Trail Pump LAN / WAN / Internet (TCP/IP) Target Trail Delivery
Source Database(s)7
Target Trail
Source Trail
Delivery
Capture
Target Database(s)
Bi-directional
Expanded HeterogeneityDatabasesOracle GoldenGate Capture: Oracle NEW DB2 for v 9.7 NEW Microsoft SQL Server Sybase ASE Teradata Enscribe SQL/MP SQL/MX NEW MySQL JMS message queues NEW
O/S and PlatformsLnux for 2008 Sun Solaris Windows 2000, 2003, XP HP NonStop HP-UX HP TRU64 HP OpenVMS IBM AIX IBM z Series zLinux
Oracle GoldenGate Delivery: All listed above, plus: NEW TimesTen, IBM i Series Netezza, Greenplum, and any ODBC compatible databases ETL products
8
If we can identify top 3 differentiators
Performance
Sub-second latency for data movement Low-impact capture, routing, transformation, and delivery of transactional data
Flexible & Extendable
Open, modular architecture Supports heterogeneous sources and targets Agile and adaptable to modern data integration architectures Maintains transactional integrity Resilient against interruptions and failures Fast Recovery after process interruptions.
Reliable
Oracle GoldenGateZero Downtime Upgrade & Migration New Application/ OS/ Database/Hardware Message Bus Query Offloading, Disaster Recovery Replica/ Standby
Oracle GoldenGateDatabase Distributed Systems
Data Synchronization across the Enterprise
Real-time BI, Operational Reporting ODS/Data Warehouse Legacy Message Bus Event Driven Architecture, SOA
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deployment
db A1
2 Zero DT Migrationapp A app A
solution X
db X
db A
db A
db A
5 Active Active DB Deployment 6 Regional DBs 7 Data level solutions/apps integration
db A
BI
DW regional part ofdb A
Popular use cases of GG
1 Data Failure Protection
app A
db A
db A
Data Failure Protection
Business ImpactsAvoid revenue & employee productivity loss Prevent decrease in customer loyalty and brand equity Decrease the manpower needed for system maintenance and recovery Limit credits and penalties due to not meeting SLAs for availability
Where it is happeningHA/MAA Infrastructure for Cloud Databases Minimizes risks during data center consolidation or migration to better platforms government regulations In Telecommunications OSS/BSS systems without interrupting operations In Financial Services/Banking Ensure continuous operations for critical applications such as payment processing, fraud detection, ATM, core-banking, trading systems Ecommerce/Retail Continuous operations for customer portal and POS systems
Active Data Guard and Golden Gate
Physical replication with Active Data Guard Best for Disaster Recovery for Oracle Applications Best for Disaster Recovery for Oracle Database 11g
Logical, heterogeneous replication with GoldenGate Only solution for real-time replication from Teradata Best for regular maintenance in heterogeneous environment
Active-Active bi-directional or multi-master replication with GoldenGate Best for custom and specialized applications
Popular use cases of GG
App A1
db A1
1 Data Failure Protection 2 Zero DT Migrationapp A
db A
Zero DT Migration
Business ImpactsAvoid revenue & employee productivity loss Prevent decrease in customer loyalty and brand equity Decrease the manpower needed for system maintenance and recovery Limit credits and penalties due to not meeting SLAs for availability SAP maintaince costs Oracle support costsModernize hardware and move to the most cost effective one
Where it is happeningHA/MAA Infrastructure for Cloud Databases Minimizes risks during data center consolidation or migration to better platforms government regulations In Telecommunications OSS/BSS systems without interrupting operations In Financial Services/Banking Ensure continuous operations for critical applications such as payment processing, fraud detection, ATM, core-banking, trading systems Ecommerce/Retail Continuous operations for customer portal and POS systems Move to the clouds Customers that want to consolidate databases and data centers to reduce costs Oracle DB customers that need ZDT Migration from 8i or 9i SAP customers Siebel CRM and JDE customers
Oracle GoldenGate 11g for Oracle Applications
Oracle GoldenGate for Application Upgrades & Migrations Siebel CRM (V6 V8 upgrading to V8.x) All platforms (Oracle, DB2(LUW & z/OS) & MSSQL) Also available to support Siebel Remote and incremental application repository zero downtime releases. JDE (all versions) Low downtime (not zero) upgrade solution All platforms except iSeries are supported. (iSeries is on the roadmap) SAP (Oracle to Oracle Online Migrations aka Triple-O)
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI
db A1
2 Zero DT Migrationapp A
db A
db A
BI
DW
Real Time DW/BI
Business ImpactsAgility to prevent costs/looses and achive revenue Performance of the production system (productivity) Decreasing Manpower costs for batch loading of DW
Where it is happeningCustomers want faster Apps (less invasive reporting), esspecially Oracle Apps BI customers that are looking to Use realtime or near real-time data from heterogeneous sources to heterogeneous targets in a mixed platform environment. BI customers that require transaction integrity for accurate and granular analysis ETL users that want to reduce overhead on source systems via log-based change data capture Also other BI customers that need stability of the BI environment BI customers where it t it takes too much time to load/refresh the Data Warehouse
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deploymentdb A
db A1
2 Zero DT Migrationapp A
db A
db A
BI
DW
Oracle GoldenGate 11g for Oracle Applications
Oracle GoldenGate is Certified for Operational Reporting for: Oracle E-Business Suite v12.x and higher Oracle JD Edwards v9.x and higher Oracle PeopleSoft Enterprise v8.51 and higher Oracle Siebel CRM v8.x and higher
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deploymentdb A db A
db A1
2 Zero DT Migrationapp A app A
db A
5 Active Active DB Deployment
db A
BI
DW
Active Active DB Deployments
Business Impacts Increased availabilty and performance but at the same time protect from data failure Better use of existing server assets
Where it is happening Telco
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deploymentdb A db A
db A1
2 Zero DT Migrationapp A app A
db A
5 Active Active DB Deployment 6 Regional DBs
db A
BI
DW regional part ofdb A
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deployment
db A1
2 Zero DT Migrationapp A app A
solution X
db X
db A
db A
db A
5 Active Active DB Deployment 6 Regional DBs 7 Data level solutions/apps integration
db A
BI
DW regional part ofdb A
Oracle Data IntegratorMilomir Vojvodic
What is ODI and how it works?ODI Data ServicesData Access Service Transformation Service Batch Data Service
ODI Declarative Design ODIDeclarative Design
ODI Agent Data Flow Conductor
1Define
2Automatically Generate Dataflow
WhatYou Want
Define How: Built-in Templates
ODI Knowledge ModulesReverse Engineer Metadata Reverse Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformati on Services W W W S S S
ODI E-LT
Staging Tables Load CDC Sources Journal ize Check Integrate Services
Target Tables Error Tables
Transform ExtractOracle Web Services DB2 Web Services
Transform Load
Sample out-of-the-box Knowledge Modules SQL Server Triggers Oracle DBLink JMS Queues Check MS Excel TPump/ Multiload Oracle Merge
SAP/R3
Log Miner
Siebel
DB2 Journals
DB2 Exp/Imp
Oracle SQL*Load er
Check Sybase
Type II SCD
Siebel EIM Schema
Benefits
27
27
If we can identify top 3 differentiators
Faster
Fastest E-LT bulk/batch performance Real-Time integration (sub-sec trickle) with CDC, replication, and SOA infrastructure Faster project setup, design and delivery Simpler to setup, configure, manage, & monitor Map using declarative SQL interfaces Deploy with fewer hardware devices Extend with knowledge module & code templates Less HW & Energy Costs with E-LT Architecture Less Time Wasted on Unnecessary ETL Mappings, Scripting, and Complex Training Less Integration Overhead Integrating with Applications, SOA, and Management Software
Simpler
Saves Money(Lower TCO, Higher ROI)
Capital Cost ReductionReduce or Eliminate Unneeded Hardware and Human Capital
ODI-based Oracle BI Apps Number of Setup Steps Number of Servers Number of connections 7 1 3
INFA-based Oracle BI Apps Number of Setup Steps Number of Servers 10 3
Number of connections
7
Popular use cases of GG
App A1
1 Data Failure Protection 3 Real Time DW/BI 4 Offload DB Deployment
db A1
2 Zero DT Migrationapp A app A
solution X
db X
db A
db A
db A
5 Active Active DB Deployment 6 Regional DBs 7 Data level solutions/apps integration
db A
BI
DW regional part ofdb A
Which of them require data transformation and bulk data movement?App A1
db A1
2 Zero DT Migrationapp A
3 Real Time DW/BI
solution X
db X
db A
BI
7 Data level solutions/apps integration
DW
Popular use cases of ODI
X, Y, A, B
solution X
solution Y
app B
app A
db XYAB
db XFiles Excel XML
db Y
db B
db A
1 Loading DW
2 Data level solutions/apps integration 3 Data migrations&conversions in modernization/consolidation projects
BI
DW
Not only DB
E-LTany
and E-T-L
OLTP to EDW/Mart
Any to OLTP DB
any
any
App/API to EDW/Mart
Any to File(s)
Any to App/API
any
File/WS/JMS to EDW/Mart
Any to OLAP
Popular use cases of ODI
solution X
solution Y
app B
app A
db X
db Y
db B
db A
1 Loading DW
Files Excel XML
BI
DW
Loading DW
Business Impacts Comparing to Manual Coding : Productivity (number of langauges to use, prebuilt code (no need to write SQL)) Maintaince (hard to change smt, every script contains special rules in several machines)and Peformance and availabilty and Costs (E-LT) Comparing to other ETL : Costs (ELT) and Productivity (still manual coding and SQL tuning)
Where it is happeningCustomers with recent M&A activities (looking to reduce capital costs) Customer has dedicated cluster of ETL servers running at/near capacity and is looking to add capacity Large teams of ETL development staff are fully engaged with maintenance activity and not able to take on new projects Siebel, PeopleSoft, JDEdwards applications customers since they are more likely to have heterogeneous environments and to value heterogeneous database support. Oracle BI Applications, Oracle EPM, Oracle BI EE Plus
Popular use cases of ODI
solution X
solution Y
app B
app A
db XFiles Excel XML
db Y
db B
db A
1 Loading DW
2 Data level solutions/apps integrationBI
DW
Data Level Solutions/Apps Integration
Business ImpactsOne architecture for all integration tasks No more performance issues No more maintenance complexity (easy change a transformation rule) get a project done in a couple of weeks than a couple of months
Where it is happeningCustomers with heterogeneous sources or targets in a mixed platform environment and with advanced knowledge of database technologies and understanding of database power For the data like Customer, Product, Supplier, Financials to create customData Hubs and Master Data Mgmt Mergers and acquisitions SOA customers Customers want to standradize on Java
Data Hubs in Key IndustriesFinancial Services Single view of high quality customer data drives accurate customer insight and improved marketing effectiveness Supports compliance and reporting KYC requirements
Retail
Harmonizes customer data from multiple channels to improve sales and marketing effectiveness Enhances opportunity and customer lifetime value identification Improves customer insight for revenue optimization and targeted customer retention Effective compliance and risk mitigation for next generation services Expands understanding of network assets and customer delivery points Improves management of regulatory compliance and reporting requirements Delivers a comprehensive view of patient for care and billing Manages patient, epidemiology, diagnosis and treatment data quality across systems and organizations Single view of citizen for better internal information sharing, service delivery, licensing, provision of child care, and fraud detection Reduce costs through system rationalisation
Telco
Energy & Utilities Utilities
Healthcare
Government
38
Understanding Performance ChoicesLess than 10MBDepends on whether an intermediary XML format is useful for other processing (use ESB), or if joining File data to tabular RDB data is required (use ODI)
XML ESB ESB ESB XML ESB depends
File ESB ESB depends File depends ODI
DB ESB depends ODI DB ODI ODI
XML File DBBetween 10-50MB
Depends on ho much cross-referencing among the data values and rows is required during transformation the more there is, the faster ODI will perform relative to ESB
XML File
DBGreater than 50MB
ODIXML depends ODI ODI
ODIFile ODI ODI ODI
ODIDB ODI ODI ODI
If the source and target are both XML, and there is no cross-referencing of data among rows, then a streaming-type or parallel-engine-type approach might scale
XML File DB
*caveat always benchmark if you are unsure and require best possible results
Popular use cases of ODI
X, Y, A, B
solution X
solution Y
app B
app A
db XYAB
db XFiles Excel XML
db Y
db B
db A
1 Loading DW
2 Data level solutions/apps integration 3 Data migrations&conversions in modernization/consolidation projects
BI
DW
Data migrations&conversions in modernization/consolidation projectsBusiness ImpactsGet double-duty from database investment by using it for transformations Reduced migration time and expense
Where it is happeningVirtualization Cloud Exadata (ODI for mixed workload use cases and handle both loading and querying workloads simultaneously.) Application Migrations require heavy data transformation mainframe application to open systems (major schema changes) new generation of DW and BI solutions
ODI Outperforms Informatica for Less
ODIE 7.5 TB /hr $0 ETL H/W L T
7tb/hr
Run ODI Directly on Exadata Complex Data Transformations Linear ETL Scalability Fully Leverages DBFS/Infiniband, Smart Storage, and Advanced Compression
4:1 advantage*
$5m 3yr savings**
1 Exadata X2-8
Informatica1.8 TB /hr
$5m ETL H/W
E
T
1.8 tb/hr
L
* TPC-H data sets with transformations ** Production hardware savings(not including Dev + Test environments, management costs or software savings)
HP Superdome 64 + XP24000 w/ Flash
Copyright 2011, Oracle Corporation and/or its affiliates
DIS Packaging, References and CompetitiveMilomir Vojvodic
Data Integration & Availability Pillar Products
$17,500(Src & Trgt DB CPUs)
Oracle GoldenGateFor Oracle (includes Active Data Guard) For Non-Oracle Database (Includes Teradata)
Use Case: HA / Real-time DI Options: Management Pack ($3,500 per CPU) Application Adapters ($20,000 per CPU) Veridata ($30,000 per mid-tier CPU) Use Cases: HA / Real-time DI Change Capture for Data Warehouses Use Case: Bulk Data Transformation Options: Application Adapters ($2300 per CPU) Use Cases: ODI-EE with Clustering & ODI-EE for SOA Use Cases
$100,000(Src & Trgt DB CPUs)
Oracle GoldenGate for MainframeIncludes HP Non-stop and DB2 Mainframe
$23,000(Target DB CPUs)
ODI Enterprise EditionOracle Data Integrator Oracle Warehouse Builder Enterprise ETL
$70,000(Target DB CPUs)
ODI SuiteOracle Data Integrator Enterprise Edition Oracle Data Integrator Management Pack WebLogic (App Server) Suite (restricted) Oracle Coherence (restricted) Oracle Service Bus (restricted) Oracle BPEL Process Manager (restricted)
$10,000(per CPU)
Active Data GuardRequires Oracle Database Enterprise Edition
Use Cases: Disaster recovery & data protection for Oracle DB
Oracle Data Integration SolutionsHigh Availability & ReplicationPlatform Specific (logical HA) Block-Level Solutions (physical HA)
Sybase Replication Server
EMC Symantec / Veritas Quest Shareplex Hitachi Data Systems Microsoft SQL Replication IBM PPRC Not suitable for HP StorageWorks heterogeneous, enterprise deployments no transaction management, no transformation, homogenous Data IntegrationETL Solutions EAI Solutions
IBM Infosphere (Ascential) Informatica SAP / Business Objects Ab Initio Invasive to source, Not real time, Requires Mid-Tier, Expensive & Monolithic
Tibco Progress Software IBM Websphere MQ Lower throughput, more overhead, not direct to database
45
Oracle ETL strategyMilomir Vojvodic
Evolving Data Integration Roadmap
Roadmap 2000 to 2007 2006
Roadmap 2007 to 2008 2007
OWB-EETL
ODI OWB-EETLOWB (free DB features)
OWB
OWB (free DB features)
Roadmap 2009 to 2010
Roadmap from Jan 2010 to Current
KMs
New ETL
ODIKMs
OWB (free DB features)
OWB (free DB features)
OWB Support PoliciesCustomer Downloadable OWB Support Policies: http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf Release GA Date Premier Support Ends Extended Support Ends Sustaining Support Ends
OWB 9.2
Jul 2003
Oct 2007
Not Available
Oct 2010
OWB 10.1OWB 10.2 OWB 11.1
Apr 2004Jul 2005 Aug 2007
Jul 2007Jul 2010 Aug 2012
Not AvailableJul 2013 Aug 2015
Jul 2010Indefinite Indefinite
OWB 11.2
Sep 2009
Current Dates: Jan 2015Future Dates: Same as DB12
Current Dates: Jan 2018Future Dates: Same as DB12
Indefinite
For more Information about Support Levels: http://www.oracle.com/us/support/lifetime-support/index.html
OWB License MigrationsExisting Product License OWB ETL features (included in Database licenses) OWB Enterprise ETL option by CPU Migrating to ODI EE license No migration credit applies. Ratio 2:1 Notes Pay full ODI EE price. Using list-to-list migration.
OWB Enterprise ETL option by NUP
Ratio 100:1
Using list-to-list. If customers have less than 100 NUP, use a percentage credit. i.e. 25 NUP would = 25% credit toward purchase of 1 processor. Customer pays the remaining 75% of the list price, minus standard and approved discounts.All 9i features became free with OWB 10.1. Note that OWB SAP Connector migrations are covered in the eSource Migration Listing
OWB included in internet Developer Suite (9i) or OWB 2.x versions
No migration credit applies.
OWB DQ and DP
No migration credit applies.
In some cases customers may buy more Licenses (if they already own some of the same type) For additional License Migration queries, contact [email protected] Additional discounts or price promotions may apply, contact [email protected]
Oracle ETL Product BundlesFeaturesBasic features for Oracle data loading
DB (free ETL)Y Oracle Gateways Y Oracle OLAP only Oracle DBMS only -
ODI-EEY Y Y Y Y Y Y Y Y Y Y
ODI SuiteY Y Y Y Y Y Y Y Y Y Y
Oracle DB Targets Non-Oracle Sources Slowly Changing Dimensions (type 1) OLAP Targets (Oracle + Essbase) Built-in Scheduler
Standard features for enterprise ETL use cases
Non-Oracle DB Targets Changed Data Capture (including OGG integration) Web Services (API & Src/Trgt) & XML Targets Data Lineage / Impact Analysis Declarative Design Environment Slowly Changing Dimensions (2 & 3)
Load Plans, Parallel Execution and RestartabilityJEE Deployment , Clustering and High Availability Java based SDK for full Programmatic Control Unified Management and Administration with OEM Pre-built Integration with OBI-EE, Hyperion, SOA/AIA, etc.Advanced features for enterprise information management
-
YY Y Y Y -
YY Y Y Y Y Y Y Y
Human Workflow (BPEL PM) Process Orchestration & Service Bus Data Relationship Management / MDM WebLogic Suite (Coherence Data Grid, B2B, Biz Rules)
OWB Enterprise ETL
Accessing sources and targetsNative support for heterogeneous databases, including the definition of new platform types, JDBC connectivity, knowledge modules, and code templates Change Data Capture mappings in ETL Publishing and consuming web services, including integrating with process orchestration technologies such as Oracle BPEL Process Manager Importing Cobol copybooks using the built-in wizard Targeting XML files
Designing and executing ETL jobsRight-time data warehousing utilizing queues and real-time mappings Reusing and embedding mapping code with pluggable mappings Specifying the load order of multiple targets in a single mapping Partitioning and parallelizing updates to tables by utilizing the data chunking feature Scheduling ETL jobs using the built-in Schedule interface