real-time data warehousing using oracle database and obiee - collaborate'11

T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011

Real-Time DW with OBIEE and Oracle Database

1Thursday, 21 April 2011


Who am I?


http://www.rittmanmead.com



• Stewart Bryson, Managing Director, Rittman Mead America

• Oracle BI/DW Architect and Delivery Specialist

• Oracle Database and BI Stack since 1996

• Regular Community Speaker

• Writer for Rittman Mead Blog:http://www.rittmanmead.com/blog

• Developer of Transcend Framework

• Email : [email protected]

• Twitter : @stewartbryson




http://www.rittmanmead.com/blog

http://www.rittmanmead.com/blog


Who is Rittman Mead?





• Oracle BI&DW Experts

• Consulting, Training and Support

• Global Company, Offices in the US, UK and India

• Clients in the UK, USA, Europe, Middle-East

• Voted UKOUG BI Partner of the Year 2008

• 16 Sessions at Oracle Open World 2010

• Twitter : @rittmanmead





Rittman Mead BI Forum, May 2011





Registrations open for the Rittman Mead BI Forum, May 2011

• Brighton, UK, May 18-20th 2011• Atlanta, GA, May 25-27th 2011• Covers OBIEE, DW, EPM• Guest speakers including:‣ Jeff McQuigg‣ Kevin McGinley‣ Jean-Pierre Dijcks‣ Mike Durran‣ Shyam Varan Nath‣ Philippe Lions‣ Emiel van Bockel

• OBIEE 11g Masterclass withMark Rittman and Tony Heljula

• Full details at http://www.rittmanmead.com/biforum2011




http://www.rittmanmead.com/biforum2011

http://www.rittmanmead.com/biforum2011


What will this session answer?





• What do I mean by “real-time data warehousing?”‣ Zero latency, or just reduced latency?‣ Do I really mean “right-time data warehousing?”

• What can I use to reduce latency?‣ What does OBIEE offer?‣ What does the Oracle Database (and OWB)

offer?• Can I blend these different offerings?‣ Of course you can!‣ It’s the name of the presentation





What do I mean by “real-time”?





• Traditional data warehouses‣ Daily to weekly batch-load cycles‣ Key-metrics are always delayed‣ Information may not be “actionable”

• Real-time data warehouses‣ Up to the minute or hour reporting‣ Key-metrics are moving‣ Information is actionable





An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact”





An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact”

!"##$%!&'"()!"#"$%&'(##%)*+)%'(,-*./")%0*1"%#(%2"%)3$"%(4%#0"%.""5%2"4($"%#0"+%/.)/)#%(.%$"*67#/,"%23)/."))/.#"66/8".'"9

!"#$%&#'()*#&)%+,-./)#$(/01-.23#4(&#%-#+)%*-#5#6(0)#-6)#)23.2)#/%2%3)/)2-7#-&%$-.(2#$(2-&(+#%2'#89:#*"*-)/*#%&)#&)%+#-./);<=

!"#/(>.+)#06(2)#.*#&)%+#-./)7#)?)2#*(/)#(@#/"#6(/)#)2-)&-%.2/)2-#3%'3)-*#%&)#&)%+#-./)#A#>1-#B6%-#%>(1-#>1*.2)**

.2-)++.3)2$)#*"*-)/*C

52#-6)#)%&+"#'%"*#(@#95#4(&#B%*#.-#$%++)'#')$.*.(2#*100(&-#>%$D#-6)2C<#&)0(&-.23#'%-%#@(&#-6)#$1&&)2-#/(2-6#B%*#%2#%$6.)?)/)2-=

8*#-)$62.E1)*#%2'#-)$62(+(3.)*#)?(+?)'7#$(/0%2.)*#/(?)'#-(#&)0(&-#(2#-6)#0&)?.(1*#'%"#>1-#B)&)#*-.++#&)+.%2-#(2#%2#(1-,(@,

6(1&*#>%-$6#0&($)**#-(#/(?)#'%-%#@&(/#-6)#*(1&$)#*"*-)/*#%2'#*-(&)#%2'#%33&)3%-)#.-#.2#-6).&#'%-%#B%&)6(1*)*#%2'#'%-%

/%&-*=

91-#B.-6#FG,6(1&#>1*.2)**#'%"*#%2'#-6)#2))'#-(#&)0(&-#%$&(**#/1+-.0+)#-./),H(2)*7#-6)#-&%'.-.(2%+#>%-$6#B.2'(B#.*#>).23

*E1))H)'#.2-(#2)%&#2(2,)I.*-)2$)=#J)B#'%-%#)I-&%$-.(2#%2'#+(%'#0%&%'.3/*#6%?)#%+*(#>))2#')?)+(0)'#-(#-&.$D+),@))'#&)0(&-.23

*"*-)/*=#91-#%&)#-6)*)#&)%++"#&)%+#-./)#A#%2'#'()*#.-#&)%++"#/%--)&C

52#&)%+.-"7#-6)&)#.*#%+B%"*#3(.23#-(#>)#%#')3&))#(@#+%-)2$"#>)-B))2#%2#)?)2-#6%00)2.23#%2'#.-#>).23#&)$(&')'#(2#%#-&%2*%$-.(2%+

*"*-)/7#0&(0%3%-)'#-(#-6)#&)0(&-.23#*"*-)/7#+(%')'7#%2'#@.2%++"#%33&)3%-)'#(2#%#'%-%#B%&)6(1*)#>)@(&)#>).23#&)%'"#-(#E1)&"=

8''#-(#-6%-#-6)#')+%"#>)-B))2#-6)#1*)&#.**1.23#%#E1)&"#-6%-#%$$)**)*#-6.*#2)B#2133)-#(@#.2@(&/%-.(2#%2'#-6)#&)*1+-#>).23

&)-1&2)'#-(#-6)#1*)&7#%2'#"(1#@.2'#%#'.*$)&2.>+)#+%3#>)-B))2#)?)2-#%2'#(>*)&?%-.(2=

K&1)7#-6)&)#%&)#-6.23*#"(1#$%2#'(#-(#/.-.3%-)#-6.*#+%3#>1-#-6)&)#B.++#%+B%"*#>)#%#$(10+)#(@#*-)0*#.2#-6)#0&($)**#B6)&)#&)'1$.23

(2)#-./)#0)&.('#.2$&)%*)*#%2(-6)&#A#@(&#)I%/0+)7#"(1#$%2#*6(&-)2#&)0(&-#)I)$1-.(2#-./)#>"#%33&)3%-.23#-6)#'%-%#-(#>)--)&

%2*B)&#-6)#E1)&"7#>1-#-6.*#*-)0#%''*#/(&)#-./)#-(#-6)#'%-%#+(%'#%2'#%33&)3%-)#06%*)=

L1&-6)&/(&)7#'.@@)&)2-#-"0)*#(@#.2$(/.23#'%-%#$%2#6%?)#'.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%$-.(2#/%"#6%?)#%#*6(&-)&

+%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))'#-(#>)#0&(0%3%-)'#-6&(136#/%2"#+%")&*#(@#6.*-(&.$%+

%33&)3%-.(2#-%>+)*#4*))#L.31&)#M<=

K6)#'.?.*.(2#>)-B))2#'%-%#%$E1.*.-.(2#%2'#'%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0%2"#2))'*#-(#%2*B)&#B6)2

$(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)'#.2#0&(?.'.237#%*#&%0.'+"#%*#0(**.>+)7#.2@(&/%-.(2#>%*)'

(2#%+&)%'"#%$E1.&)'#'%-%7#(&#'(#"(1#%+*(#2))'#-(#&)0(&-#(2#2)B+"#%'')'#@%$-1%+#.2@(&/%-.(2C

L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+)#06(2)#$(/0%2"P*#$%++

$)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2#%>(1-#Q6(B#3(('#%

$1*-(/)&#.*P#%2'#-6)#+.D)+.6(('#-6%-#*(/)(2)#B.-6.2#-6%-

$1*-(/)&P*#')/(3&%06.$#B(1+'#*B.-$6#-(#%2(-6)&#

*100+.)&=

K6.*#.2@(&/%-.(2#2))'*#-(#>)#0&)*)2-)'#E1.$D+"#4B6.+*-#-6)"

%&)#*0)%D.23#-(#-6)/<#%2'#0&)@)&%>+"#B.-6.2#-6)#$(2-)I-#(@#-6)

+.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/%2%3)*#-6)#$1*-(/)&

.2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6%-#%2"#(@#-6)#/)-&.$*#1*)'

>"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/)&#0&(@.-%>.+.-"#%2'

0&)'.$-)'#$61&2#B(1+'#>)#.2@+1)2$)'#>"#)?)2-*#(@#-6)#+%*-#@)B

6(1&*=

S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23#)/>)'')'#95#@12$-.(2%+.-"

-(#%+)&-#-(#0(-)2-.%+#@&%1'#4*-($D#/%&D)-*#%2'#$&)'.-#$%&'

$(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1*)&*<#/%"#B)++#2))'#-(

D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.?.-"=

***+,-./0.12345,416,+537

FIGURE 1: BI latency

Recording Propagation Aggregation Query

Event Report

Real-time query latency





Business Case





• We want to be able to segment our reporting between “real-time” needs and static reports

‣ Most users will want real-time numbers delivered with dashboards

‣ Some users (financial-minded) don’t want numbers to change each time they run a report

• Use the standard Oracle BI Stack

‣ Oracle Database Enterprise Edition

‣ Oracle Business Intelligence Enterprise Edition





Latency versus Query Performance






OLTP Reporting

Federated OLTP/EDW Reporting

EDW with Realtime Component

Standard EDW






OLTP Reporting


EDW with Realtime Component

Standard EDW

Latency

Query Performance





Oracle Next-Generation Reference Architecture





Requirement: Identifying and Capturing Change





Requirement: Change Data Capture






Identify Change

• Oracle CDC• OWB Trickle-feed

mappings• Trigger-based

capture• Recording

modification dates• MINUS Command






Identify Change





Propagate Change


mappings• Message Queues• Views over

database links• File movements






Identify Change





Propagate Change




Apply Change to Target

• OWB Mappings and Process Flows

• Other ETL Tools• Custom ETL code• Custom transaction

code





Oracle Asynchronous Change Data Capture






Hot LogOnline redo logs mined locallyChange tables are updated and stored locally

Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely

Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely

Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely






Hot LogOnline redo logs mined locallyChange tables are updated and stored locally

Latency

Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely

Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely

Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely






Identify Change





Propagate Change





• OWB Mappings and Process Flows

• Other ETL Tools• Custom ETL code• Custom transaction

code






Identify Change

Propagate Change







Identify Change

Propagate Change


•Trickle-feed mappings allow the development of Streams Capture and Apply processes as continuously running mappings.





• Layering in OLTP data to provide the real-time component

• General approach is to use a BI tool such as OBIEE

• No real database or modeling involvement





Classic Batch Load Star Schema: Process Flow





Classic Batch Load Star Schema: Fact Table Load





Federation with OBIEE





OBIEE Metadata and Data Layers






Relational, OLAP, File and Application Data Sources

• Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters







• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...


Enterprise Semantic Model






• The Web Catalog containsdefinitions of reports, alertsdashboards, KPIs, based on...


• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...


Web Catalog

Enterprise Semantic Model





Flow of Data Through the Three-Layer Semantic Model





Flow of Data Through the Three-Layer Semantic Model

Simplification of the Data ModelIntegration of Disparate DataSources

Addition of Business Logic and Calculations





Logical Table Sources Map to Physical Models





Logical Table Sources Respect the Physical Model





WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE ) union allselect T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3from SAWITH0 D3group by D3.c2, D3.c3)select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4from SAWITH1 D2





WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5from SAWITH0 D1





WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5from SAWITH0 D1

• The BI Server understands from the Fragmentation content how to “optimize” the query

• Only one LTS is needed to provide the answer





Fragmentation with OLTP and EDW

Demonstration





• Strengths

‣ Low latency

‣ No complex modeling or ETL

‣ Incorporated into any standard dimensional EDW

• Weaknesses

‣ Performance

‣ Difficult to conform multiple systems





EDW with a Real-Time Component





• Modeling complimentary real-time structures in an EDW

• Use functionality of the Oracle Database and OWB

• Complex ETL and Database interaction





The Real-Time Partition

Table Partition

Table Partition

Table Partition

Fact Table






Table Partition

Table Partition

Table Partition

Fact Table

Physical Table

Fact Real-Time Partition






Physical Table

• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch

• No indexes, small data set• Partition Exchange at the end of the day






Table Partition

Table Partition

Table Partition

Physical Table








Table Partition

Table Partition

Table Partition

Physical TableTable Partition







Early-Arriving Facts





• Grab the most recent dimension record when that entity already exists

• Use a default “Not Known Yet” record when they don’t

• Prior to the partition exchange, we run our late-arriving dimension loads

• We update the Real-Time Partition with the correct surrogate keys‣ OWB generates an efficient MERGE‣ Small data sets in the Real-Time Partition

What if Type 2 dimension changes haven’t arrived yet?





Real-Time Integration: Fact Table RT Load





Real-Time Integration: Correct Dimensions





Real-Time Integration: “Closing the Books” on the Day





Plugging it together with OBIEE (Non-Fragmentation)

• Using standard database functionality (UNION ALL)

• Could be done with a view• Using Surrogate Keys in the

RT Partition• OBIEE doesn’t have to

manage different LTS’s





EDW with a Real-Time Component

Demonstration





Summary





• Latency versus performance and scalability

• Latency and performance versus degree of difficulty

• Different variations on the two test cases





Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011

Real-Time DW with OBIEE and Oracle Database


real-time data warehousing using oracle database and obiee - collaborate'11

Technology

bi latency

realtime query latency

rittman mead bi forum

rittman mead blog

realtime numbers

realtime data warehousing

oracle databasestewart

oracle open world