performance tuning oracle’s bi applications - · pdf filetry to tune the bi apps model...

44
Contact Us 510.818.9480 | www.kpipartners.com © KPI Partners Inc. Start Here Jeff McQuigg | Senior BI Architect April 24, 2013 Performance Tuning Oracle’s BI Applications

Upload: lytuong

Post on 22-Feb-2018

227 views

Category:

Documents


1 download

TRANSCRIPT

Contact Us510.818.9480 | www.kpipartners.com

© KPI Partners Inc.

Start Here

Jeff McQuigg | Senior BI Architect

April 24, 2013

Performance Tuning Oracle’s BI Applications

Agenda

2

①Introducing The Performance Layer

②Building The Performance Layer

③Mapping Into Oracle BI

④Implementation Considerations

⑤Q&A

Agenda

Performance Tuning Oracle’s BI Applications 3

Introducing The

Performance Layer

4

Targeted At Organizations

Who Have:

� Large Data Volumes

� Highly Customized Tables

� Questionable Design Extensions

� Aggressive Performance Targets

� Slow Hardware

� Only a Data Warehouse

5

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

These performance concepts

are applicable to any BI system

� BI Apps (OBIA) Stars

� Custom Stars in OBIA

� Custom Built Warehouses

� SQL Server

6

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

Wide tables carry more data than you need

Dashboards only need a few fields

Smaller is faster

7

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

Eliminate other and conflicting priorities

Singular focus on performance

Performance starts with clear design goals

8

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

Great performance requires perfect design for how it is used

A Dashboarding environment is an “Application”

Use a top-down design approach to support that application

Specialized design for specialized usage

9

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

Pre-built logic

Clean star models

Reduced data weight

Tables which match usage by Oracle BI

10

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

Top-Down design yieldsF

11Performance Tuning Oracle’s BI Applications

Introducing The Performance Layer >> Oracle’s View On Data Warehouse Architecture

Keep the BI Apps/DW

model mostly as-is & add

a Performance LayerF

① Build a new data model

② Copy data from BI Apps/DW tables

③ Bring only what you need

④ Denormalize & pre-calculate

*Optimize To Usage*

12

Pe

rfo

rma

nce

Min

i-F

rid

ge

BI A

pp

s D

ata

Fri

dg

e

ETL

Introducing The Performance Layer

Performance Tuning Oracle’s BI Applications

13Performance Tuning Oracle’s BI Applications

① The Performance Layer is an

industry standard architecture

② Design is driven only by report

performance improvement

③ Travel light

④ No need to alter BI Apps or DW

Introducing The Performance Layer - Takeaways

Building The

Performance Layer

14

1. Identify a priority area (select a Fact table)

2. Identify common use cases (reports w/ prompts & data

security)

3. Analyze resulting physical SQL

4. Try to tune the BI Apps model first! (Indexes, etc)

15

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

5. Prototype a new data model to match those

needs (SQL handcrafting of new tables)

6. Adjust SQL & benchmark (SQL handcrafting needed)

7. Map into Oracle BI & test (Unit & Regression)

8. Benchmark the Oracle BI report using

prototyped tables (Reports have many SQLsn)

16

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

9. Build the tables using INFA & DAC - Complete

Oracle BI RPD mapping

10. Formal Regression Test

11. Deploy

12. Enjoy praise from users

17

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

Reduce I/O with extreme prejudice

• Tune the BI Apps model first! It may work for you with low effort

• Employ techniques to eliminate I/O wherever possible

• Partition Elimination, Compression, Indexes, Aggregates, Star Transformations

• Let the Performance Layer do the work, not the report query

• Follow the KISS principle: Use a simple and clean Star. No Snowflakes!

• Ensure OBI is mapped properly and uses correct tables with perfect SQL

• Favor a general approach as opposed to a case-by-case approach

• A rising tide lifts all boats

18

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

� There are 4 kinds of tables in the Performance Layer:1. Skinny Dimension and Fact tables

2. New Dimension tables

3. Mini-Dimension tables

4. Fact Aggregate tables

� Built directly from the base BI Apps or DW tables

� Goal: use these tables in as many reports as possible

� Guiding principles and performance influences:1. Application use cases drive the layer’s design

2. Use minimal data for the job at hand

3. Aggregate Fact data when needed

4. Denormalize dimensions to eliminate extra joins

5. Pre-Build calculations to eliminate extra joins

6. Pre-Split data sets based on logical usage

19

BI Apps or

DW Perf. Layer

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

20

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

① Single column, local bitmap

indexes on all Fact table FKs

(_WIDs) and filter fields (DELETE_FLG)

② Single column bitmap indexes

on all dimensional fields used in

any sort of prompt or report filter

③ Special composite B-Tree

indexes to assist Snowflaked

areas

④ Composite B-Tree indexes on

large dimensions for join backs

(for list reports)

① Single column, local bitmap

indexes on all Fact table FKs

(_WIDs) and filter fields (DELETE_FLG)

② Single column bitmap indexes

on all dimensional fields used in

any sort of prompt or report filter

③ Special composite B-Tree

indexes to assist Snowflaked

areas

④ Composite B-Tree indexes on

large dimensions for join backs

(for list reports)

Query Indexing (4 Types)

� For all Fact tables of a reasonable

size (e.g., > 5M rows)

� Usually partition on Month (Range

or Interval)

� The Database can easily

eliminate the majority of the table

� Allows for smaller, local indexes

� For all Fact tables of a reasonable

size (e.g., > 5M rows)

� Usually partition on Month (Range

or Interval)

� The Database can easily

eliminate the majority of the table

� Allows for smaller, local indexes

Table Partitioning

Before Beginning:

Tune the OOTB Model

� Skinny Tables are highly selective versions of the BI Apps or DW

tables• “Horizontal Aggregation” – E.g., 10 columns vs. 100 in the base table

� Both Dimensions and Facts

� Very easy to build and use

� Goal: Reduce Avg. Row Length to 1/5th - 1/20th original size

� Include only the columns you will need for top-down reporting analysis

• If you don’t need Customer Address, don’t include it• Ignore Meta Data columns (e.g., INTEGRATION_ID, etc.)

� Row sets are identical (1:1) with the base tables• For Dimensions use the same ROW_WIDs - can be used with existing fact tables easily

21

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

� Build using:1. Create Table as Select (CTAS)

2. Insert /*+ APPEND */

3. Materialized Views

� Compress the table

� Use Parallel hints & options

� Don’t forget partitions

� Enhance the tables with

calculation logic

� Database is very fast at these

operations – expect only a few

minutes for 100M rows

22

CREATE TABLE WC_ACCT_BUDGET_SF

COMPRESS NOLOGGING PARALLEL (DEGREE 8)

PARTITION BY RANGE(PERIOD_END_DT_WID) INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))

(PARTITION Part_01 VALUES LESS THAN (20100101))

AS SELECT /*+ PARALLEL(F,8) */

F.PERIOD_END_DT_WID,

F.X_PERIOD_END_DT_WID,

F.COMPANY_ORG_WID,

F.GL_ACCOUNT_WID,

F.X_POSTED_TOTAL_AMT,

case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES,

FROM W_ACCT_BUDGET_F F,

W_GL_ACCOUNT_D GL_D

WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

� Enhance _SF tables with logic

� Identify CASE WHEN statements which require other dimensions

• Potential great benefit if the join can be eliminated

• Don’t over do it – table will get less skinny with each column

� Identify any data set splitting from the RPD

• HR Workforce Events table has both Events and Snapshots records but they are always used separately in the RPD

• Usage drives design: Split them out!

• Huge benefit for Event counting metrics (~10% of table)

23

case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES,

FROM W_ACCT_BUDGET_F F,

W_GL_ACCOUNT_D GL_D

WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;

Create table WC_WRKFC_EVT_EVENTS_SF …

… WHERE SNAPSHOT_IND = 0

Create tableWC_WRKFC_EVT_MONTH_SNP_SF …

… WHERE SNAPSHOT_IND = 1

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

24

Real Examples I/O Benefit

Sub Ledger

(custom)

24X

Workforce Snap 11X

Workforce Events 56X

GL Balance 21X

Acct Budget 32X

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

Typical to get a 10X to 20X and even

50X I/O benefit in the _SF vs. the base

_F in size

� Reduced AVG_ROW_LEN

� COMPRESSION

� Record Set Splitting

� All without any aggregation

Skinny Dimensions also have benefits:

1. All I/O is a killer and slows down the entire system

2. De-normalize into a Star (eliminate snowflakes & outer joins)

3. Real World Ex. #1: 2 wide dims � 2 skinny dims: 6X query improvement

4. Real World Ex. #2: One query going from 11s to 4s with one skinny dim

5. Real World Ex. #3: GL Account Dimension: 37X I/O benefit

� Pre-build major pieces of commonly used but complex logic into the Data Model

• Over-relying on the RPD or Reports for logic can harm performance

• Let the ETL for the Performance Layer do the work not the query

� Example #1: Large binning and bucketingCASE WHEN FACT.ORDER_AMT BETWEEN 0 and 100 THEN ‘0-100’ ELSE CASE WHEN FACT.ORDER_AMT BETWEEN 101 and 200 THEN ‘101-200’ … END

• Build a new dimension table to hold these values –WC_CUST_ORDER_QTY_BAND_D

� Example #2: Date format conversions – dynamically building a new column with a string concatenation statement:

substring(T66755."PER_NAME_MONTH" , 1, 4) , '') + '-' + isnull(right(T66755."PER_NAME_MONTH" , 2) , '')

• Build a new column in the W_DAY_D table & index it

25

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

� Simply a higher level or levels of a larger dimension

• A combination of several Kimball concepts

• Granularities will be mixed

� Make a new table from the large, base dimension

• Contains distinct combinations

• Use only commonly used fields

• Get cues from dashboard prompts, column selectors, report filters

� Create a new ROW_WID

� Compress and index as normal, Parallel if needed for creation

� Easy to build and map

26

Create table WC_EMPLOYEE_MD COMPRESS asselect

ROWNUM AS ROW_WID, W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME

from (

select distinct W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME

from W_EMPLOYEE_D);

This real world example created 5,400

records from a W_EMPLOYEE_D of 9+

Million rows.

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

_MD tables are used in the Performance Layer in two places:

1. Link into Skinny Facts

• Use a separate FK in addition to the base _WID

• Fact table has both EMPLOYEE_WID and EMPLOYEE_MD_WID

� Thus the _SF can join to all of the following:

• New Mini Dimension (_MD) (~1% rows, some columns)

• New Skinny Dimension (_SD) (100% rows, some columns)

• Base BI Apps/DW Dimension (_D) (100% rows, 100% columns)

� The OBI RPD can select which one is best for each query

� Benefits of linking into the _SF1. The same set of fact rows are selected – no benefit2. Reduced dimension I/O, CPU and buffer space3. Faster join-back on list reports4. Very fast prompts, especially when constrained

27

_D_SD

_MD

Conceptual Size

Differences

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

2. Use them for very high level Fact Aggregates• Build a Fact aggregate at the Mini-Dimension level

� Allows greater field inclusion with excellent aggregation ratios• Multiple fields are available - not just one

� A good Mini Dimension and Skinny Fact/Aggregate can serve a large % of dashboard queries

� MD’s & Fact Aggregates offer extreme performance:1. Real World Ex #1: From time-out after 10 minutes to 4 seconds

2. Real World Ex #2: GL Account MD: 131X I/O benefit

28

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

� Aggregates are used when summary reports exist

� Pre-aggregate the dataset to make it smaller & faster

� Sometimes they are the only solution

� Typically a minimum of a 10:1 ratio is used

� Use all database tools as with any fact table • Partitioning, Indexing, Star Transformations, Compression

� For extreme needs, consider merging facts together• Ex: Monthly Actuals and Budgets

• Be mindful of gaps in datasets and non-conformed dimensions

� Advanced implementations use partition management to build only changed data for faster load times

29

Building The Performance Layer

Performance Tuning Oracle’s BI Applications

30Performance Tuning Oracle’s BI Applications

① Building the underlying tables is relatively simple

② But strong SQL & Tuning expertise is needed

③ Take cues from dashboard, report & RPD configuration

④ Any savings in I/O helps the overall system

⑤ Use all of the available database performance tools

Building The Performance Layer- Takeaways

Mapping Into Oracle BI

31

Link as much as possible to allow for the best performance across all

scenarios

32

BI Apps

Performance

Layer

Best

Better

Base

Performance Tuning Oracle’s BI Applications

Mapping Into Oracle BI

� The 3 Fact tables are mapped

like any aggregate

� The Skinny Fact (_SF) will

have fewer dimensions and

fewer metrics mapped to it• Along the Employee dimension however it

is the same as the base _F

� OBI will prefer to use the _SF

over the _F• Uses regular aggregate navigation

concepts

• Uses the _F when needed as a “backup

plan”

33

_A

_SF

_F

Performance Tuning Oracle’s BI Applications

Mapping Into Oracle BI

� Raise the priority group on the base _D to have

OBI prefer the _SDAs both LTS grains are identical, OBI needs more info to make a

choice

34

_D _SD

Performance Tuning Oracle’s BI Applications

Mapping Into Oracle BI

Create a dummy hierarchy level and map the LTSs for the Mini Dimension and Fact Aggregate to it

� The grain of the Mini Dimension is arbitrary

� As long as OBI knows it is higher than the other LTSs it will be preferred (Priority groups not needed)

35

_MD

_A

Performance Tuning Oracle’s BI Applications

Mapping Into Oracle BI

36Performance Tuning Oracle’s BI Applications

Table MappingThe mapping of tables is straightforward

Link Tables As Much As PossibleLet Oracle BI make the best choice

Mapping Into Oracle BI - Takeaways

Implementation

Considerations

37

The whole prototyping process can be done on a

simple star in roughly two weeks

Allow for more time if you have:• Large data volumes

• Difficult performance targets

• More complex models and logic

• Many disparate report patterns or lots of reports to consider

• More stars are needed (e.g., Actuals and Budgets together)

Development effort depends on # new objects• Typically only another two weeks needed (ETL & OBI RPD)

• A few more for regression test and deployment

38Performance Tuning Oracle’s BI Applications

Implementation Considerations

� Use Production data volumes for

accurate analysis

� Use Production DDL, ETL code,

OBI RPD and OBI Webcat

� Quiet, unused machine for

accurate benchmarking

� Use hardware that is as similar to

Prod as possible• KPI uses a database benchmarking tool to

compare environments

39Performance Tuning Oracle’s BI Applications

Implementation Considerations

� Additional ETL & RPD Development• Use SQL scripts instead of Informatica mappings (less effort, faster

execution)

� Additional Testing – Regression test is easy

� Additional ETL Run Time – may be critical

� Additional Database size - minor

� Customization Propagation / Impact Analysis• True of any aggregate

� Complex logic will be more difficult• E.g. #1: Financial Analytics uses snowflakes with multiple segment

hierarchies

• E.g. #2: HR Workforce Event & Snapshot logic uses effective dates for future dated events

40Performance Tuning Oracle’s BI Applications

Implementation Considerations

Q & A

41

www.kpipartners.com

The Leader In Oracle BI & EPM 42

Strategic Consulting | Systems Implementation | Training

Depot Repair Analytics

Fixed Asset Analytics

Manufacturing Analytics

Salesforce.com Analytics

Student Info Analytics

Subledger (SLA) Analytics

and moreF

Transform Data Into InsightTransform Data Into Insight

� Staff built from

Oracle/Siebel/Hyperion

engineering teams

� On-site, off-shore and blended

shore delivery models

� Exclusive pre-built solutions for

Oracle BI & E-Business Suite

Oracle BI

Hyperion

Endeca

Exalytics

Oracle BI

Hyperion

Endeca

Exalytics

Email: [email protected]

Web: kpipartners.com/contact

KPI World Headquarters

39899 Balentine Drive

Suite #375

Newark, CA 94560

Phone: (510) 818-9480

Contact Us

The Leader In Oracle BI & EPM 43

New York, NY

Chicago, IL

Boston, MA

Minneapolis, MN

San Diego, CA

Greensboro, NC

North America Offices

Bangalore, India Hyderabad, India

Global Offices

www.kpipartners.com

44