perfect solution outstanding experience. oracle materialzied views a performance tuning gem
TRANSCRIPT
perfect solution outstanding experience
perfect solution outstanding experience
ORACLE MATERIALZIED VIEWSA PERFORMANCE TUNING GEM
perfect solution outstanding experience
Agenda
• A Game Changer• What is a Materialized View• Purposes of Materialized Views• Enhancing SQL Performance• Query Rewrite• Real World Example• Advanced Query Rewrite Concepts• Q/A
perfect solution outstanding experience
A Game Changer
• Provide SQL Tuning Services for OLTP and DSS (Data Warehouse)
• Most Tuning Methods Consist of• SQL Redevelopment• Converting Correlated Sub Queries to Simply Sub Queries • Partitioning/Compression• Index Development• SQL Baselines
• Data warehouse tuning cases started to pop up where none of the conventional methods would provide the solution
perfect solution outstanding experience
A Game Changer
• Data warehouse performance goals
• The Report that Started it All• Local Fund Budget Report• Heavily Utilized Report Running almost 4 minutes• Conventional methods would not work
Type of Report General Performance Goal
Heavily Utilized 5 seconds or less
Moderately Utilized 30 seconds or less
Occasionally Utilized 2 Minutes or less
Maximum Time Allowed 3 Minutes
perfect solution outstanding experience
A Game Changer
• Local Fund Budget Report was tuned to return results in about 4 seconds, down from almost 4 minutes
• How did it happen?• The SQL in the report was not touched• New indexes were not added• The existing table structure behind the report remained the
same
SOLUTION:Query Rewrite using Materialized Views
perfect solution outstanding experience
What is a Materialized View
First, let’s define a View• A View stores a SELECT statement and executes the stored
SELECT statement when referenced via another SQL statement.
View: HR_DEPT_VW
SELECT DEPTID,MANAGER_IDFROM DEPT_TBLWHERE CATEGORY = ‘HR’
SELECT COUNT(1)FROM HR_DEPT_VW
Table: DEPT_TBL
DEPTIDMANAGER_IDCATEGORY
perfect solution outstanding experience
What is a Materialized View
Now, let’s define a Materialized View• Like a View, a Materialized View stores a SELECT statement, in
addition the results of the SELECT are stored in a table.
MView: HR_DEPT_MV
SELECT DEPTID,MANAGER_IDFROM DEPT_TBLWHERE CATEGORY = ‘HR’
Table: DEPT_TBL
DEPTIDMANAGER_IDCATEGORY
MView Table: HR_DEPT_MV
DEPTIDMANAGER_ID
perfect solution outstanding experience
What is a Materialized View
Now, let’s define a Materialized View• Like a View, a Materialized View stores a SELECT statement, in
addition the results of the SELECT are stored in a table.
SELECT COUNT(1)FROM HR_DEPT_MV
MView: HR_DEPT_MV
SELECT DEPTID,MANAGER_IDFROM DEPT_TBLWHERE CATEGORY = ‘HR’
Table: DEPT_TBL
DEPTIDMANAGER_IDCATEGORY
MView Table: HR_DEPT_MV
DEPTIDMANAGER_ID
perfect solution outstanding experience
Purposes of Materialized Views
• Replication• Easily replicate tables from one database to another
• Enhancing SQL Performance• Facilitate Incremental Warehouse Loading Operations (ETL)
• Use Materialized View Logs to track data changes in tables, which will be used to identify records needing to be updated, inserted, and deleted in the data warehouse.
perfect solution outstanding experience
Enhancing SQL Performance
• How do Materialized Views help SQL Perform Better?• Results of the associated query are stored in a table• Data can be pre-aggregated and pre-joined, cutting down the
resources and time needed to execute the query
TABLE: GL_ENTRIES
FISCAL_PERIOD_SIDENTRY_AMT
38,020,294 rows
MV: MV_GL_ENTRIES
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows2:40.03 Min
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM MV_GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows0:0.03 Min
perfect solution outstanding experience
Enhancing SQL Performance
• There are two ways to utilize MVs to enhance SQL performance• Directly referencing the MV as in the example below• Using the Oracle Query Rewrite functionality
TABLE: GL_ENTRIES
FISCAL_PERIOD_SIDENTRY_AMT
38,020,294 rows
MV: MV_GL_ENTRIES
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM MV_GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows2:40.03 Min
95 rows0:0.03 Min
perfect solution outstanding experience
Query Rewrite
What is Query Rewrite?• Query rewrite transforms a SQL statement expressed in terms of
tables or views into a statement accessing one or more materialized views that are defined on the detail tables.
• The transformation is transparent to the end user or application, requiring no intervention and no reference to the materialized view in the SQL statement.
• Because query rewrite is transparent, materialized views can be added or dropped just like indexes without invalidating the SQL in the application code.
perfect solution outstanding experience
Query RewriteExample of Query Rewrite
TABLE: GL_ENTRIES
FISCAL_PERIOD_SIDENTRY_AMT
38,020,294 rows
MV: MV_GL_ENTRIES
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
Generate Plan
Generate Plan
WINNER
perfect solution outstanding experience
Query RewriteExample of Query Rewrite
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
perfect solution outstanding experience
Query RewriteMaterialized View DDL
CREATE MATERIALIZED VIEW MV_GL_ENTRIES BUILD IMMEDIATEREFRESH COMPLETEENABLE QUERY REWRITE ASSELECT FISCAL_PERIOD_SID FISCAL_PERIOD_SID,SUM(ENTRY_AMT) ENTRY_AMTFROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
REFRESH COMPLETETells Oracle that when this MV is refreshed all data will first be removed and then a complete reload will occur. Most MVs created for Query Rewrite will be a complete refresh.
ENABLE QUERY REWRITETells Oracle that this MV can be considered for Query Rewrite. If this Query Rewrite is Disabled then Oracle will not consider this MV as an option for Query Rewrite.
BUILD IMMEDIATETells Oracle to run the SELECT statement and load the MV table right away. The Build can also be deferred.
perfect solution outstanding experience
Query RewriteThe STALENESS Factor• A MV can be in one of several STALENESS states
• FRESH – The data in the MV accurately reflects the data in the base tables
• STALE – Data in one or more of the base tables has changed and the MV may no longer accurately reflect the data in the base tables
• UNUSABLE – The MV is currently being refreshed• NEEDS_COMPILE – The table structure of one or more of the base
tables has changed and the MV needs to be compiled• UNKNOWN – One of two situations cause an UNKNOWN status
• An “ALTER MATERIALIZED VIEW….CONSIDER FRESH” has been run• The MV is based on a PREBUILT TABLE (Advanced Topic)
A Materialized View will only be considered for
Query Rewrite if it’s in one of the two STALENESS
states
OR STALENESS = STALE and
the following database parameter is set
query_rewrite_integrity=stale_tolerated
perfect solution outstanding experience
Query RewriteThe STALENESS Factor
TABLE: GL_ENTRIES
FISCAL_PERIOD_SIDENTRY_AMT
38,020,294 rows
MV: MV_GL_ENTRIES
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
Generate Plan
Generate Plan
WINNER
UPDATE GL_ENTRIESSET ENTRY_AMT = 3WHERE FISCAL_PERIOD_SID = 12
STALE
WINNER
perfect solution outstanding experience
Query RewriteThe STALENESS Factor
TABLE: GL_ENTRIES
FISCAL_PERIOD_SIDENTRY_AMT
38,020,294 rows
MV: MV_GL_ENTRIES
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
95 rows
SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID
Generate Plan
Generate Plan
WINNER
UPDATE GL_ENTRIESSET ENTRY_AMT = 3WHERE FISCAL_PERIOD_SID = 12
STALE
WINNER
BEGIN DBMS_MVIEW.REFRESH (‘MV_GL_ENTRIES’);END;
FRESH
perfect solution outstanding experience
Query RewriteThe STALENESS Factor• Query Rewrite works best for Materialized Views that are
based on tables that remain static throughout the day• Transactional or OLTP databases typically do not work well
due to the constant updating of data that will require constant refreshing of the MV to maintain a FRESH state.
• Query Rewrite works well with data warehouses
perfect solution outstanding experience
A Real World Example
• First, a little lingoSubject Area
Fact
Dimension
Dimension
Dimension
Dimension
Dimensions have a one-to-many
relationship with the Fact
Star Schema
Basically, just a bunch of tables
with data
perfect solution outstanding experience
A Real World Example
Subject Area
Fact
Dimension
Dimension
Dimension
Dimension
perfect solution outstanding experience
A Real World Example
Local Fund Budget Report
perfect solution outstanding experience
A Real World Example
Generate PlanCost 207,0003:53 Minutes
perfect solution outstanding experience
Materialized View
A Real World ExampleLet’s Create an Materialized View
Create Materialized View
CREATE MATERIALZIED VIEW KUALI_ADMIN.KF_MVA_BUDGET_BALANCE…SELECT BUD.FISCAL_PERIOD_SID…, SUM(BUD.UA_BUD_CY_ANN_EXP)FROM SYSADM.PS_UA_EMPL_JOB_DTL EMP, SYSADM.PS_UA_BUD_ACNT_TYP ACNT, KUALI_ADMIN.KF_D_ORG ORG, KUALI_ADMIN.KF_D_FISCAL_PERIOD FP, KUALI_ADMIN.KF_D_PROJECT PROJ, KUALI_ADMIN.KF_D_OBJECT_CODE OC, KUALI_ADMIN.KF_D_ACCOUNT ACCT, KUALI_ADMIN.KF_F_BUDGET_BALANCE BUDWHERE BUD.ACCOUNT_SID = ACCT.ACCOUNT_SIDAND BUD.OBJECT_CODE_SID = OC.OBJECT_CODE_SIDAND BUD.PROJECT_SID = PROJ.PROJECT_SIDAND BUD.FISCAL_PERIOD_SID = FP.FISCAL_PERIOD_SIDAND BUD.UA_BUD_ACCT_DEPT_ORG_SID = ORG.ORG_SIDAND BUD.BUD_ACCT_TYP_SID = ACNT.BUD_ACCT_TYP_SID (+)AND BUD.EMPL_JOB_DETAIL_SID = EMP.EMPL_JOB_DTL_SID (+)GROUP BY BUD.FISCAL_PERIOD_SID…
Pre-Aggregate
Pre-Join
perfect solution outstanding experience
Materialized View
A Real World Example
Generate PlanCost 207,0003:53 Minutes
Generate PlanCost 483
4 Seconds
WINNER
perfect solution outstanding experience
A Real World Example
Generate PlanCost 207,0003:53 Minutes
Materialized View
Generate PlanCost 483
3 Seconds
WINNERExplain Plan without MV
Explain Plan with MV
Expensive Joins
NO Joins
MV used 98.5% Less Temp Space
for Sorting
CBO Cost207,000 vs 483
perfect solution outstanding experience
A Real World Example
• Another alternative would have been to create a new, summarized Subject Area
• Benefits of Materialized Views over building additional summarized Subject Areas• Materialized Views are more agile and quicker to
implement• Existing reports need not be updated to point to a new
set of tables• Report writers will not have to be concerned with
choosing the best Subject Area from which to create reports
perfect solution outstanding experience
Advanced Concepts
perfect solution outstanding experience
In the Beginning…
• Due to differences in tables and joins in each report a different MV was created
• Every time a change was made to the report the MV had to be updated
• Became a maintenance nightmare
• Something had to change or MVs were not going to be a viable option
Budget MV1
Budget Report 1
Budget Report 2
Budget Report 3
Budget MV2
Budget MV3
Budget MV4
perfect solution outstanding experience
Budget Balances
A New Methodology…
Budget Balances
MV1
Budget Report 1
Budget Report 2
Budget Report 3
GL Entries
GL Entries MV1
GL Entries Report 1
GL Entries Report 2
GL Entries Report 3
Labor Entries
Labor Entries MV1
Labor Entries Report 1
Labor Entries Report 2
Labor Entries Report 3
perfect solution outstanding experience
How did we do it…
• Must Know Advanced Concepts• Query Delta Joins• Materialized View Delta Joins• Nested Materialized Views
perfect solution outstanding experience
Query Delta Joins
• A query delta join is a join that appears in the query but not in the materialized view.
• A query delta join will also occur if a field from a dimension is included in the query but is not included in the MV.
• In order for the join to work, the materialized view must contain the joining key.
perfect solution outstanding experience
Example Materialized View
Example Query Delta Join
MV Includes Budget Balance Fact joined to Account Dimension
MV Includes FK columns to the Object Code and Account Dimensions
perfect solution outstanding experience
Example Query Delta Join
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
MV ERD
Example Query #1
Consists of columns, tables, and joins that all exist in the MV
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
Query ERD
perfect solution outstanding experience
Example Query Delta Join
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
MV ERD
Example Query #2
This query adds the ACCOUNT_NBR field which does not exist in our test MV
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD- ACCOUNT_NBR
Query ERD
The Delta Join is made possible by including the joining key from the FACT in the MV
perfect solution outstanding experience
Example Query Delta Join
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
MV ERD
The Delta Join is made possible by including the joining key from the FACT in the MV
Example Query #3
OBJECT_CD and the Object Code Dimension was added to the query
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
Query ERDObject Code- OBJECT_CODE_SID- OBJECT_CD
perfect solution outstanding experience
Materialized View Delta Joins
• A materialized view delta join is a join that appears in the materialized view but not the query.
• All delta joins in a materialized view are required to be lossless with respect to the result of common joins. (We’ll get to this concept a little later.)
perfect solution outstanding experience
Example Materialized View Delta Join
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
MV ERD
MV Delta Join is made possible by guaranteeing a “lossless” join between the FACT and DIMENSION
Budget Balances- SUM(UA_BUD_CY_ANN)
Query ERD
Query Example
This query removes any reference to the Account Dimension, which exists in the MV definition.
perfect solution outstanding experience
Combined MV and Query Delta Join
Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)
Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD
MV ERD- Removes the Account Dimension (MV Delta Join)- Adds the Object Code Dimension (Query Delta Join)
Budget Balances- OBJECT_CODE_SID- SUM(UA_BUD_CY_ANN)
Query ERDObject Code- OBJECT_CODE_SID- OBJECT_CD
Query Example
perfect solution outstanding experience
How are Delta Joins Possible?
• Table Relationships Must be Configured Properly• Lossless Joins• Non-Duplicating Joins
• Dimension Join Keys must be included in the SELECT of the MV definition
perfect solution outstanding experience
Lossless Joins
• A lossless join guarantees that the result of common joins is not restricted
Budget Balances- ACCOUNT_SID- COUNT(1)
Fact
Account- ACCOUNT_SID
Dimension
ACCOUNT_SID = ACCOUNT_SID
No rows were lost when Budget Balance was joined to Accounts
LosslessJoin
perfect solution outstanding experience
Lossless Joins
• A lossless join guarantees that the result of common joins is not restricted
Budget Balances- ACCOUNT_SID- COUNT(1)
Fact
Employee Detail- EMPL_JOB_DETAIL_SID
Dimension
EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID
Not aLossless
Join
Rows were lost when Budget Balance was joined to Employee Detail
Not aLossless
Join
perfect solution outstanding experience
Lossless Joins
• A lossless join guarantees that the result of common joins is not restricted
Budget Balances- ACCOUNT_SID- COUNT(1)
Fact
Employee Detail- EMPL_JOB_DETAIL_SID
Dimension
EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID
Not aLossless
Join
Outer Joins will guarantee a Lossless Join
EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID (+)
LosslessJoin
perfect solution outstanding experience
Non-Duplicating Join
• A non-duplicating join guarantees that the result of common joins will not cause duplicates
• There must be a many-to-one relationship between the joins from the Fact to the Dimensions
Many-to-OneMany-to-Many
perfect solution outstanding experience
How to let Oracle Know
• To let Oracle know of the Lossless and Non-Duplicating joins the following must be configured on the tables in the subject area:• Primary Key constraints must be created on all
Dimensions• Foreign Key constraints must be create on Fact join
columns• Join columns on the Fact must be set to NOT NULL
unless that column will be outer joined to the Dimension
perfect solution outstanding experience
Example Primary Key Constraint
• PK and FK constraints can cause havoc and slowness in a data warehouse
• Oracle has provided a way to create the constraints while avoiding the baggage
DISABLE – Does not enforce the constraint on rows created after the constraint is created. Good for ETL.
NOVALIDATE – Does not enforce the constraint on rows that exist in the table when the constraint is created.
perfect solution outstanding experience
Example Primary Key Constraint
• If the PK and FK constraints are in DISABLE and NOVALIDATE mode there is one more step that must be taken in order to let Oracle know we can RELY on these constraints
RELY – Tells Oracle that regardless of the state of the constraint, Oracle can assume the constraint is valid. Without this MV Rewrite functionality will not work properly if constraints are not enabled.
perfect solution outstanding experience
Include Join Columns in MV
Subject Area
Fact
Dimension
Dimension
Dimension
Dimension
perfect solution outstanding experience
Nested Materialized Views
Budget Balances
Budget Balances
MV1
Budget Report 3
Budget Report 4
Budget Report 5
Budget Balances
Nested MV1
Budget Balances
Nested MV2
Budget Report 1
Budget Report 2
Budget Report 6
Budget Report 7
60 Minute Refresh
2 Minute Refresh
4 Minute Refresh
perfect solution outstanding experience
Compression
• Compressing data improves I/O because less blocks of data are read from or written to disk and sent back and forth between disk and database servers
• I/O is the biggest bottleneck in most Databases so this improves the performance of SELECTs
• Compression also saves disk space• Basic compression is standard with Oracle
Enterprise Edition and works well with MVs• Compress your MVs
perfect solution outstanding experience
Compression
Data Uncompressed Compressed
Size in MB (76.5% Compression rate) 829.44 194.56
Seconds to Query Shell Cd = C2 5.20 3.80
Seconds to Query Shell Cd = A41 21.87 12.99
Seconds to Query all Shell Cd 30.37 22.64
perfect solution outstanding experience
Document your MVs
• We document our MVs in wiki and include:• Reports tuned by MV• Data Model of the MV• Change History• Constraints required along with DDLs to create them• FK Index Requirements• Required Database Grants• Materialized View DDL• Materialized View Index DDLs
perfect solution outstanding experience
To Sum it Up
• Materialized View Query Rewrite is a powerful tool that can dramatically increase query performance
• Tuning your Warehouse with MVs instead of summarized subject areas is far more efficient and flexible
• Make sure to understand a few advanced concepts (Delta Joins and Nested MVs) before starting your adventure
• Use Compression• Document your MVs
perfect solution outstanding experience
Q/A
Sean KelleherEmail: [email protected]: 480.444.7714Cell: 602.363.9563Web: www.tbginc.com
Please feel free to contact Sean with any questions