oracle business intelligence 11g performance tuning obi ... · pdf file• on an oracle...

Oracle Business Intelligence 11g

Performance Tuning OBI 11g - All The Options

Antony Heljula December 2012

© Peak Indicators Limited 2

Agenda Performance Tuning OBI 11g – All the Options

• Overview • Identifying Bottlenecks • Oracle BI Optimisation • Database Optimisation • Data Model Optimisation • Multi-Dimensional Cubes • Upgrade Database Hardware • Oracle Engineered Systems • Real-Time Optimisation • In Summary


Performance Tuning OBI 11g – All The Options

Overview


Overview Performance Issues

• Organisations of all shapes and sizes can be affected by performance issues – it is not just large companies

• As data volumes grow rapidly it is becoming increasingly difficult to achieve consistent and scalable performance in the long-term – especially when attractive new BI features/functions add extra load onto the underlying data sources

• Performance issues are rarely down to a single cause, often it is a combination of factors

• The way performance issues are dealt with will be different in every Organisation and can be dependent on a variety of things such as:

• Urgency Ownership Quick win vs Long Term • Budget Politics Complexity of Data Source • Available Resources Licensing etc


Overview Aim of Presentation

• Oracle probably provides more optimisation capabilities than any other vendor (covering database, middleware, hardware etc)

• The aim of this presentation is to outline the key options that exist for tuning Oracle BI 11g, with the objective of achieving major performance gains rather than just “tweeks”

• Some options are low-cost and some are not! The aim is also is to help you maximise “business value” by maximising the benefit from the low-cost options before considering the next alternative

• The main focus of this presentation will be on optimising a Data Warehouse environment, but the final topic will cover “real-time optimisation”


Overview Order of Play

• The topics are structured in this order, which is generally how most Organisations would approach the resolution of performance issues against a relational Data Warehouse:

Oracle Engineered System

Optimise Database

Upgrade Database Hardware

Multi-Dimensional Cube Engine

Optimise Oracle BI

Addition DB Licensed Options

Optimise Data Model


Overview Before we start….

• It is vital the Organisations give themselves the best possible chance of addressing performance issues in the cheapest/shortest way possible:

• Maintain development environments with realistic “production” data volumes • Perform a suitable stress test before moving into Production • Never assume that performance will be better in the Production environment!

(Production may have bigger servers, but there will be 100x more users and bigger data volumes)

• To minimise costs, it is vital to identify and deal with performance issues early on in the development cycle, rather than wait until UAT/Production

• Try to avoid having 1 week at the end of your development project set aside for “performance tuning”. Developers need to develop with performance in mind – tackle performance issues as and when they occur



Oracle BI Optimisation


Optimise Database



Optimise Oracle BI


Optimise Data Model


Oracle BI Optimisation Overview

• With any BI implementation, most of the processing is done down at the database level, this means that quite often Oracle BI server will be sitting idle (waiting just like the end user is!)

• Oracle BI itself is configured by default for around 2000 total users per instance (catering for 100 concurrent sessions). These limits can be easily increased to scale for more users – but increasing them does not make the system run faster

• There is one major feature (BI Server “caching”) which can be used for optimisation, but the main focus should be on making sure Oracle BI is generating efficient queries and the dashboards/reports are not causing an unreasonable/unnecessary amount of load on the system


Oracle BI Optimisation General Recommendations

• When it comes to tuning the application itself, it is mostly down to being as reasonable/sensible as possible with the front end requirements to avoid performance issues:

• Keep the number of reports on a dashboard down to a minimum • Avoid having large pivot tables where possible • Encourage the use of drill-downs. So you drill-down from a summary report

(rather than just starting with a large detail report) • Minimise the Agents (schedules) being sent during working hours (especially

those which are delivered to many recipients – it is all done concurrently) • Minimise the columns left in the “Exclude” sections within Answers (if columns

aren’t needed then remove them completely from the report) • Make sure query filters make appropriate use of database indexes


Oracle BI Optimisation Avoid “Excluded” Columns

• A common mistake is to end up with lots more columns to the report than is actually required

• In the example below, the final report only shows 3 columns with 4 more being “excluded”

• The SQL queries will however involve all 7 columns • Recently we have found this type of issue to cause significant performance

degradation on customer sites ranging from 7 seconds to 3 minutes per report


Oracle BI Optimisation Efficient Query Filters

• It is also quite easy for report developers to write inefficient query filters • One common mistake is to apply functions on to columns. This is because applying a function on a

column will render any indexes on that column unusable • In the example below, the “TimeStampAdd” function was applied to the “Order Date” column

rather than to the variable the other side. The database could not use the index on the “Order Date” column so a full table scan was performed instead

• By correcting the logic the query ran 2x faster (7 seconds instead of 16)

TimeStampAdd(“Order Date”, 7) is greater than @{VAR_DATE} “Order Date” is greater than TimeStampAdd(@{VAR_DATE}, -7)

If you want to make use of an index, don’t apply a function

on the column


Oracle BI Optimisation BI Server Caching

• Caches the results of “Logical” SQL queries issued to the BI server

• BI Server will fetch results from the server cache if a user issues the same query or a sub-set of a previous query

• Cache is shared amongst users and across multiple BI servers

• Tuned for large volumes of “small” result sets (<10K records)

• Cache can be configured (number of files, total volume, max number of records, max file size etc)

WebLogic

BI Server

BI Presentation Services


Oracle BI Optimisation BI Server Caching • BI Server Caching can bring significant gains in reporting performance – queries

returned from BI Server cache are always instant

• Performance issues can sometimes be resolved by “pre-seeding” the BI Server cache with report results prior to the users running their reports

• But BI Server Caching is not your “silver bullet”! • Although it can provide valuable and cost-effective “quick-wins” to resolve

performance issues, it does have limitations • BI Server caching doe not replace the need for “longer-term” solutions (such as

enhancing the physical data-sources to be generally more efficient for reporting)


Oracle BI Optimisation Oracle BI Server Caching - Limitations

• To effectively tune a Dashboard Page using BI Server Caching, you have to build reports that will cache all the possible dashboard prompt selections and column selector combinations

• Otherwise performance will dive as soon as the user makes a new selection • Doing this for every dashboard page is not maintainable or scalable in the long term,

each result set could be massive! (well beyond the limits of what is supported)



Database Optimisation


Optimise Database



Optimise Oracle BI


Optimise Data Model


Database Optimisation Summary

• On an Oracle database, there are many tuning methods which are commonly used on a Data Warehouse. For example:

• Gather statistics • Parallel Query • Star Transformation • Bitmap Indexes

• We will attempt to answer the following questions:

• Do they actually work? • What performance gains do they actually deliver? • In what situations are they most effective?

• Partitioning • Bitmap Join Indexes • Compression

NOTE: Items in red require Oracle

DB Enterprise Edition

Partitioning is an additional licensed option for DB Enterprise Edition

Oracle OLAP is discussed

later on



Performance Test

• A performance test was conducted based on the following: • Star-schema data-model consisting of 30 million fact records • 4 BI Dashboards with “Summary” reports (multiple years/months) • 2 BI Dashboards with “Drill-Down” reports (single month/organization etc)

• Database Server consisted of:

• 1 x dual core CPU • 1 x Hard Disk • 4 x GB RAM

NOTE: This is very lightweight hardware for a database, but it is useful for exaggerating performance issues.

It is not appropriate hardware for testing Parallel Query, so a proper

test for Parallel Query will be shown later



Overall Results

• With all these tuning options: • Summary reports are 2x faster (but still too slow!) • Detail reports are 27x faster



Another Test - Parallel Query

• Another test was performed to highlight the benefits of Parallel Query – especially when it comes to scalability:

• Database Server with 8 x CPU cores, 8 x HDD, 100GB RAM • Tests involved running queries processing 2.5K to 25Million records

Without parallel query enabled, scalability becomes an issue

Parallel Query scales better and is 3x faster when

processing 25 million rows Performance was comparable until you

process >250K records



Summary



Data-Model Optimisation


Optimise Database



Optimise Oracle BI


Optimise Data Model


Data-Model Optimisation Overview

• We will now discuss two commonly used methods for improving the performance of database queries: • Removal of Snow-Flakes/Denormalisation • Aggregation

• The first one aims to improve performance by reducing the number of table joins

• Aggregation aims to improve performance by summarising data so there is less data process at report run-time


Data-Model Optimisation Removal of Snow-Flakes

• In our test data-model we have 3 snow-flakes, this is where the you get multiple dimension tables hanging off each other

• In the example above, if a SQL query need data from all tables then it would require 8 table joins….and this could hinder performance?


Data-Model Optimisation Removal of Snow-Flakes

• As a test, we merged the dimensions together so that now there were only 5 tables in total and a maximum of 4 table joins:


Data-Model Optimisation Removal of Snow-Flakes - Result

• The result was very surprising!

• By moving FKs on to the central fact table: • “Summary” queries increased from 58 to 60 seconds on average • “Drill-down” queries increased from 37 to 40 seconds on average

• By merging the tables together we are increasing data volumes (as the data is not as normalised). This increases the number of disk reads – which is our main bottleneck

• With “Star Transformation” and other database optimisations enabled maybe the sensible approach is to focus on reducing data volumes instead of reducing table joins?


Data-Model Optimisation Removal of Snow-Flakes – 2nd Approach

• A 2nd approach was tested. This time, instead of merging the Dimension tables together, we moved the FK columns in to the central fact table

• This meant all the necessary dimension columns were linked directly to the central fact

• In theory “Star Transformation” prefers to have pure star-schema diagrams like the one on the right-hand side below:


Data-Model Optimisation Removal of Snow-Flakes – 2nd Approach

• The result again was very surprising!

• By moving FKs on to the central fact table: • “Summary” queries increased from 58 to 137 seconds on average • “Drill-down” queries increased from 37 to 70 seconds on average

• So it appears that with modern database optimisation techniques such as “Star Transformation” and “Bitmap Indexes”, it is more important to keep your central fact table small to reduce the total number of physical disk reads (which is commonly the main bottleneck in a BI system)

• Lesson Learnt: Reducing the number of joins is not always effective if it means you have to increase the volume of data


Data-Model Optimisation Aggregation

• Earlier on we had managed to tune a set of reports using standard database features, but the response times of the “Summary” reports were still too high

• We will now try out the effectiveness of creating a set of summary tables (Materialized Views) and configure Oracle BI to make use them whenever it was appropriate to do so

By today’s standards, an

average response time of 36 seconds

is too high


Data-Model Optimisation Aggregation

• In our test environment, we have managed to build a single aggregate which can: • Reduce the number of records from 31M down to 8M • Support all 4 “Summary” dashboards • Provide at least 2 levels of drill-down across all the hierarchies


Data-Model Optimisation

Aggregation - Results • Finally, with just one aggregate all our

response times are now acceptable!



Summary so Far



Multi-Dimensional Cubes


Optimise Database



Optimise Oracle BI


Optimise Data Model


Multi-Dimensional Cubes Overview

• Oracle has two well-established “cube” engines in its product line: • Oracle OLAP • Oracle Essbase

• Whilst they are architecturally and functionally quite different, they

both offer good potential for optimising your BI infrastructure


Multi-Dimensional Cubes When Do You Need a Cube Engine?

• Sometimes however, data volumes or reporting complexity are such that the level of “optimisation” required goes beyond manageable levels – you have too many relational aggregates:

Oracle BI

MVs


Multi-Dimensional Cubes When Do You Need a Cube Engine?

• A Cube Engine’s calculation performance is sometimes so superior you can sometimes replace the complexity with just a single Cube

Oracle BI

OLAP

Essbase

Oracle OLAP is the cube engine inside

the Oracle Database

Essbase is the cube engine used as the

primary data source for all Hyperion EPM applications. It sits

outside of the Oracle Database on its own

dedicated servers


Multi-Dimensional Cubes Cube vs Relational

• The benefits of Oracle OLAP and Oracle Essbase are clear to see in this performance chart

• The performance of OLAP/Essbase is far great than an equivalent set of relational MVs – especially as the number of concurrent queries grows

• Based on our own internal tests

• Useful for “Summary” reports only



Disadvantages

• There are some factors however that reduce the attractiveness of cube engines: • Storage required for cube engines can be far greater • Cube engines do not scale as well (large cubes can be harder to maintain than the equivalent

relational structures) • Typically, the main dimension hierarchy columns are included in the cubes, not the whole

range of dimension attributes • Not so useful for detail / drill-down queries • Data quality rules are stricter than is required for relational structures (so you may not

always be able to build the cubes you want without fixing data first) • Oracle BI does not support “all” features of the cube engines • Not all types of Oracle BI functions supported e.g. Time Series • If the cube structure changes, it has to be re-modelled from scratch (Oracle OLAP)



Unofficial Recommendation

• Based on the disadvantages and in light of other performance tuning techniques which are now available:

• If you are already licensed for Oracle Essbase or Oracle OLAP then it could be worth trying it out

BUT • If you do not have licenses purchased already then you may want to consider

other options first e.g. relational aggregation, upgrading db hardware or an Oracle Engineered System





Optimise Database



Optimise Oracle BI


Optimise Data Model


Upgrade Database Hardware Overview

• If you fully understand your existing bottlenecks, then new hardware can provide significant improvements to system performance and scalability

• Remember – hardware costs are always on the decrease, so you might be surprised at what is available!

• “Commodity” servers with large amounts of memory (>100GB) are common nowadays. A lot of companies could easily fit their databases totally inside the DB Cache

• New database hardware can easily make 2-3x performance gains without having to reconfigure the application


Upgrade Database Hardware Possibilities and Benefits

Hardware Component

Benefits

Increase CPUs Greater parallel query capability Greater scalability to support more users (Might incur additional license cost)

Increase RAM Increasing memory available for the Database Cache will result in reduced disk reads – easy way to double your performance. For example: A 60 second query coming from disk reduced to 30 seconds when the data was totally cached

Increase Number of Hard Disks Spreading data over more disks increases disk throughput, reducing the bottleneck. Doubling the number of disks can reduce query times by 25-50%

Use Solid State Drives (SSD) Not as fast as RAM, but is cheaper SSDs are much faster than using Hard Disks. Either replace your Hard Disks completely with SSDs or use the Oracle Database “Smart Flash Cache” where data can be cached on SSD, as an extension to the DB Cache (NOTE: SFC available on linux/solaris only)


Upgrade Database Hardware But Remember!

• When judging a server’s performance, most people will look at the number of available CPUs (cores)

• But For a database server, memory and disk throughput is more important! There is no point having 16 CPUs if they have to sit idle waiting for the disks

• Also, disk capacity is always increasing, it leads people to buy fewer disks. The number of disks you have is more important than their individual capacity! Stripe data over as many disks as possible Buy as much memory as possible Make use of SSDs to give an extra performance boost (if your data volumes

exceed the total RAM available)



Oracle Engineered Systems


Optimise Database



Optimise Oracle BI


Optimise Data Model


Oracle Engineered Systems For when only a Ferrari will do?

• Oracle provides its own Engineered Systems which adopt all the latest optimisation techniques e.g. in-memory, SSD, columnar compression

• Three relevant ones are:

• Oracle Exalytics • Oracle Database Appliance • Oracle Exadata

• Significant performance gains 20x, 40x, 50x faster etc

• They come with some features that you don’t get if you build servers yourself!


Oracle Engineered Systems Exalytics

One customer found some reports (especially with pivot

tables) ran 20% faster on Exalytics without any tuning or

configuration


Oracle TimesTen Customer Example

• With TimesTen 11.1.1.2 in-memory aggregates, each dashboard page was on average 26x faster compared to the Data Warehouse • 25 queries tested across 6 dashboard pages Aggregates reside in-memory

and on the BI Server: No disk bottlenecks

No network bottlenecks


Oracle Engineered Systems Oracle Database Appliance

• 2 x Oracle Linux server nodes • A total of:

• 24 x Intel Xeon CPU cores • 192 GB RAM • 12 TB of Hard Disk storage (20 x HDDs) • 946 GB of SSD storage for DB redo logs (4 x SSDs)

• Pay-as-you-grow pricing

• Hybrid Columnar Compression available for 10-15x compression


Oracle Engineered Systems Oracle Exadata X3-8

• 160 CPU cores • 4 TB memory • 22.4 TB of Smart Flash Cache (SSD) • 50 TB hard-disk storage • Hybrid Columnar Compression • 168 CPU cores at disk level – to process/filter data on the disks rather than

in the database cache

• Eight, Quarter and Half rack configurations available



Real-Time Optimisation


Real-Time Optimisation Overview

• Using a BI tool to report against real-time (OLTP) data sources can provide significant performance headaches:

• Data structures are designed for individual insert/updates not large SQL queries • Queries often are complex, involving many tables • Outer-joins often required • Few options for improving performance (you can’t just enable Star Transformation!) • No scalability, query response times grow exponentially as data volumes increase:

OLTP Data Volumes

Query Response Time


Real-Time Optimisation The Solution

• Build an Operational Data Store (ODS) • Data extracted frequently from the OLTP • Data stored in ODS in a format suitable for reporting • ODS only stores the data needed for near-real-time reporting

OLTP OD S

Oracle BI 11g


Real-Time Optimisation The Solution

• Because an ODS only ever contains the data needed for “operational” reporting, the data volumes do not grow

• This means an ODS is 100% scalable in the long term • For example, consider a call centre:

• The number of “closed” Service Requests will grow over time • The number of “open” Service Requests will stay constant • For real-time reporting, the ODS only needs to contain “open” Service Requests

OLTP Data Volumes

ODS Data Volumes


Real-Time Optimisation Two Examples

• Financial News & Media organisation: • ODS implemented for “real-time” reporting against CRM system • ODS updated with 1 minute cycles • 8000% performance improvement across 450 real-time reports

• Global Corporate Finance organisation: • ODS implemented for “real-time” reporting for planning HR compensation • Previously users could not report directly off OLTP (queries had to run against

15+ tables and were too slow). This meant users could only generate reports once a day

• Data now available for reporting in ODS within 15 seconds, down from 24 hours



In Summary


In Summary • There are a wide range of performance optimisation techniques available –

applicable to customers of all sizes and types

• The most effective tuning options are (in no particular order): • Data aggregation • Star Transformation • Parallel Query • Table Partitioning • Oracle Exalytics

• “in-memory” optimisation is becoming the trend. Examples are: • Oracle Exalytics 1 TB RAM • Oracle Exadata 4 TB RAM, 22 TB Flash Cache

Lessons learnt from these 5 options: 1) Summarise data 2) Query efficiently 3) Query in parallel 4) Quickly Exclude irrelevant data 5) Process more in-memory


Questions?

Helping Your Business Intelligence Journey

oracle business intelligence 11g performance tuning obi ... · pdf file• on an oracle...

Documents