Download - Ugif 04 2011 france ug04042011-jroy_part1
© 2010 IBM CorporationApril 6, 2011
Informix Warehouse Accelerator
Jacques RoyIBM, Informix development
2 © 2010 IBM Corporation
Agenda
■ Data warehouse industry trends■
■ Data warehouse on Informix■
■ Infomrix warehouse accelerator■
3 © 2010 IBM Corporation
Sate of Data Warehousing 2011
DBMS Market in 2011:■ DBMS market at the close of 2009 was approximately $21.2
billion (2010 data not yet available)■ Data Warehouse DBMS market was approximately 35% of the
DBMS market or $7.42 billion
Key Findings:■ Data warehouse DBMSs have evolved to a broader analytics
infrastructure supporting operational analytics, corporate performance management and other new applications and uses.
■ Cost is driving interest in alternative architectures but performance optimization is driving multi-tiered data architectures and a variety of deployment options - notably a strong interest in in-memory data mart deployments.
4 © 2010 IBM Corporation
Sate of Data Warehousing (cont.)
Market Dynamics for 2011■ Today, smaller data warehouses, those less than 5 TB's of source
system extracted data (SSED) are the only "data warehouse" for the entire organization and are commonly solving organizations' analytic needs.
Analysis:■ Gartner only rarely encounters an organization which has actually
delivered on the Enterprise Data Warehouse (EDW) vision. The EDW remains a design principle, but it is rarely if ever actually deployed. Gartner estimates that between 70% and 75% of all systems referred to as EDW are actually single business departments in nature.
■ Optimization techniques such as summaries, aggregates and indexes are simply the result of performance restrictions inherent to normalized data and the way the RDBMS manages rows and columns.
5 © 2010 IBM Corporation
State of Data Warehousing (Cont.)
A Glimpse Into the Future■ Vendor solutions began to focus even more on the ability to
isolate and prioritize workload types including strategies for dual warehouse deployments and mixing OLTP and OLAP on the same platform.
■ In-memory DBMS solutions provide a technology which enables OLTP/OLAP combined solutions. Organizations should increase their emphasis on financial viability during 2011 and even into 2012 as well as aligning their analytics strategies with vendor road maps when choosing a solution.
7 © 2010 IBM Corporation
Existing Informix Warehouse Features
■ Performance & Scalability– Inherent SMP Multi-threading– Parallel Data Query (PDQ)– Light Scan for fast table scans– Online Index build– Efficient Hash Joins– Auto Fragment Elimination– Memory Grant Manager (MGM)– High Performance Loader– Optimistic Concurrency
■
■ Easy of Management– Time cyclic data management using Range Partitioning– Sophisticated Query Optimizer for OLTP and Warehousing
8 © 2010 IBM Corporation
Informix Warehousing Moving Forward
■ Goal is to provide a comprehensive warehousing plat form that is highly competitive in the marketplace–
– Incorporating the best features of XPS and Red Brick into IDS for OLTP/Warehousing and Mixed-Workload
–
– Using the latest Informix technology in:• Continuous Availability and Flexible Grid• Data Warehouse Accelerator using latest industry technology
–
– Integration of IBM’s BI software stack
13 © 2010 IBM Corporation
BI Tools for Informix
The Performance Management Framework Cognos identifies best-practice decision areas, or information sweet spots by business function:
Cognos 10 provides a comprehensive set of BI tools for:
� Reporting
� Analysis
� Dashboards
� Scorecards
Performance Management Framework for:
� Solutions for different areas of the organization
14 © 2010 IBM Corporation
Third Generation of Database Technology
According to IDC’s Article (Carl Olofson) – Feb. 2010
1st Generation:
- Vendor proprietary databases of IMS, IDMS, Datacom
2nd Generation:
- RDBMS for Open Systems, dependent on disk layout, limitations in scalability and disk I/O
- Database tuning by adding updating stats, creating/dropping indexes, data partitioning, summary tables & cubes, force query plans, resource governing
3rd Generation: IDC Predicts that within 5 years:
■ Most data warehouses will be stored in a columnar fashion
■ Most OLTP database will either be augmented by an in-memory database (IMDB) or reside entirely in memory
■ Most large-scale database servers will achieve horizontal scalability through clustering
15 © 2010 IBM Corporation
Market Data: Key Drivers of Change
16 © 2010 IBM Corporation
Informix Warehouse Accelerator
How is it different?• Performance: Unprecedented response
times to enable 'train of thought' analysis frequently blocked by poor query performance.
• Integration: Connects to IDS through deep integration providing transparency to all applications.
• Self-managed workloads: queries are executed in the most efficient way
• Transparency: applications connected to IDS, are entirely unaware of IWA
• Simplified administration: appliance-like hands-free operations, eliminating many database tuning tasks
What is it?
The Informix Warehouse Accelerator (IWA) is a workload optimized, appliance-like, add-on, that enables the integration of business insights into operational processes to drive winning strategies. It accelerates select queries, with unprecedented response times.
Breakthrough Technology Enabling New Opportunities
17 © 2010 IBM Corporation
Breakthrough Technologies for Performance
1
2
34
5
6
7 1
2
34
5
6
7
Row & Columnar DatabaseRow format within IDS for transactional workloads
and columnar data access via accelerator for OLAP queries.
Extreme CompressionRequired because RAM is the limiting factor.
Massive ParallelismAll cores are used within used for queries
Predicate evaluation on compressed data
Often scans w/o decompression during evaluation
Frequency PartitioningEnabler for the effective parallel access of
the compressed data for scanning. Horizontal and Vertical Partition
Elimination.
In Memory Database3rd generation database technology avoids I/O. Compression allows huge databases
to be completely memory resident
Multi-core and Vector Optimized Algorithms
Avoiding locking or synchronization
18 © 2010 IBM Corporation
IWA: Characteristics
• A dedicated SMP system (Linux on Intel x86_64)• No changes to the applications
–Applications continue to attach to IDS.
–When applicable query needs to be executed, IDS exploits the accelerator transparently to the applications
–Fencing and protection of IDS against possible accelerator failures
• Order of magnitude performance improvement • Reducing need for tedious tuning of IDS (partitioni ng, indexes, etc.) • Appliance-like form-factor
–Hands free operations
• Significantly improved price/performance and TCO as a combined effect of:
–Accelerating intensive & complex analytics queries
–Orders of magnitude performance improvement for accelerated queries
–Reduced DBA effort for tuning accelerated queries
19 © 2010 IBM Corporation
Sample Customer Results: Case Study #1
Query Description Informix Informix w ISAO Notes Improvement
1 Find Top 100 Entities 1:28:22 0:01:28 Fact Table Scan 6023.23%
2 Find Top 100 Members 1:22:32 0:01:05 Fact Table Scan 7640.45%
3Summarize all transactions by State
and County 1:34:37 0:00:14 Fact Table Scan 41708.49%
4Summarize the top 10 Commodities
by State and County 1:05:03 1:03:35
IWA did not support this subquery query 102.29%
5
Detailed Report on Specific Programs, States, Counties and Years 0:00:00 0:00:00 Index Read 83.45%
6Detailed Report on Specific
Programs in a Date Range 0:00:06 0:00:06 Index Read 108.41%
7
Summarize all transactions by State, County, City, State, Zip, Program, Program Year, Commodity and Fiscal Year 1:48:58 0:00:41 Fact Table Scan 15800.89%
8
Find Entities where the payments do not equal total Member Transaction Amounts
Failed - Long Transaction
Failed - Long Transaction
I did not configure enough logs to support the query
Totals 7:19:37 1:07:09 654.69%
20 © 2010 IBM Corporation
Government Agency Datamart
� Performance expectation goals were up to 20X OLAP-style Queries
� Tests were done on a Intel x86_64 SMP box running Linux RHEl
� Microstrategy Report was used, which generates 667 SQL statements
�537 are SELECT statements.
� Datamart for this report has 250 Tables and 30 GB Data size
� Informix Panther and IWA run this report in 67 seconds.
� 7 seconds in IWA and 60 seconds in Informix (TEMP table processing, etc)
� Without IWA, total runtime on Informix 11.70 on the same HW is 40 Minutes!
� The same report today runs on XPS & SUN HW (Sparc M9000) and takes 90 mins.
� Performance gain for the customer would be ~90x !!!
21 © 2010 IBM Corporation
IWA Referenced Hardware Configuration
Intel(R) Xeon(R) CPU X7560 @ 2.27GH 4 X 8
Memory 512G
6 disks 300 GB SAS hard disk drives each
- 4-processor, 4U rack-optimized enterprise server with Intel® Xeon® processors.
- 8-core, 6-core and 4-core processor options with up to 2.26 GHz (8-core), 2.66 GHz (six-core) and 1.86 GHz (four-core) speeds with up to 16 MB L3 cache
- Scalable from 4 sockets and 64 DIMMs to 8 sockets and 128 DIMMs
- Optional MAX5 32-DIMM memory expansion
- 16x 1.8" SAS SSDs with eXFlash or 8x 2.5" SAS HDD s
Options:
22 © 2010 IBM Corporation
IWA Software Components
■ Linux on Intel x86_64 (RHEL 5 or SUSE SLES 11)■
■ IDS 11.70 + IWA code modules including IDS Stored Procedures (Informix Ultimate Warehouse Edition)
■
■ ISAO Studio Plug-in – GUI for Mart definition■
■ OnIWA – On Utilities for Monitoring IWA