how to increase performance in ibm cognos
TRANSCRIPT
© 2015 IBM Corporation
PDA for CognosCompeting with SQL Server
Sanjeev Datta – Cresco Practice DirectorMark Yingling – IBM Analytics Solution Architect
[email protected]@us.ibm.com
© 2015 IBM Corporation
Fast on Fast Analytics
The Synergy of IBM Cognos and IBM PureData System for Analytics
3 © 2015 IBM Corporation
IBM Cognos Business Intelligence
Leverage data: Access information in any volume, combination and complexity
Provide insights: Understand your business like never before through self-service analysis at any time on any device
Make confident decisions: Validate your analysis by leveraging predictive information to gain complete visibility into your business
Outperform expectations: Transform your business from a reactive operation to a successful and proactive market leader
A forward-looking view of your business performance through stunning dashboards and reports
4 © 2015 IBM Corporation
Data
Database
Cognos Business Intelligence
RDBMS Adapter
SQL
Data
Database
Data
Data
Growing HistoryMore SourcesRelated DataExtensive TuningInitial Use Case
Limited HistoryTuned Query
SQL SQL
Increased Data Volume
Over time
5 © 2015 IBM Corporation
Multiple Data Sources
Cognos Business Intelligence
RDBMS RDBMS Files Files . . .
External Sources
SQL
• Multiple interfaces• Data movement• Impact on sources • Data consistency• Specialized tuning• Where’s the data?
SQL
6 © 2015 IBM Corporation
Multiple Data Sources
Cognos Business Intelligence
RDBMS RDBMS Files Files . . .
• Multiple interfaces• Data movement• Impact on sources • Data consistency• Specialized tuning• Where’s the data?
External Sources
SQL
Data warehouse
7 © 2015 IBM Corporation
Multiple Data Sources
• Single database interface• No data movement to Cognos layer for joins, etc.• Standardized administration and tuning• Data quality handled during warehouse load• Reduced impact on source systems• Improved performance for queries and reports
Cognos Business Intelligence
RDBMS
Data warehouse
8 © 2015 IBM Corporation
Requirements Summary• Handle large and growing data volumes
• Integrate into existing environment
• Leverage Cognos and relational database skills
• Provide better and predictable performance
• Minimal to no database and system administration and tuning
• Support for a variety of workloads – queries, reporting, dashboards, analytics
• Simple to get up and running
RDBMS
Data warehouse
Cognos Business Intelligence
9 © 2015 IBM Corporation
Solution
C O G N O S +PureData System for Analytics
10 © 2015 IBM Corporation
Appliance Features Production ready Rack mountable appliance Installed in a standard, customer provided rack Entire integrated appliance tested and packaged at the factory Full function Netezza Platform Software (NPS) with IBM Netezza Analytics Self Encrypting Drives; Up to 16TB1 of user data
Ease of Use Same ease of use and features as larger appliances
- Load and go with no tuning or administration Installation by IBM or an IBM Partner certified to install the N3001-001
Availability & Support Highly available, Full redundancy
− All redundant hardware, 4 disk spares, hot swap power supply Remote access for support; Call Home enabled
1 Assuming 4X compression
PureData System for Analytics N3001-001Bringing speed and simplicity to midsize organizations for big outcomes
11 © 2015 IBM Corporation
IBM Netezza Analytics
Bring the analytics to the data not the data to the analytics
Included
Features
Built-in, in-database analytic functions- Data mining, prediction, transformations,
statistics, geospatial, data preparation Full integration with tools for BI &
visualization- IBM Cognos, Microstrategy, Business
Objects, SAS, MS Excel, SSRS, Kognitio, Qlikview
Full integration with tools for model building & scoring- IBM SPSS, SAS, Open Source R, Fuzzy
Logix Full integration for custom analytics
- Open Source R, Java, C, C++, Python, LUA
Data Preparation
Predictive Analytics
Geospatial Analytics
Advanced Statistics
12 © 2015 IBM Corporation
Big Data and Business Intelligence Ready
Real-time AnalyticsInfoSphere Streams Developer Edition 2 users, non-production licenses
Business Intelligence Cognos software, 5 Analytics User licenses, plus 1 Analytics Administrator license
Hadoop Data ServicesIBM BigInsights v4 for Apache Hadoop® to manage ~100 TB of Hadoop data
Included with the PureData System for Analytics N3001
Data Integration & TransformationInfoSphere DataStage 280 PVUs, 2 concurrent Designer Client licenses and InfoSphere Data Click
Data Warehouse Appliance
Up to 16TB capacity for your Data Warehouse / Data Mart
IBM Fluid Query
Supporting Hadoop Solutions and Streaming Analytics
Open Source “R”
Netezza Analytics
13 © 2015 IBM Corporation
Use cases
Features
Business IntelligenceThe power of IBM Cognos with PureData for Analytics
Leading Business Intelligence- Interactive analysis- Compelling visualizations - web, mobile or email- Enterprise scalability
Optimized for PureData for Analytics- Offers high performing OLAP over relational
experience- Cognos Dynamic Query Mode extends benefits of
PureData by adding in-memory & caching on top of already fast appliance performance
- Exploits Netezza analytic in-database functions
Rapid deployment of answers to key business questions
Included with PureData for Analytics:IBM Cognos Business Intelligence 10.2.1
5 Analytics User licenses, 1 Analytics Administrator license1
Included
Reporting, analysis, scorecards, dashboards Data visualization Mobile business intelligence … and many others
1PureData System for Analytics N3001 must be the data source for Cognos.
14 © 2015 IBM Corporation
When is PureData System for Analytics a Good Fit
Data Volume– At least 0.5 TB of data
Performance– The existing data warehouse/mart solution is not performing– Lots of aggregate tables are required to make the DW perform
• Increases tuning effort and reduces flexibility Maintenance & Customer skill set
– Many resources are required to maintain the data warehouse (>1-2 DBA’s)– The skill set required to tune the existing system is high– The DBA team is slow to react to new business requirements and resulting
query patterns
15 © 2015 IBM Corporation
Dynamic Query Mode is optimized for PDA
Offers a high-performing OLAP Over Relational experience via hybrid
SQL/MDX techniques
Avoids redundant queries through security-aware metadata, data, and
query plan cache management
Provides built-in query visualization tool
Leverages 64-bit architecture
Uses JDBC connection to Netezza
Advanced sorting behavior that aligns DMR queries with other OLAP data
sources
16 © 2015 IBM Corporation
Executing a Dimensionally Modeled Relational (DMR) report with Dynamic Query Mode
Dimensional report results in MDX query against execution engine If the dimension and measure data is in cache, query is computed directly without
accessing database If the data is not in the cache the necessary data is gathered with a relational
SQL query
Using Cognos with PureData for Analytics
17 © 2015 IBM Corporation
High performance analytics over growing data volumes
Aggregate awarenessAggregate acceleration
Optimize in-memory caching with in-database processing
Dynamic Cubes
18 © 2015 IBM Corporation
• Security is applied on top of the caches, so all users benefit
BI query service
Database
Warehouse
Aggregates
Netezza Data Warehouse
Result Set Cache
Expression Cache
Member Cache
Query Data Cache
Aggregate Cache
Over 80% of queries are < 3 seconds
Over half of queries are sub-second
Dynamic Cubes find the shortest path to the answer
19 © 2015 IBM Corporation
TPC-DS 10 TB warehouse performance with Dynamic Cubes
28.8 billion row fact table 65 million members in largest
dimension (Customer)
Subsequent open
First open
20 © 2015 IBM Corporation
Example: Kerberos authentication is a key new feature in Netezza 7.2. The Cognos and Netezza engineering teams collaborated to ensure that Netezza 7.2 and Cognos BI 10.2.2 provided a seamless Kerberos single-sign on experience before either 7.2 or 10.2.2 were released.
The Cognos and Netezza engineering teams can easily collaborate to resolve an IBM technical service request.– There is significantly more barriers to technical support when multiple
vendors are involved.
Collaboration between the IBM Labs
Cognos and Netezza engineers work together to ensure customer success
21 © 2015 IBM Corporation
Integrating Netezza Analytics into Cognos
Netezza Analytic Functions are available as Stored Procedures or UDFs
Create Mining results in Netezza tables and access them during report generation
Read-Only analytic Stored Procedures and UDX can be executed directly from Cognos reports
22 © 2015 IBM Corporation
Benefits of a SPSS Modeler and PureData System for Analytics
Visual, Easy to Use Interface – Faster time to solution and understanding– Expand to Line of Business users
Scalable and Optimized for PureData System for Analytics– Limited/no data movement – analysis executed within the DB
(SQL Pushback, UDFs, In-database Mining, In-database Scoring)– No programming - SQL is automatically generated– Analytics run10x-100x faster
Analytics Flexibility and Deployment– Executed on a purpose built appliance (powered by Netezza)– SPSS Algorithms and Netezza Analytics available– Works with SPSS greater SPSS portfolio
Fortune 100 telco company using SPSS Modeler and Netezza– Scoring 100M customers, 1 model + 10 predictors < 4 seconds!– Scoring 100M customers, 20 models + 20 predictors < 10 seconds!
Performance and Ease of Use
23 © 2015 IBM Corporation
Qualcomm responds to business needs more quickly with PureData System for Analytics
Time reduced to daysfrom months which was spent on development
600 times faster query performance
Solution components • IBM PureData System for Analytics (powered by
Netezza technology)• IBM Cognos® Business Intelligence
“By having an optimized and integrated system, we now can leverage all our data to look for new opportunities and focus our attention where we will see the most return.”
- Kim Konotchick, Senior IT Manager, Qualcomm
Time reduced to daysto market on new solutions
24 © 2015 IBM Corporation
FleetRisk Advisors help trucking operators prevent more accidents with stronger and faster risk prediction models
20% reductionin the incidence of minor accidents
80% reductionin serious accidents amount trucking company customers
30% increasein driver retention rates, with commensurate decreases in recruiting and training costs
Solution components • IBM® PureData™ System for Analytics (powered by Netezza® technology)• IBM SPSS® Collaboration and Deployment Services• IBM SPSS Modeler• IBM SPSS Modeler Desktop• IBM SPSS Modeler Server
“Our new solution has enabled us to push the boundaries of predictive risk analysis, which has translated into real value for our trucking operator customers that rely on it.”
—Patrick Ritto, chief technology officer
25 © 2015 IBM Corporation25
Cognos and Netezza – a blazing combination
5 reasons to use Cognos BI with Netezza
1. Interactive analysis – engaging self-service interfaces2. Enterprise scalability – supports thousands of users3. Compelling visualizations – on the web, mobile, or emailed4. Optimized queries – intelligently balances local and remote data
processing5. No wait time – instantaneous responses when in-memory cache
is leveraged
C O G N O S + Blazing Results=
Netezza Solutions
100+ Joint Customers
© 2015 IBM Corporation
© 2015 IBM Corporation
PureData System for Analytics N3001-001 vs
Microsoft SQL Server
27 © 2015 IBM Corporation
https://www.rocksolidsql.com/News/News.aspx?NewsCategoryKey=9785e502-23c2-4ea4-baf6-1b01927d14d1
Aging Install Base 80%+ 2008 or Older
28 © 2015 IBM Corporation
SQL Server PDA Mini (N3001-001)
…and hope it runs! …turn the key and Go!!
29 © 2015 IBM Corporation
Four Things to Know about SQL Server 2008 R2
1. OLTP optimized solution • Microsoft SQL Server 2008 R2 is an OLTP optimized databases. Its optimizer is not designed
or built to handle the complex queries inherent in analytic workloads.
2. No HA Built In• The SQL Server SMP based Fast Track Solution is a single server only. There are no HA
capabilities. Can you trust your mission critical warehouse to a system with no HA capability?
3. Scalability is Limited• Both the single server and the software limit the scalability of the solution. You cannot simply
add more resources and grow the system and expect more performance out of the system due to the inherent limitations of an SMP architecture for data warehousing.
4. Inefficient data compression• SQL Server uses some old, inefficient algorithms that rely on the data in the table being
always in sorted order to work at all. If the data is random, just like it is generated in an OLTP system, or batched into a DWH, then SQL Server will get very little, if any, compression.
30 © 2015 IBM Corporation
Mini Appliance test results
IBM PureData System for Analytics
Mini Appliance (N3001-001)MS SQL Server
3seconds
1Avnet beta test performed using customer workload on PureData System for Analytics N3001-001 compared to MS SQL Server 2008
384seconds
What could you do if your queries were 127x faster?
vs.
Avnet beta test using customer workload
31 © 2015 IBM Corporation
PDA Mini Appliance test results
1GrassRoots beta test performed using customer workload on PureData System for Analytics N3001-001 compared to MS SQL Server 2008
What could you do if your queries were 44x faster?
IBM PureData System for Analytics
Mini Appliance (N3001-001)MS SQL Server
10seconds
444seconds vs.
'We were blown away by the performance, we loaded 600 million records and the Netezza Mini Mako appliance performed 45x faster than our MSSQL 2008 instance. What was more impressive was how quick the Netezza Mini was when used with Tableau. Even with 600 million records, we
were able to use Tableau in an almost interactive fashion. No more waiting for minutes for the data to be retrieved and visualized"
Grass Roots beta test using customer workload
© 2015 IBM Corporation
Lets Look At How PureData System for Analytics stacks up to
The Latest Windows Columnar Competitive Database
© 2015 IBM Corporation
33 © 2015 IBM Corporation
Test Scenarios
Workload tested– Two sets of queries
• Sales report style queries (80%)• Data Scientist style queries (20%)
Two modes of execution tested– Serial Execution test (single user test)
• Used to isolate single query performance• Single connection iterates all queries in the workload
– Heavy Mixed Throughput test (30 user test)• Time to complete a set number of reports for each user
34 © 2015 IBM Corporation
The Systems Tested (Initially)
Current Columnar WindowsCompetitive database
PureData System for Analytics N3001-001
Fully HA appliance2x x3650M4 (20 cores) 128GB RAM24x 600GB HDD
System (hardware & software) Cost $170,000Maintenance for 3 years $51,000Total Cost over 3 years $221,000
Single server, no HA 20 cores Intel Ivy Bridge EX 512 GB RAM 8 x 900GB HDD
Server Cost $40,500Software Cost $137,500Maintenance for 3 years $111,000Total Cost over 3 years $289,000
30% more expensive, and much slower
for systems compared
PureData Analytics N3001-001 vs. 2014 Columnar Database Competitor (Intermediate and Complex Analytics workload) Based on IBM internal tests comparing PureData System For Analytics with a comparable competitor configuration (version available as of 03/15/2015) executing a materially identical analytics workload in a controlled laboratory environment. Test measured 30, 20, and 10 concurrent user report throughput to execute identical SQL query workloads on the same data. Competitor configuration includes competitor recommended software options and features. IBM configuration: PureData System for Analytics N3001-001. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Users of this document should verify the applicable data for their specific environment. Contact IBM and see what we can do for you. Pricing is based on publically available information and published list prices as of 3/15/2015. . Total cost over 3 years takes into account cost of hardware, software, and maintenance.
35 © 2015 IBM Corporation
Testing
PureData System for Analytics N3001-001
Single server, no HA 20 cores Intel Ivy Bridge EX 512 GB RAM 8 x 900GB HDD
Initially, one of the Data Scientist style queries never finished on the columnar database, but completed in seconds on PureData System for
Analytics
Fully HA appliance2x x3650M4 (20 cores) 128GB RAM24x 600GB HDD
PureData Analytics N3001-001 vs. 2014 Columnar Database Competitor (Intermediate and Complex Analytics workload) Based on IBM internal tests comparing PureData System For Analytics with a comparable competitor configuration (version available as of 03/15/2015) executing a materially identical analytics workload in a controlled laboratory environment. Test measured 30, 20, and 10 concurrent user report throughput to execute identical SQL query workloads on the same data. Competitor configuration includes competitor recommended software options and features. IBM configuration: PureData System for Analytics N3001-001. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Users of this document should verify the applicable data for their specific environment. Contact IBM and see what we can do for you. Pricing is based on publically available information and published list prices as of 3/15/2015. . Total cost over 3 years takes into account cost of hardware, software, and maintenance.
Current Columnar WindowsCompetitive database
36 © 2015 IBM Corporation
Hardware and Software Tuning vs. No TuningPureData System for Analytics
N3001-001
Fully HA appliance2x x3650M4 (20 cores) 128GB RAM24x 600GB HDD
System (hardware & software) Cost $170,000Maintenance for 3 years $51,000Total Cost over 3 years $221,000
Single server, no HA 20 cores Intel Ivy Bridge EX 512 GB RAM 8 x 900GB HDD 1.2TB High IOPS FlashServer Cost $58,000Software Cost $137,500Maintenance for 3 years $111,000Total Cost over 3 years $306,500
After adding flash storage – PLUS a week of expert tuning, the team was able to get the
“problem” query to run
No indexes, no aggregates, no tuning at all…
Faster with no tuning.
PureData Analytics N3001-001 vs. 2014 Columnar Database Competitor (Intermediate and Complex Analytics workload) Based on IBM internal tests comparing PureData System For Analytics with a comparable competitor configuration (version available as of 03/15/2015) executing a materially identical analytics workload in a controlled laboratory environment. Test measured 30, 20, and 10 concurrent user report throughput to execute identical SQL query workloads on the same data. Competitor configuration includes competitor recommended software options and features. IBM configuration: PureData System for Analytics N3001-001. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Users of this document should verify the applicable data for their specific environment. Contact IBM and see what we can do for you. Pricing is based on publically available information and published list prices as of 3/15/2015. . Total cost over 3 years takes into account cost of hardware, software, and maintenance.
Current Columnar WindowsCompetitive database
37 © 2015 IBM Corporation
No Tuning for IBMPureData System for Analytics
N3001-001
Single server, no HA 20 cores Intel Ivy Bridge EX 512 GB RAM 8 x 900GB HDD 1.2TB High IOPS Flash
Fully HA appliance2x x3650M4 (20 cores) 128GB RAM24x 600GB HDD
1TB 30 User Concurrent Execution Workload
159 qphqueries per Hour
181 qphqueries per Hour
14% Slower, 39% More expensivePLUS more tuning required
for systems compared
PureData Analytics N3001-001 vs. 2014 Columnar Database Competitor (Intermediate and Complex Analytics workload) Based on IBM internal tests comparing PureData System For Analytics with a comparable competitor configuration (version available as of 03/15/2015) executing a materially identical analytics workload in a controlled laboratory environment. Test measured 30, 20, and 10 concurrent user report throughput to execute identical SQL query workloads on the same data. Competitor configuration includes competitor recommended software options and features. IBM configuration: PureData System for Analytics N3001-001. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Users of this document should verify the applicable data for their specific environment. Contact IBM and see what we can do for you. Pricing is based on publically available information and published list prices as of 3/15/2015. . Total cost over 3 years takes into account cost of hardware, software, and maintenance.
Current Columnar WindowsCompetitive database
Over a week of expert tuning PLUS adding Flash on Competitor
38 © 2015 IBM Corporation
PDA Mini Beats The Windows Competitive Database
Pure Data System to Analytics is Simply Faster – for systems compared– Faster to get up and running– Faster and easier to get blazing performance– Faster performing
And More Importantly, Pure Data System to Analytics– Costs Less– Does not force sacrifices in performance
Not just faster and easier than the older, traditional Row Store version -- but also the new Columnar version as well
39 © 2015 IBM Corporation
PDA The Smart Choice Over SQL Server
Proven architecture for complex analytics on large data volumes
True appliance simplicity, quick time to value
In-database analytics functions to keep processing within the database
Simple: No indexes, No Tuning!
Includes entitlements for• Business Intelligence – Cognos
• Data Integration and Transformation – InfoSphere DataStage
• Hadoop Data Services – BigInsights
• Real Time Analysis – InfoSphere Streams …
40 © 2015 IBM Corporation
Q&A
© 2015 IBM Corporation
Tactical Institute Utilizes IBM Watson Analytics to Gain CrimeAnalysis – June 1 at 11 AM CST
Complimentary Workshop: PureData Analytics – June 22 in Dallas, TX
Register for these events & more: crescointl.com/events
Or RSVP email [email protected]
Upcoming events
© 2015 IBM Corporation
Thanks for joining!Cresco brings your data and business together via Analytics Expertise in software management, technical and management consulting, training and support.
Contact Us >> Chat with us on www.crescointl.com> Call 844.6.CRESCO> Email [email protected]
43 © 2015 IBM Corporation