db2 10.5 for zlinux.nedb2ug.ppt 10.5 for zlinux.nedb2ug.pdf · db2 10 for z/os and db2 10.1 linux,...
TRANSCRIPT
© 2014 IBM Corporation
Information Management for System z
DB2 for zLinuxpresented to the New England DB2 User’s Group
…a view from the lab… from a performance guy
… who works with lots of customers
Berni SchieferDistinguished EngineerInformation Management Performance & [email protected]
Agenda
The History
DB2 Today
DB2 Linux Unix Windows for System z
Best Practices
© 2014 IBM Corporation2
History of DB2 for zLinux
In 2000, DB2 7.1 was one of the first software programscertified for use on Linux for System Z
The IBM DB2 and Linux for System Z development teams work together on the DB2 on Linux for System Z stack
Quality, reliability, and performance proven by hundreds of satisfied DB2 on Linux for System Z customers
In 2013 IBM delivered DB2 10.5 for Linux on System Z In Q1 2014 DB2 10.5 FP3 was delivered
4
running DB2More than
2000 2003 2006 2009 2012 2015
© 2014 IBM Corporation
Choose from 2 Linux for System Z Distributions
IBM is committed to open standards Linux DB2 is optimized to run on SuSE and Red Hat Linux distributions
Favorite levels SLES 11 SP3 RHEL 6.5
5 © 2014 IBM Corporation
In 2014 we expect both SLES12 and RHEL7
DB2 LUW : One Database Platform for All Business Applications
Massive scale / 24x7 OLTP Platform
Active / Active 24x7Massive Data Scale Warehouse
Customer Infrastructure Optimised Deployment
NoSQL Spatial Analytics HADOOP Extensibility
In Memory BLU Acceleration
Temporal Data
Workload Management
Intelligent Compression
Advanced Application Functionality
Advanced Data Functionality
Autonomic Management
Data S
tudio
Continual Data Ingest/Access
Fine grained Security
Oracle Compatibility
Multi-tenant Enterprise Platform
MongoDBAPI
JSON/Graph Store Pure XML
Industry Standard API’s
Administration
Development
Performance Management
Data Warehouse Management
Configuration Management
Advanced RecoveryTools
HostConnectivity
Deployment Options
Enterprise Data Platform for all Business Applications
Available on zLinux
“Before we made a final decision we benchmarked some of the key database management systems. That includes Oracle, SQL Server and DB2. We ended up choosing DB2 for several reasons. One was reliability, second was performance and perhaps the most important factor was ease of use” – Bashir Khan, Director of Data Management and Business Intelligence
DB2 for LUW : Three Overarching Tenants
ResiliencyKeep data available and secure despite component failures, planned events, disasters, intrusions.
Cost EffectivenessAchieve leading performance with fewer resources.Do more with less.
Simplicity Simple to operateRich set of programming interfaces and tools to support agile development.
Example Technologies & Proof Points
Comprehensive Disaster Recovery with DB2 HADR
Online Operations, Reorg Avoidance and other
Comprehensive Data Security
Developer-friendly and ISV-friendly APIs including noSQL & JSON
Rich application semantics including time travel query
Automatic Configuration and Tuning
Automatic Maintenance
Exceptional scalability across scale-up and scale-out architectures
Industry Leading Data Compression
Comprehensive Flash/SSD Exploitation
Integrated Workload Management
ResiliencyKeep data available and secure despite component failures, planned events, disasters, intrusions.
Cost EffectivenessAchieve leading performance with fewer resources.Do more with less.
Simplicity Simple to operateRich set of programming interfaces and tools to support agile development.
DB2 10.5DB2 10
3x Query Performance Boost50% Compression BoostTemporal QuerynoSQL Graph StoreHADR Multiple Standby
DB2 9.7Advanced Compression
TCO & Performance 3x Query Performance New Index Exploitation Adaptive Compression Multi-temp Storage Real-time Warehousing
Ease of Development Temporal Query 98% SQL Compattibiltiy Graph Store RCAC
Reliability / Availability pureScale Integration &
Enhancements WLM Enhancements Reorg Avoidance HADR Mutliple Standby
TCO & Performance 360O Monitoring
In memorySQL based
Reorg Avoidance Ease of Development XML extended to
DPF, MDC, RP “Currently Commited”
isolation Numerous SQL
enhancements
Reliability /Availability HADR Advances
Active Standby Trap Resilience General Online Schema
Evolution
TCO & PerformanceMemory-optimized BLU Acceleration Workload Consolidation with pureScaleEven more performance !
Ease of Development Enhanced SQL CompatibilityMore noSQL Integration (MongoDB API)
Reliability / AvailabilityRolling UpdatesHADR with pureScalepureScale Active / active DR
EnhancementsOnline add/drop MemberOther availability enhancements
DB2 10.5Analytics at the Speed of ThoughtFuture Proof VersatilityEnhanced SQL & noSQL function
What ?– Allow indexes to be defined with an expression, eg.
CREATE INDEX i1 ON emp (UPPER(lastname), salary+bonus)
Value Proposition– Efficient execution of SQL statements with such expressions, eg.
SELECT * FROM emp WHERE UPPER(lastname) = ? SELECT * FROM emp WHERE salary+bonus = ?
– Avoid the drawbacks of work-around (index on generated column)• Space consumption of extra column• Potential need to modify applications to reference the new column
Index on Expression
What ?– Allow indexes to be defined so that NULL keys are excluded, eg:
CREATE [UNIQUE] INDEX i1(c1, c2) EXCLUDE NULL KEYS Value Proposition
– Support applications whose semantics require unique enforcement, but only where keys are not NULL
– Storage savings ! (Avoid indexing NULLs if they are infrequently queried) Notes
– A “NULL key” is one where all key components are NULL
Excluding NULL Keys from Indexes
C1 C2
1 NULL Unique Constraint
InsertC1 C2
NULL NULL
C1 C2
1 NULL
NULL NULL
C1 C2 Regular Index ENK Index
1 NULLFail FailNULL NULL
C1 C2 Regular Index ENK Index
1 NULLFail SuccessNULL NULL
Excluded from ENK index
What ?– Allow tables to be defined with a row size which exceeds page size, eg:
CREATE TABLESPACE TS1 PAGESIZE 4KCREATE TABLE T(c1 VARCHAR(4000), VARCHAR(4000)) IN TS1
Value Proposition– Support applications that require long row definitions– Avoid a lengthy table redefinition to change pagesize
Notes– New maximum row length: 1,048,319 bytes– Excess row data stored in a LOB– Performance penalty when need to go ‘off page’ for a portion of row
• Usually OK – in most scenarios instances of long rows are rare– If you expect long rows to be common, try to use a larger page size
Extended Row Size
Online inplace reorg support on a table using adaptive compression
Fastpath option for online inplace reorg to clean up overflow records only
Reorg with RECLAIM EXTENTS can cleanup partially empty extents for Insert Time Clustered tables
$ db reorg table T1 inplace cleanup overflows
REORG Enhancements
DB2 Supports Different SQL Dialects
Steady increase in compatibility over time– More and more complex applications
DB2 10.5 estimated to provide >98% statement compatibilityData is based on DCW (Database Conversion Workbench) DB2 reports
Aggregated SQL Compatibility by Release
Simple format for data exchange
– Self-describing, schema-less– Very simple (eg. tag:value format)– Human readable – Based on JavaScript, initially targeted for
web applications,… but,…
Use is rapidly spreading
– The data interchange format for the Web• JavaScript is very popular in mobile and
systems-of-engagement applications– Analytics
• Eg. An organization stores a large quantity of web statistics stored as JSON documents, and wants to perform analytics
{"firstName": "John","lastName" : "Smith","age" : 25,"address" :{
"streetAddress": "21 2nd Street","city" : "New York","state" : "NY","postalCode" : "10021"
},"phoneNumber":[
{"type" : "home","number": "212 555-1234"
},{
"type" : "fax","number": "646 555-4567"
}]
}
Background : What is JSON ?
Typical JSON Open Source Datastore Attributes
Optimized for high speed data ingest Relaxed/absent ACID properties
– Logging is often turned off or done asynchronously to improve performance– “Fire and forget” inserts
• Applications include checking logic to verify update occurred– No concept of commit or rollback; each JSON update is independent
• Applications implement compensation logic to update multiple documents with ACID properties
Data is sharded for scalability– Shards are replicated asynchronously for availability– Queries to replica nodes can return back-level data sometimes
JSON documents are stored in “collections”– No “join” across collections, requires “in application” joins
Limited options for security, temporal, geo-spatial,...
The Best of Both Worlds -Agility with a Trusted Foundation
• What : Built-in JSON support in DB2 !•Including support for popular noSQL JSON APIs
• Why : Preserve mature DBMS features;Leverage existing skills and tools•Multi-statement Transactions•ACID•Extreme scale, performance and high availability•Comprehensive Security•Management/Operations•…
DB2 JSON Datastore : Concept & Motivation
Binary formatted JSON stored in the database (in LOBs)
Btree indexing to JSON elements for fast query processing
Java API and command line Optional “Fire-forget” inserts Supports transactions Smart query re-write DB2 ecosystem of tools Extend support to more
applications and developers
JSON API JSON Command Shell
JDBC Driver
SQL via DRDA
Java Apps
Key Shopping Cart
1 <binary JSON>
2 <binary JSON>
3 <binary JSON>item
“item”:”camera”
ApplicationsJava PHP NodeJS
AIM Developed MongoDB Wire ProtocolNoSQL JSON Wire Listener
BSON Wire Protocol
DB2inlined
LOBs
DB2 JSON Datastore : Features
www.ibm.com/developerworks.com and search for “DB2 JSON”
DB2 is an ideal data server on the new System zEC12. Customers know there are compelling availability and server virtualization/consolidation opportunities on System.With the IBM System zEC12, System z becomes an even more attractive platform for deploying DB2..
Data Center Virtualization with System z, z/VM & Linux
DB2 10
Business Intelligencewith System z & Linux
Cognos & DB2 10
High Availability with System z, z/VM & Linux
DB2 10 with HADR
Modernizationwith System z & Linux
DB2 10
IBM System zEC12 with Linux and DB2Yet Another High-performance DB2 Platform
© 2014 IBM Corporation19
zEC12 Continues the Mainframe Heritage
0
1000
2000
3000
4000
5000
1997G4
1998G5
1999G6
2000z900
2003z990
2005z9 EC
2008z10 EC
2010z196
300MHz
420 MHz
550 MHz
770 MHz
1.2 GHz
1.7 GHz
4.4 GHz
5.2 GHz
MH
z
2012zEC12
5.5 GHz
6000
© 2014 IBM Corporation20
DB2 Database BI Workload on z10/z196/zEC12 DB2 POPS workload
– Ten selected complex database warehouse queries• Measured high hit / warm state
Configuration– 16 processors (IFLs)– 128 GB of main memory– DB2 10.1 for Linux on System z
– Exploiting intra-query parallelism– Not I/O constrained
© 2014 IBM Corporation22
DB2 Performance for Complex BI Queries 30 percent more throughput comparing zEC12 versus z196 Close to 2x comparing zEC12 versus z10
Testcase0
5
10
15
20
25
DB2 BI Queries
Overall run time (lower is better)
z10z196zEC12
Tim
e i n
sec
© 2014 IBM Corporation24
So What Is DB2 for zLinux?(and what is it not) DB2 LUW for zLinux is
– “JUST” DB2 LUW– 99.9%+ the same code– 99.9%+ the same look/feel– 99.9%+ the same skills
DB2 LUW for zLinux has 98%+ Oracle Compatibility– PL/SQL– Datatypes
DB2 LUW is not identical to DB2 for zOS, but– Common client infrastructure
• Jcc/odbc/cli• Perl/php/python/ruby/…
– Optim Data Studio tools (Data Studio, OPM, OPWT)– Exceptionally similar DDL/DML/DCL
© 2014 IBM Corporation25
DB2 for z/OS vs DB2 for LUW (cont.)(not exhaustive, features may be missing, doesn’t over DB2 for zOS v11)
© 2014 IBM Corporation26
DB2 10 for z/OS and DB2 10.1 Linux, Unix & Windows
Multi-row INSERT, FETCH & multi-row cursor UPDATE, Dynamic Scrollable Cursors, GET DIAGNOSTICS, Enhanced UNICODE SQL, join across encoding schemes, IS NOT DISTINCT FROM, VARBINARY, FETCH CONTINUE, Enhanced MERGE, SELECT from MERGE, routine versioning, timestamps w/timezone
Inner and Outer Joins, Table Expressions, Subqueries, GROUP BY, Complex Correlation, Global Temporary Tables, CASE, 100+ Built-in Functions including SQL/XML, Limited Fetch, Insensitive Scroll Cursors, UNION Everywhere, MIN/MAX Single Index, Self Referencing Updates with Subqueries, Sort Avoidance for ORDER BY, and Row Expressions, 2M Statement Length, GROUP BY Expression, Sequences, Scalar Fullselect, Materialized Query Tables, Common Table Expressions, Recursive SQL, CURRENT PACKAGE PATH, VOLATILE Tables, Star Join Sparse Index, Qualified Column names, Multiple DISTINCT clauses, ON COMMIT DROP, Transparent ROWID Column, Call from trigger, statement isolation, FOR READ ONLY KEEP UPDATE LOCKS, SET CURRENT SCHEMA, Client special registers, long SQL object names, SELECT from INSERT, UPDATE or DELETE, INSTEAD OF TRIGGER, Native SQL Procedure Language, BIGINT, file reference variables, XML, FETCH FIRST & ORDER BY in subselect & fullselect, caseless comparisons, INTERSECT, EXCEPT, not logged tables, OmniFind, spatial, range partitions, data compression, session variables, DECIMAL FLOAT, optimistic locking, ROLE, TRUNCATE, index & XML compression, created temps, inline LOB, administrative privileges, implicit cast, date/time changes, currently committed, moving sum & average, index include columns, row and column access control, time travel queryUpdateable UNION in Views, GROUPING SETS, ROLLUP, CUBE, more Built-in Functions, SET CURRENT ISOLATION, multi-site join, MERGE, MDC, XQuery, XML enhancements, additional data type (array, row, cursor), global variables, even more vendor syntax, temp table compression, MODULEs
z
luw
common
So What Is the Same? ( )
Editions– Advanced Enterprise Server Edition (AESE)– Enterprise Server Edition
PVU– 100 PVU/core (IFL) for older generations– 120 PVU/core for new systems
Core Features– (Adaptive )Compression – HADR
• Including multiple standby and reads on secondary– WLM (workload managemet)– Database Partitioning Feature
• MPP data warehousing– Federation / Replication– LBAC/FGAC– LDAP /– Flashcopy Manager– ….
© 2014 IBM Corporation27
So What Is Different / Missing ( )
Editions– Advanced Workgroup Server Edition– Workgroup Server Edition– Express/Express-C Edition
PVU– PVU-less licenses
Features Unavailable– DB2 pureScale (needs GPFS, Mellanox IB/RoCE /w uDAPL from STG)
• HADR or DB2 data sharing for zOS are alternatives– DB2 BLU Acceleration– Encryption (3rd party tool (Vormetric) not available)….– Text search– Hypervisor Edition
• Can build your own image using the IWD toolkit– Kerberos plug-in– Various GUI tools that are normally client-side
• Replication Center , Data Studio etc.
© 2014 IBM Corporation28
So What Is Unique?
Not so much in the core DB2 code– First of all, DB2 is more than 99% the same code across all platforms
• x64 Linux, pLinux, zLinux, AIX, Solaris, HP-UX, Windows– Linux for System z is a complete Linux implementation
• Pro; Reusable skills, common function• Con: Difficult to Differentiate
Core platform values provide value transparently– Inherent reliability creates unique opportunities
• E..g “virtual HA” (2 LPARs with HADR to protect against software failures– Optimized virtualization permits high consolidation ratio– Enormous scale-up potential reduces need for scale-out
Exception:– Code to have an up but idle DB2 “go to sleep” to help z/VM
conserve/consolidate resources– db2set DB2_MIN_IDLE_RESOURCES– Designed for zLinux, available on all Linux
© 2014 IBM Corporation29
High Value Scenarios for DB2 for zLinux
Consolidation– Bring many small, typically lightly loaded databases together into a highly
consolidated and virtualized environment under z/VM
Migrate unhappy Oracle on zLinux customers – Oracle may be pushing them to Exadata– Oracle may be charging outrageous prices– Migration is typically relatively easy/trivial
• Especially if using a packaged app that already supports DB2
Use DB2 as the persistent data store inside another SWG solution– Websphere Portal Server– Cognos 10 content server– … many more
A regular database for OLTP or query processing
© 2014 IBM Corporation30
Best Practices (System config)
Use a modern System z machine– The System z processor has caught up a lot on a per core (IFL) basis
If you want to use z/VM (most do) remember– z/VM is not “free” (resource-wise) – especially wrt. memory
• The more IFLs the more memory you need to reserve for z/VM– Ensure you have current maintenance for z/VM
• Every test we do “finds” interesting things.– Give DB2 dedicated processors where possible
If your database does disk I/O (most do) remember– These days you care less about storage space
• You care about storage performance (random IOPS, sequential MB/sec)– Avoid ECKD disk (it is not a good fit for DB2), use SCSI disks
• With SCSI you can disable file system caching and keep memory for DB2– Follow standard zLinux best practices
© 2014 IBM Corporation31
Best Practices (Linux)
Ensure you are entitled to Linux support– You can choose to get support from IBM or
from a Linux distribution company • Redhat• SUSE
Use a current Linux version– SLES 11 (ideally SP3) or RHEL6 (ideally RHEL6.5)– Consult with the zLinux performance team on specifics
Apply standard Linux tuning best practices for DB2– Virtual memory tuning
• vm.Swappiness: 0• vm.Dirty_ratio: 10• vm.Dirty_background_ratio: 5
– General OS tuning• http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/topic/com.ibm.db2.luw.qb.server.doc/
doc/c0057140.html
© 2014 IBM Corporation32
Best Practices (DB2 software/sizing)
Use current DB2 LUW software– I know the tendency on zOS is to use n-1 VERSIONS– Linux is different
DB2 10.5 has been well-received (across platforms)– Most customers have deployed DB2 10.1 FP3 – DB2 10.5 FP3 is GA
• has cumulative maintenance from 9.7 and 10.1 included
Don’t overthink the sizing. First order approximations– DB2 for zOS = DB2 LUW– DB2 LUW = Oracle– DB2 LUW on Power7 = DB2 LUW on zLinux on a z196
KISS principle– Resist the urge to over-specify all possible options
• Just because they exist doesn’t mean you have to use them
© 2014 IBM Corporation33
Best Practices (DB2 configuration)
Use DB2 automatic storage– A modest number of storage paths that map sensibly to the SAN storage
Minimize the use of distinct page sizes– 8K is a good default (no more than 2)– Specify the smaller one at create database time for the catalog
Minimize the number of bufferpools– 1 for data/index, 1 for temp is usually sufficient
Go ahead and use plenty of tablespaces– Often helpful to split data/index/LOB (esp LOB) into separate tablespaces– No need to put EACH table in its own tablespace (common with DB2 for zOS)
Use the Autonomics – Autoconfigure & STMM
• But give DB2 a budget via instance_memory or database_memory
Compression– Turn on adaptive compression at create table time for top ~50 large tables
• Also index compression.© 2014 IBM Corporation34
Best Practices (Tools)
There is a strong tools portfolio – Many are free– Even more are included in Advanced Enterprise Server Edition (AESE)
Optim Data Studio– Database Administration– Developer
Optim Performance Manager (OPM)– With Extended Insight for application server environments (e.g. WAS)
Optim Query Workload Tuner (OQWT)– Integration with OPM– Next generation of “DB2 design advisor”
Optim Capture/Replay– Realistic Pre-production
© 2014 IBM Corporation35
Users
Application Servers
Network
InfoSphere Optim Solutions for Managing PerformanceIdentify, diagnose, solve and prevent performance problems
• Drill-down into problem detail and related context• Analyze captured data
2. Diagnose
• Alert of potential problems• Visual quick scan of complex environment
1. Identify
4. Prevent
• Receive expert advice for problem resolution• Correct the problem (SQL, database)
3. Solve
DBMS & OS
• Monitor and analyze historical data trends for planning• Auto-manage workloads
• Optim Query Workload Tuner •Optim Configuration Manager
• Optim Performance Manager • OMEGAMON XE DB2 Performance Expert •Optim Configuration Manager
• Optim Performance Manager Extended Edition
• Optim Query Capture Replay • All the other tools
36 © 2014 IBM Corporation
Customers using zLinux with DB2 LUW:
Nationwide Insurance
Peter Hahn (Germany)
Russian Hydrometeorological Research Institute – World Data Center (Russia)
Wessels+Müller AG (Germany)
White Cube – z114
Total System Services (TSYS)
Boston University
Calculo S.A. (Spain)
University of AZ College of Pharmacy –new and cool…….
© 2014 IBM Corporation37
Summary
DB2 LUW is a mature product on zLinux– Many years of development/testing/deployment
DB2 LUW on zLinux is full function – Nearly every capability is available on zLinux
– Yes, I know there are 2 visible exceptions
DB2 LUW on zLinux is high performance– Regular testing on latest System z hardware (OLTP and BI)
DB2 10.5 is production ready and available on zLinux– DB2 10.5 FP3 is available and production ready
100’s of satisfied customers– continued strong interest
© 2014 IBM Corporation38