2015 01 26_5212_2__transparent_archiving_with

© 2015 IBM Corporation

zAnalytics DB2 Update Day 2015 – March 23-27, 2015

1

Archiving DB2 Data

Christian MichelDB2 for z/OS utilities [email protected]



2

© Copyright IBM Corporation 2015. All rights reserved.

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

Disclaimer



3

Introduction to problem space

Archive Transparency Support in DB2 11 for z/OS

Temporal Support in DB2 for z/OS revisited

–Comparison to Archive Transparency

–Enhancements in DB2 11

–Auditing Support

IBM DB2 Analytics Accelerator –

High Performance Storage Saver (HPSS)

Combining HPSS with DB2 Archive Transparency

Summary

Agenda



4

Organizations are Challenged with Data Growth

Poor Application Performance

Business users & customers wait for application response; DBA’s spend majority of time

fixing performance issues

Buying more storage is not a “cheap” fix when you add

the operational burden

Manage Risk & Compliance

Increasing Costs

The “keep everything” strategy can impact disaster recovery and data retention

& disposal compliance

Storage Capex20%

Storage Operational Costs80%



5

In a database system, querying and managing

tables that contain a large amount of data is

a common problem

Performance of maintaining large tables is

a key customer pain point

One known solution to address this is to archive

the inactive/cold data to a different environment

Comes with challenges on the ease of use and performance:

• How to provide easy access to both current and archived data

within a single query

• How to make data archiving and access “transparent” with minimum

application changes

Address Operational Challenges with Archiving

Poor Application Performance



6

DATE CUSTNO BILLNO AMOUNT

2015/01 1 1472 11,11 €

2015/01 2 1473 23,45 €

2015/01 3 1474 99,33 €

2015/02 1 1475 2,73 €

2015/02 2 1476 29,99 €

2015/02 3 1477 44,08 €

TableBILLSARC

Two tables approach for current and archive data

DB2 supports to „transparently“ archive records through DELETE statement

DB2 supports to „transparently“ retrieve records without application change

DB2 11 for z/OS Archive Transparency


2014/10 1 1463 12,43 €

2014/10 2 1464 66,35 €

2014/10 3 1465 31,87 €

2014/11 1 1466 11,74 €

2014/11 2 1467 45,67 €

2014/11 3 1468 11,34 €

2014/12 1 1469 1,24 €

2014/12 2 1470 93,57 €

2014/12 3 1471 22,99 €

TableBILLS

DATE CUSTNO BILLNO AMOUNT----- ------- ------- -------2014/10 1 1463 12,43 €2014/10 2 1464 66,35 €2014/10 3 1465 31,87 €2014/11 1 1466 11,74 €2014/11 2 1467 45,67 €2014/11 3 1468 11,34 €2014/12 1 1469 1,24 €2014/12 2 1470 93,57 €2014/12 3 1471 22,99 €2015/01 1 1472 11,11 €2015/01 2 1473 23,45 €2015/01 3 1474 99,33 €2015/02 1 1475 2,73 €2015/02 2 1476 29,99 €2015/02 3 1477 44,08 €

SELECT * FROM BILLS ORDER BY DATE,CUSTNO;



7

TableBILLSARC

Archived data is stored in the „archive table“

Active data is stored in the „archive enabled table“

DB2 Archive Transparency – Naming Conventions


2015/01 1 1472 11,11 €

2015/01 2 1473 23,45 €

2015/01 3 1474 99,33 €

2015/02 1 1475 2,73 €

2015/02 2 1476 29,99 €

2015/02 3 1477 44,08 €


2014/10 1 1463 12,43 €

2014/10 2 1464 66,35 €

2014/10 3 1465 31,87 €

2014/11 1 1466 11,74 €

2014/11 2 1467 45,67 €

2014/11 3 1468 11,34 €

2014/12 1 1469 1,24 €

2014/12 2 1470 93,57 €

2014/12 3 1471 22,99 €

TableBILLS

Archive Table

Archive-Enabled Table



8

SET CURRENT SCHEMA='TRANSARC';CREATE TABLE CALLS (

CALLER VARCHAR(20) NOT NULL,DESTNBR VARCHAR(20) NOT NULL,CALLBEGIN TIMESTAMP NOT NULL,CALLEND TIMESTAMP NOT NULL);

CREATE TABLE CALLS_ARCHIVE LIKE CALLS;

ALTER TABLE CALLS ENABLE ARCHIVE USE CALLS_ARCHIVE;

Create archive-enabled and archive tables with matching column definitions

– Other table attributes can be completely different, most likely the archive table

will use a different partitioning key

– Connect the two tables with ALTER TABLE...ENABLE ARCHIVE USE...

DB2 Archive Transparency – Getting Started

CALLER DESTNBR CALLBEGIN CALLEND

… … … …

CALLS:


… … … …

CALLS_ARCHIVE:



9

Archive relationship is reflected in SYSIBM.SYSTABLES:

Archive-enabled table:

TYPE = 'T', ARCHIVING_TABLE refers to the archive table

Archive table:

TYPE = 'R', ARCHIVING_TABLE refers to the archive-enabled table

LOAD REPLACE will be disallowed for archive-enabled table:

DB2 Archive Transparency – Relationship Established

DBNAME TSNAME CREATOR NAME TYPE ARCHIVING_SCHEMA ARCHIVING_TABLE

DSN00021 CALLS TRANSARC CALLS T TRANSARC CALLS_ARCHIVE

DSN00023 CALLSRAR TRANSARC CALLS_ARCHIVE R TRANSARC CALLS

DSNU050I 072 02:59:48.98 DSNUGUTC - LOAD DATA INDDN LOADTMPL LOG NO REPLACE EBCDIC CCSID(37, 0, 0) DSNU650I -DB2A 072 02:59:48.98 DSNURWI - INTO TABLE TRANSARC.CALLS WHEN(4:5)=X'0003' DSNU076I -DB2A 072 02:59:48.98 DSNURWI - KEYWORD=REPLACE INVALID FOR ARCHIVE ENABLED TABLE SPACE=DSN00021.CALLSDSNU650I -DB2A 072 02:59:48.99 DSNURWI - ("CALLER" POSITION(7) VARCHAR, DSNU650I -DB2A 072 02:59:48.99 DSNURWI - "DESTNBR" POSITION(*) VARCHAR, DSNU650I -DB2A 072 02:59:48.99 DSNURWI - "CALLBEGIN" POSITION(*) TIMESTAMP EXTERNAL(26), DSNU650I -DB2A 072 02:59:48.99 DSNURWI - "CALLEND" POSITION(*) TIMESTAMP EXTERNAL(26)) DSNU012I 072 02:59:48.99 DSNUGBAC - UTILITY EXECUTION TERMINATED, HIGHEST RETURN CODE=8



10

Let‘s add some more tables and data to our sample

DB2 Archive Transparency – Sample Data

CREATE TABLE CUSTOMERS (CUSTNO INTEGER NOT NULL,NAME VARCHAR(200) NOT NULL);

CREATE TABLE CONTRACTS (PHONENBR VARCHAR(20) NOT NULL,CUSTNO INTEGER NOT NULL,ACTIVE CHAR(1) NOT NULL);

CREATE TABLE CONTRACTS_ARCHIVE LIKE CONTRACTS;ALTER TABLE CONTRACTS ENABLE ARCHIVE USE CONTRACTS_ARCHIVE;

INSERT INTO CUSTOMERS VALUES(1, 'Christian');INSERT INTO CUSTOMERS VALUES(2, 'Peter');INSERT INTO CUSTOMERS VALUES(3, 'James');

INSERT INTO CONTRACTS VALUES('123',1,'Y');INSERT INTO CONTRACTS VALUES('888',2,'Y');INSERT INTO CONTRACTS VALUES('707',3,'Y');INSERT INTO CONTRACTS VALUES('456',2,'Y');

INSERT INTO CALLS VALUES('123','888','2015-02-28 12:12:34','2015-02-28 12:15:22');INSERT INTO CALLS VALUES('888','707','2015-02-28 12:18:11','2015-02-28 12:20:08');INSERT INTO CALLS VALUES('888','123','2015-03-01 09:55:04','2015-03-01 10:02:44');INSERT INTO CALLS VALUES('123','888','2015-03-01 10:05:48','2015-03-01 10:15:07');



11

DB2 Archive Transparency – Sample Data

CUSTNO NAME

1 Christian

2 Peter

3 James

PHONENBR CUSTNO ACTIVE

123 1 Y

888 2 Y

707 3 Y

456 2 Y


123 888 2015-02-28 12:12:34 2015-02-28 12:15:22

888 707 2015-02-28 12:18:11 2015-02-28 12:20:08

888 123 2015-03-01 09:55:04 2015-03-01 10:02:44

123 888 2015-03-01 10:05:48 2015-03-01 10:15:07

PHONENBR CUSTNO ACTIVE CALLER DESTNBR CALLBEGIN CALLEND

CALLS:

CALLS_ARCHIVE:

CONTRACTS:

CONTRACTS_ARCHIVE:

CUSTOMERS:



12

2 methods are available to move data from the active table into the archive table

– DELETE from archive-enabled table

– REORG DISCARD on archive-enabled table with LOAD into archive table

DELETE behavior controlled by built-in global variable

SYSIBMADM.MOVE_TO_ARCHIVE, which has three modes

Archiving mode, all rows deleted will be moved, no insert/update allowed:

SET SYSIBMADM.MOVE_TO_ARCHIVE='Y'; -- no insert/update allowed

Enabled mode, all rows deleted will be moved, insert/update allowed:

SET SYSIBMADM.MOVE_TO_ARCHIVE='E'; -- insert/update allowed

Disabled mode, all rows deleted will be gone:

SET SYSIBMADM.MOVE_TO_ARCHIVE='N'; -- 'N' is the default

Access to global variable requires authorization

Scope of the global variable is session

DB2 Archive Transparency – How Data Gets Moved



13

First a quick test to update a row in archive mode:

Now move a row to the archive after updating it:

DB2 Archive Transparency – Move Data with DELETE

SET SYSIBMADM.MOVE_TO_ARCHIVE='E';UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='456';DELETE FROM CONTRACTS WHERE ACTIVE='N';


123 1 Y

888 2 Y

707 3 Y

CONTRACTS:


CONTRACTS_ARCHIVE:

456 2 Y456 2 N

SET SYSIBMADM.MOVE_TO_ARCHIVE='Y';UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='456';

AN ARCHIVE-ENABLED TABLE IS NOT ALLOWED IN THE SPECIFIED CONTEXT.REASON CODE 2. SQLCODE=-20555, SQLSTATE=428I6, DRIVER=4.17.30



14

First a quick test to update a row in archive mode:

Now move a row to the archive after updating it:

DB2 Archive Transparency – Move Data with DELETE

SET SYSIBMADM.MOVE_TO_ARCHIVE='Y';UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='456';

AN ARCHIVE-ENABLED TABLE IS NOT ALLOWED IN THE SPECIFIED CONTEXT.REASON CODE 2. SQLCODE=-20555, SQLSTATE=428I6, DRIVER=4.17.30

SET SYSIBMADM.MOVE_TO_ARCHIVE='E';UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='456';DELETE FROM CONTRACTS WHERE ACTIVE='N';


123 1 Y

888 2 Y

707 3 Y

CONTRACTS:


CONTRACTS_ARCHIVE:

456 2 N



15

Discard rows from archive-enabled table:

Then load discarded rows into archive table:

DB2 Archive Transparency – Move Data with Utilities

TEMPLATE DISCTMPL DSN TRANSARC.CALLS.ARCHIVE UNIT(SYSDA) DISP(NEW,CATLG,CATLG) TEMPLATE PNCHTMPL DSN TRANSARC.CALLS.PUNCH UNIT(SYSDA) DISP(NEW,CATLG,CATLG)REORG TABLESPACE DSN00021.CALLS

SHRLEVEL REFERENCE SORTDEVT SYSDA NOSYSRECCOPYDDN COPYTMPL DISCARDDN DISCTMPL PUNCHDDN PNCHTMPLDISCARD FROM TABLE TRANSARC.CALLS WHEN (CALLEND < '2015-03-01 00:00:00')

TEMPLATE ARCHTMPL DSN TRANSARC.CALLS.ARCHIVE DISP(SHR,KEEP,KEEP)LOAD DATA INDDN ARCHTMPL LOG NO RESUME YES EBCDIC CCSID(00037,00000,00000)INTO TABLE TRANSARC.CALLS_ARCHIVE WHEN(00004:00005) = X'0003'

( "CALLER" POSITION(7) VARCHAR, "DESTNBR" POSITION(*) VARCHAR, "CALLBEGIN" POSITION(*) TIMESTAMP EXTERNAL(026), "CALLEND" POSITION(*) TIMESTAMP EXTERNAL(026) )


888 123 2015-03-01 09:55:04 2015-03-01 10:02:44

123 888 2015-03-01 10:05:48 2015-03-01 10:15:07

CALLS:


123 888 2015-02-28 12:12:34 2015-02-28 12:15:22

888 707 2015-02-28 12:18:11 2015-02-28 12:20:08

CALLS_ARCHIVE:



16

Access to current data only or current data plus archive data is controlled by built-

in global variable SYSIBMADM.GET_ARCHIVE

Access to current data only:

SET SYSIBMADM.GET_ARCHIVE='N'; -- 'N' is the default

Access to current and archive only:

SET SYSIBMADM.GET_ARCHIVE='Y';

DB2 will transform any query to the archive-enabled table into a UNION ALLwith the archive table for transparent access

Access to global variable requires authorization

– Ability to restrict certain applications from querying archive data

Scope of the global variable is session

DB2 Archive Transparency – Query Data



17

Query active and archive data separately:

And now run a transparent archive query:

DB2 Archive Transparency – Query Data Sample

SET SYSIBMADM.GET_ARCHIVE='N';SELECT * FROM CALLS;

SELECT * FROM CALLS_ARCHIVE;

CALLER DESTNBR CALLBEGIN CALLEND------ ------- --------------------- ---------------------888 123 2015-03-01 09:55:04.0 2015-03-01 10:02:44.0123 888 2015-03-01 10:05:48.0 2015-03-01 10:15:07.0

CALLER DESTNBR CALLBEGIN CALLEND------ ------- --------------------- ---------------------123 888 2015-02-28 12:12:34.0 2015-02-28 12:15:22.0888 707 2015-02-28 12:18:11.0 2015-02-28 12:20:08.0

SET SYSIBMADM.GET_ARCHIVE='Y';SELECT * FROM CALLS;

CALLER DESTNBR CALLBEGIN CALLEND------ ------- --------------------- ---------------------123 888 2015-02-28 12:12:34.0 2015-02-28 12:15:22.0888 707 2015-02-28 12:18:11.0 2015-02-28 12:20:08.0888 123 2015-03-01 09:55:04.0 2015-03-01 10:02:44.0123 888 2015-03-01 10:05:48.0 2015-03-01 10:15:07.0


888 123 2015-03-01 09:55:04 2015-03-01 10:02:44

123 888 2015-03-01 10:05:48 2015-03-01 10:15:07

CALLS:


123 888 2015-02-28 12:12:34 2015-02-28 12:15:22

888 707 2015-02-28 12:18:11 2015-02-28 12:20:08

CALLS_ARCHIVE:



18

Transparent archive queries of course also work with multiple tables, or even a

mix of archive-enabled and regular tables:

And now run as a transparent archive query:

DB2 Archive Transparency – Complex Query Sample

SET SYSIBMADM.GET_ARCHIVE='N';SELECT CU.NAME,CA.DESTNBR,CA.CALLBEGIN FROM CUSTOMERS CU, CONTRACTS CO, CALLS CAWHERE CU.CUSTNO=CO.CUSTNO AND CO.PHONENBR=CA.CALLERORDER BY CU.NAME, CA.CALLBEGIN;

NAME DESTNBR CALLBEGIN--------- ------- ---------------------Christian 888 2015-03-01 10:05:48.0Peter 123 2015-03-01 09:55:04.0

SET SYSIBMADM.GET_ARCHIVE='Y';SELECT CU.NAME,CA.DESTNBR,CA.CALLBEGIN FROM CUSTOMERS CU, CONTRACTS CO, CALLS CAWHERE CU.CUSTNO=CO.CUSTNO AND CO.PHONENBR=CA.CALLERORDER BY CU.NAME, CA.CALLBEGIN;

NAME DESTNBR CALLBEGIN--------- ------- ---------------------Christian 888 2015-02-28 12:12:34.0Christian 888 2015-03-01 10:05:48.0Peter 707 2015-02-28 12:18:11.0Peter 123 2015-03-01 09:55:04.0



19

Views are also modified transparently:

And now run as a transparent archive query:

DB2 Archive Transparency – Views

SET SYSIBMADM.GET_ARCHIVE='N';CREATE VIEW CALLACTIVITY(CUSTNAME,DESTINATION,CALLTIME) ASSELECT CU.NAME,CA.DESTNBR,CA.CALLBEGIN FROM CUSTOMERS CU, CONTRACTS CO, CALLS CAWHERE CU.CUSTNO=CO.CUSTNO AND CO.PHONENBR=CA.CALLER;

SELECT * FROM CALLACTIVITYORDER BY CUSTNAME, CALLTIME;

CUSTNAME DESTINATION CALLTIME--------- ----------- ---------------------Christian 888 2015-03-01 10:05:48.0Peter 123 2015-03-01 09:55:04.0

SET SYSIBMADM.GET_ARCHIVE='Y';SELECT * FROM CALLACTIVITYORDER BY CUSTNAME, CALLTIME;

CUSTNAME DESTINATION CALLTIME--------- ----------- ---------------------Christian 888 2015-02-28 12:12:34.0Christian 888 2015-03-01 10:05:48.0Peter 707 2015-02-28 12:18:11.0Peter 123 2015-03-01 09:55:04.0



20

Dynamic SQL only applies UNION ALL if SYSIBMADM.GET_ARCHIVE='Y'

Runtime performance degradation of static DML with the built-in global variable

SYSIBMADM.GET_ARCHIVE is avoided with a design of two sections for static

DML with reference to archive enabled tables.

Given a static SQL SELECT * FROM CALLS; and if the package is bound with option

ARCHIVESENSITIVE(YES), then DB2 binds this SQL twice:

1. Original section:

Bind the original SELECT * FROM CALLS; for current data only. The majority of

applications request for current data only. There is no performance degradation

caused since no UNION ALL query transformation is done.

2. Extended section:

Bind again an extended section by expanding CALLS to UNION ALL of CALLSand its associated archive table CALLS_ARCHIVE.

No change required for the application.

At execution time DB2 chooses the section depending on the value of the built-in

global variable SYSIBMADM.GET_ARCHIVE:

– ‘N’ (the default): DB2 chooses the original section

– ‘Y’: DB2 chooses the extended section to get both current and archive data

DB2 Archive Transparency – Performance Considerations



21

While archiving is enabled, newly added columns in the archive-enabled table are

propagated to the archive table with the following attributes:

– Name,

– Data type

– Length (including inline LOB lengths), precision, scale

– FOR BIT, SBCS, or MIXED DATA attribute for a character string column

– Null attribute

– Hidden attribute

– Field procedure

DB2 Archive Transparency – DDL Propagation



22

DB2 Archive Transparency – Adding Columns

ALTER TABLE CALLS ADD COLUMN CALLZONE INTEGER;

INSERT INTO CALLS VALUES('123','888','2015-03-02 14:12:00','2015-03-02 14:17:17',5);


888 123 2015-03-01 09:55:04 2015-03-01 10:02:44

123 888 2015-03-01 10:05:48 2015-03-01 10:15:07

CALLS:


123 888 2015-02-28 12:12:34 2015-02-28 12:15:22

888 707 2015-02-28 12:18:11 2015-02-28 12:20:08

CALLS_ARCHIVE:

CALLZONE

NULL

NULL

CALLZONE

NULL

NULL

123 888 2015-03-02 14:12:00 2015-03-02 14:17:17 5



23

ALTER TABLE CONTRACTSADD COLUMN STATUS CHAR(1)ADD COLUMN MODIFY_TS TIMESTAMP NOT NULL GENERATED ALWAYS

FOR EACH ROW ON UPDATE AS ROW CHANGE TIMESTAMP;

UPDATE CONTRACTS SET STATUS='V' WHERE CUSTNO=2;UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='707';

SET SYSIBMADM.MOVE_TO_ARCHIVE='E';DELETE FROM CONTRACTS WHERE ACTIVE='N';

A row change time stamp can be used to record the time a row was moved to the

archive table:

DB2 Archive Transparency – Archive Timestamp


123 1 Y

888 2 Y

CONTRACTS:

CONTRACTS_ARCHIVE:


456 2 N

STATUS MODIFY_TS

NULL 2015-03-12 10:07:45.39952

NULL 2015-03-12 10:07:45.39952

STATUS MODIFY_TS

NULL 2015-03-12 10:07:45.39952

707 3 Y NULL 2015-03-12 10:07:45.39952

V 2015-03-12 10:14:20.17883

707 3 N NULL 2015-03-12 10:14:20.51477NULL 2015-03-12 10:17:52.01165



24

A row change time stamp can be used to record the time a row was moved to the

archive table:

DB2 Archive Transparency – Archive Timestamp

ALTER TABLE CONTRACTSADD COLUMN STATUS CHAR(1)ADD COLUMN MODIFY_TS TIMESTAMP NOT NULL GENERATED ALWAYS

FOR EACH ROW ON UPDATE AS ROW CHANGE TIMESTAMP;

UPDATE CONTRACTS SET STATUS='V' WHERE CUSTNO=2;UPDATE CONTRACTS SET ACTIVE='N' WHERE PHONENBR='707';

SET SYSIBMADM.MOVE_TO_ARCHIVE='E';DELETE FROM CONTRACTS WHERE ACTIVE='N';


123 1 Y

888 2 Y

CONTRACTS:

CONTRACTS_ARCHIVE:


456 2 N

STATUS MODIFY_TS

NULL 2015-03-12 10:07:45.39952

NULL 2015-03-12 10:07:45.39952

STATUS MODIFY_TS

NULL 2015-03-12 10:07:45.39952

V 2015-03-12 10:14:20.17883

707 3 N NULL 2015-03-12 10:17:52.01165



25

A transparent archiving relationship can be dropped with:

This will result in two completely independent tables:

– LOAD REPLACE possible again on the previously archive-enabled table

– DELETE will no longer move rows

– Tables can be manipulated or dropped individually

Archiving relationship can be re-established as before:

While the archiving relationship is established:

– You can not drop the archive table:

– Dropping the archive-enabled table also drops the archive table

– Columns can not be dropped

– LOAD REPLACE on the archive-enabled table disallowed

– LOAD REPLACE on the archive table allowed

DB2 Archive Transparency – Archiving Relationship

DROP TABLE CALLS_ARCHIVE; ALTER, DROP, OR REVOKE AFFECTING OBJECT TYPE TABLE CANNOT BE PROCESSED

BECAUSE OBJECT TRANSARC.CALLS OF TYPE TABLE IS DEPENDENT ON IT.SQLCODE=-478, SQLSTATE=42893, DRIVER=4.17.30

ALTER TABLE CALLS DISABLE ARCHIVE;

ALTER TABLE CALLS ENABLE ARCHIVE USE CALLS_ARCHIVE;



26

Benefit from different organization for active data vs. archive data:

– Active data partitioned by access key for access distribution

– Archive data partitioned by date (e.g. archive date in row change timestamp)

for sequential inserts and easy data maintenance over data life cyle

DB2 Archive Transparency – Usage Scenario


2015/01 101 3928 34,00 €

2015/01 117 3931 25,00 €

2015/02 101 5248 43,00 €

2015/02 185 5251 53,00 €


2015/01 202 3929 9,00 €

2015/01 252 3932 45,00 €

2015/02 202 5249 44,00 €

2015/02 246 5252 53,00 €


2015/01 303 3930 53,00 €

2015/01 318 3933 19,00 €

2015/02 303 5250 52,00 €

2015/02 364 5253 54,00 €


2014/10 101 1463 50,00 €

2014/10 103 1466 26,00 €

2014/10 202 1464 29,00 €

2014/10 245 1467 3,00 €

2014/10 301 1468 53,00 €

2014/10 303 1465 53,00 €

BILLSARC:


2014/11 101 1822 13,00 €

2014/11 143 1825 27,00 €

2014/11 202 1823 37,00 €

2014/11 221 1826 31,00 €

2014/11 303 1824 11,00 €

2014/11 330 1827 34,00 €


2014/12 101 2741 46,00 €

2014/12 176 2744 49,00 €

2014/12 202 2742 54,00 €

2014/12 276 2745 35,00 €

2014/12 303 2743 37,00 €

2014/12 309 2746 25,00 €

Part1

Part2

Part3

BILLS:Part1

Part2

Part3



27

DB2 11 for z/OS NFM

Packages bound with:

–APPLCOMPAT(V11R1) and

–ARCHIVESENSITIVE(YES)

JDBC packages may have to be rebound, e.g.:

DB2 Archive Transparency – What‘s Needed?

set CLASSPATH=C:\Program Files\IBM\SDP\dsdev\jar\db2jcc.jar;C:\Program Files\IBM\SDP\dsdev\jar\db2jcc_license_cisuz.jar

java com.ibm.db2.jcc.DB2Binder -url jdbc:db2://mydb2host.mydomain.com:12345/DB2LOC-user MYUSER -password MYPWD -action replace -bindOptions "ARCHIVESENSITIVE(YES)”



28

An integrated solution to maintain active and archive data in two separate tables

with different maintenance options

Each row exists only once, either in the active or in the archive table

DB2 will move and retrieve rows transparantely, no application change required

except for selection of archive/retrieval mode

Automatic propagation of new data columns

Relationship can be dropped/recreated for special maintenance activities

DB2 Archive Transparency – Summary



29

New concept of period (SYSTEM_TIME and BUSINESS_TIME periods)

– A period is represented by a pair of date/time columns in DB2 relations, one column

stores start time, the other one stores end time

– SYSTEM_TIME period captures DB2’s creation and deletion of records. DB2

SYSTEM_TIME versioning automatically keeps historical versions of records

– BUSINESS_TIME period allows users to create their own valid period for a given record.

Users maintain the valid times for a record.

Temporal tables: System-period Temporal Table (STT), Application-period

Temporal Table (ATT), bitemporal table (BTT)

New DML syntax allow query/update/delete data for periods of time

– Period specification with base table reference:

SELECT … FROM ATT/BTT FOR BUSINESS_TIME AS OF exp/FROM exp1 TO exp2/BETWEEN exp1 AND exp2 ...;

SELECT … FROM STT/BTT FOR SYSTEM_TIME AS OF exp/FROM exp1 TO exp2/BETWEEN exp1 AND exp2 ...;

– Period clause with base table reference:

UPDATE/DELETE FROM ATT/BTT FOR PORTION OF BUSINESS_TIME FROM exp1 TO exp2 ...;

DB2 10 Temporal Support - Revisited



30

Archive Transparency vs. System-Period Temporal Tables

Archive-enabled table

– Current data only

– One row per data point either in active or

archive table

– Suited to reduce operational burden for

large tables by splitting data

– DB2 takes care of moving data to archive

– Automatic query transformation when

needed

– Introduced with DB2 11

System-period temporal table

– Current and historic data

– Multiple rows per data point, all

previous versions in history table

– Suited for auditing/journaling of

changes

– „Time machine“ to query data as of a

certain timestamp

– DB2 keeps track of historic changes

– Automatic query transformation when

needed

– Introduced with DB2 10, additional

functionality provided with DB2 11



31

SET CURRENT SCHEMA='TEMPORAL';CREATE TABLE CONTRACTS (

PHONENBR VARCHAR(20) NOT NULL,CUSTNO INTEGER NOT NULL,ACTIVE CHAR(1) NOT NULL,ROW_START TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW BEGIN,ROW_END TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW END,TRANS_ID TIMESTAMP(12) GENERATED ALWAYS AS TRANSACTION START ID,PERIOD SYSTEM_TIME(ROW_START, ROW_END));

CREATE TABLE CONTRACTS_HISTORY LIKE CONTRACTS;ALTER TABLE CONTRACTS ADD VERSIONING USE HISTORY TABLE CONTRACTS_HISTORY;...

Create system-period temporal table and history table with matching column

definitions and connect them:

DB2 System-Period Temporal Table – Getting Started

CONTRACTS:

CONTRACTS_HISTORY:

PHONENBR CUSTNO ACTIVE ROW_START ROW_END TRANS_ID

… … … … … …


… … … … … …


123 1 Y 2015-03-16 05:32:50.198519135 9999-12-30 00:00:00.0 NULL

888 2 Y 2015-03-16 05:32:50.723358262 9999-12-30 00:00:00.0 NULL

707 3 Y 2015-03-16 05:32:51.058016836 9999-12-30 00:00:00.0 NULL


456 2 Y 2015-03-16 05:32:51.391996666 2015-03-16 05:33:51.264893214 NULL

456 2 N 2015-03-16 05:33:51.264893214 2015-03-16 06:53:28.853194106 NULL



32

Views have been used by some DB2 customers intensively to get more data

access control

Views will become more important for data consumability in HTAP (Hybrid

Transactional/Analytical Processing)

DB2 11 removes restrictions that period specification and period clause can only

be specified with base table references and appropriate type of temporal tables.

DB2 11 Temporal Support on Views

SET SYSIBMADM.GET_ARCHIVE='N';CREATE VIEW ACTIVECONTRACTS(CUSTNO,CUSTNAME,PHONENBR) ASSELECT CU.CUSTNO,CU.NAME,CO.PHONENBR FROM CUSTOMERS CU, CONTRACTS COWHERE CU.CUSTNO=CO.CUSTNO AND CO.ACTIVE='Y';

SELECT * FROM ACTIVECONTRACTS ORDER BY CUSTNO;

SELECT * FROM ACTIVECONTRACTS FOR SYSTEM_TIME AS OF '2015-03-16 05:33:00'ORDER BY CUSTNO;

CUSTNO CUSTNAME PHONENBR------ --------- --------

1 Christian 1232 Peter 8883 James 707

CUSTNO CUSTNAME PHONENBR------ --------- --------

1 Christian 1232 Peter 4562 Peter 8883 James 707



33

For temporal applications, the temporal special register support provides a way to

get time sensitive data without changing the applications -- the time machine

concept.

The majority of system-period temporal applications have requests for current

data only, there should be no performance degradation caused by UNION ALL to

history table.

DB2 will add an appropriate AS OF CURRENT TEMPORAL SYSTEM_TIME to the

executed statement.

DB2 11 Temporal Special Registers

SET CURRENT TEMPORAL SYSTEM_TIME=TIMESTAMP('2015-03-16 05:33:00');SET CURRENT TEMPORAL SYSTEM_TIME=CURRENT TIMESTAMP - 1 DAY;SET CURRENT TEMPORAL SYSTEM_TIME=NULL;

SELECT CURRENT TEMPORAL SYSTEM_TIME FROM SYSIBM.SYSDUMMY1;



34

Performance considerations for static DML. Packages bound with 2 sections:

1. Original section: Bind the original statement for current data only, no UNION ALL.

2. Extended section: Bind again an extended section with UNION ALL.

Requires packages bound with SYSTIMESENSITIVE(YES)

Scope of special register is session

Special register also available for BUSINESS_TIME, requires BIND with

BUSTIMESENSITIVE(YES)

DB2 11 Temporal Special Registers

SET CURRENT TEMPORAL SYSTEM_TIME=TIMESTAMP('2015-03-16 05:33:00');SELECT * FROM ACTIVECONTRACTS ORDER BY CUSTNO;



35

PM99683 for DB2 10 and DB2 11 provides an auditing feature for system

versioning to track

– WHEN the data is modified.

To meet regulatory compliance requirements, it is quite common and mandatory

to audit and track

– WHO modified the data, and

– WHAT action (SQL statement) caused the data modification.

Without built-in DB2 support, complex application logic (e.g., usage of complex

triggers) is usually needed which can result in big maintenance overhead and

poor performance.

New syntax is introduced to define non-deterministic generated expression

columns.

In addition, the ON DELETE ADD EXTRA ROW clause is provided and intended to

be used when the system-period temporal table contains a non-deterministic

generated expression column.

Temporal Auditing Support – PM99683



36

Audit change user and operation:

Temporal Auditing Support – Example

CREATE TABLE CONTRACTS (PHONENBR VARCHAR(20) NOT NULL,CUSTNO INTEGER NOT NULL,ACTIVE CHAR(1) NOT NULL,CHGUSER VARCHAR(128) GENERATED ALWAYS AS (SESSION_USER), CHGOP CHAR(1) GENERATED ALWAYS AS (DATA CHANGE OPERATION),ROW_START TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW BEGIN,ROW_END TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW END,TRANS_ID TIMESTAMP(12) GENERATED ALWAYS AS TRANSACTION START ID,PERIOD SYSTEM_TIME(ROW_START,ROW_END));

CREATE TABLE CONTRACTS_HISTORY LIKE CONTRACTS;ALTER TABLE CONTRACTS ADD VERSIONING USE HISTORY TABLE CONTRACTS_HISTORY

ON DELETE ADD EXTRA ROW;

PHONENBR CUSTNO ACTIVE CHGUSER CHGOP ROW_START ROW_END TRANS_ID

123 1 Y CHM I 2015-03-16 06:44:15.514209257 9999-12-30 00:00:00.0 NULL

888 2 Y CHM I 2015-03-16 06:44:15.934389265 9999-12-30 00:00:00.0 NULL

707 3 Y CHM I 2015-03-16 06:44:16.270720207 9999-12-30 00:00:00.0 NULL

PHONENBR CUSTNO ACTIVE CHGUSER CHGOP ROW_START ROW_END TRANS_ID

456 2 Y CHM I 2015-03-16 06:44:16.60572574 2015-03-16 06:51:04.548575473 NULL

456 2 N CHM U 2015-03-16 06:51:04.548575473 2015-03-16 06:53:28.853194106 NULL

456 2 N CHM D 2015-03-16 06:53:28.853194106 2015-03-16 06:53:28.853194106 NULL

CONTRACTS:

CONTRACTS_HISTORY:



37

Additional expressions are available for auditing purposes:

Temporal Auditing Support – Available Expressions

... CHGUSER VARCHAR(128) GENERATED ALWAYS AS (SESSION_USER), CHGOP CHAR(1) GENERATED ALWAYS AS (DATA CHANGE OPERATION),CHGACCT VARCHAR(255) GENERATED ALWAYS AS (CURRENT CLIENT_ACCTNG),CHGAPPL VARCHAR(255) GENERATED ALWAYS AS (CURRENT CLIENT_APPLNAME),CHGCORR VARCHAR(255) GENERATED ALWAYS AS (CURRENT CLIENT_CORR_TOKEN),CHGUID VARCHAR(255) GENERATED ALWAYS AS (CURRENT CLIENT_USERID),CHGWKST VARCHAR(255) GENERATED ALWAYS AS (CURRENT CLIENT_WRKSTNNAME),CHGSRVR CHAR(16) GENERATED ALWAYS AS (CURRENT SERVER),CHGSQLID VARCHAR(128) GENERATED ALWAYS AS (CURRENT SQLID),...

… CHGUSER CHGOP CHGACCT CHGAPPL CHGCORR CHGUID CHGWKST CHGSRVR CHGSQLID ...

... CHM I JCC041709.152.204.137 db2jcc_application::FFFF:9.152.204.137.6

3634.CEA78545A5EAchm 9.152.204.137 STLEC1 CHM ...

... CHM U JCC041709.152.204.137 db2jcc_application::FFFF:9.152.204.137.6

3987.CEA786CBBA0Dchm 9.152.204.137 STLEC1 CHM ...

... CHM D JCC041709.152.204.137 db2jcc_application::FFFF:9.152.204.137.6

4010.CEA7875553A1chm 9.152.204.137 STLEC1 CHM ...



38

Tables with system time period:

– Version of every update on a single row will be kept

– Partitioning approach:

• Active table partitioned by key, e.g. accounted

• History table partitioned by system_time_end timestamp for sequential insert

Archive-enabled tables (archive transparency):

– Row exists either in active or archive table

– Partitioning approach:

• Add modify_ts column as archive timestamp

• Archive-enabled table partitioned by key, e.g. accounted

• Archive table partitioned by modify_ts timestamp for sequential insert

• Allows for archiving to IDAA on partition level

Table Layout Alternatives



39

Well suited for time-partitioned tables where:

– Only the recent partitions are used in a transactional context (frequent data

changes, short running queries).

– The entire table is used for analytics (data intensive, complex queries).

IBM DB2 Analytics Analyzer (IDAA) – High Performance Storage Saver (HPSS)

Accelerator

Part #1Query from Application

Or

No longer present on DB2 Storage

Part #1 Part #2 Part #3 Part #4 Part #5 Part #6 Part #7

DB2

Active Historical



40

Save z storage for read-only data by storing it on the accelerator only while DB2

table spaces will be purged

Data can still be queried through accelerator

– Requires that entire workload consists of queries that can be accelerated

– Needs special register GET_ACCEL_ARCHIVE set to ‘Y’ or packages bound with option

GETACCELARCHIVE(YES)

Queries for current data can run in DB2 or accelerator

Queries for current and archived data can only be run on accelerator

Move of data from DB2 to accelerator accomplished through GUI or stored

procedure SYSPROC.ACCEL_ARCHIVE_TABLES:

– Data is loaded into accelerator if not already loaded

– Automatically takes image copy of each partition to be archived

– Automatically removes data from archived table space partitions

– Archived partitions will be marked as read-only

High Performance Storage Saver (HPSS)



41

Updates to moved data prevented by setting partitions to PRO state:

– PRO = Persistent Read Only

– Read access is allowed (through accelerator)

– Updates are prohibited

• SQL INSERT, UPDATE, DELETE statements receive SQLCODE -904 RC00C90635

– Table space partitions only

• UTS and classic partitioned

– STOP DB and START DB commands do not affect the PRO status

– MODIFY RECOVERY won’t delete reference to most current image copy

• Make sure old image copy data sets are not deleted by other processes

– Prevents COPY from taking image copy of empty partitions

High Performance Storage Saver – PRO State



42

Reusing our earlier sample for the BILLS table:

HPSS with DB2 Archive Transparency – Sample


2015/01 101 3928 34,00 €

2015/01 117 3931 25,00 €

2015/02 101 5248 43,00 €

2015/02 185 5251 53,00 €DATE CUSTNO BILLNO AMOUNT

2015/01 202 3929 9,00 €

2015/01 252 3932 45,00 €

2015/02 202 5249 44,00 €

2015/02 246 5252 53,00 €DATE CUSTNO BILLNO AMOUNT

2015/01 303 3930 53,00 €

2015/01 318 3933 19,00 €

2015/02 303 5250 52,00 €

2015/02 364 5253 54,00 €


2014/10 101 1463 50,00 €

2014/10 103 1466 26,00 €

2014/10 202 1464 29,00 €

2014/10 245 1467 3,00 €

2014/10 301 1468 53,00 €

2014/10 303 1465 53,00 €

BILLSARC:


2014/11 101 1822 13,00 €

2014/11 143 1825 27,00 €

2014/11 202 1823 37,00 €

2014/11 221 1826 31,00 €

2014/11 303 1824 11,00 €

2014/11 330 1827 34,00 €


2014/12 101 2741 46,00 €

2014/12 176 2744 49,00 €

2014/12 202 2742 54,00 €

2014/12 276 2745 35,00 €

2014/12 303 2743 37,00 €

2014/12 309 2746 25,00 €

Part1

Part2

Part3

BILLS:Part1

Part2

Part3

SET CURRENT_SCHEMA='CHM';CREATE TABLE BILLS (DATE CHAR(7) NOT NULL,CUSTNO INTEGER NOT NULL,BILLNO INTEGER NOT NULL,AMOUNT DECIMAL(10,2) NOT NULL

) PARTITION BY RANGE(CUSTNO) (PARTITION 1 ENDING AT (199),PARTITION 2 ENDING AT (299),PARTITION 3 ENDING AT (399));

CREATE TABLE BILLSARC (DATE CHAR(7) NOT NULL,CUSTNO INTEGER NOT NULL,BILLNO INTEGER NOT NULL,AMOUNT DECIMAL(10,2) NOT NULL

) PARTITION BY RANGE(DATE) (PARTITION 1 ENDING AT ('2014/10') INCLUSIVE,PARTITION 2 ENDING AT ('2014/11') INCLUSIVE,PARTITION 3 ENDING AT ('2014/12') INCLUSIVE);

ALTER TABLE BILLS ENABLE ARCHIVE USE BILLSARC;



43

Add tables to accelerator:




44

Start archiving of table partitions




45

Load archive-enabled table




46

Query tables:


SET SYSIBMADM.GET_ARCHIVE='Y';

SET CURRENT QUERY ACCELERATION=ENABLE;

SET CURRENT GET_ACCEL_ARCHIVE=YES;

SELECT * FROM BILLS ORDER BY DATE,CUSTNO;

Enable Archive Transparency

Enable query acceleration

Enable data retrieval from HPSS

Retrieve data

DATE CUSTNO BILLNO AMOUNT------- ------ ------ ------2014/10 101 1463 50.002014/10 103 1466 26.002014/10 202 1464 29.002014/10 245 1467 3.002014/10 301 1468 53.002014/10 303 1465 53.002014/11 101 1822 13.002014/11 143 1825 27.002014/11 202 1823 37.00...2015/01 303 3930 53.002015/01 318 3933 19.002015/02 101 5248 43.002015/02 185 5251 53.002015/02 202 5249 44.002015/02 246 5252 53.002015/02 303 5250 52.002015/02 364 5253 54.00



47

In order for DB2 to let archive transparency exploit HPSS correctly, both the

archive-enabled table and the archive table need to be loaded into the accelerator

Only read-only archive table partitions are moved to accelerator

Queries for current data only (archive-enabled table) can run in either DB2 or on

the accelerator (if special register CURRENT QUERY ACCELERATION is set to

ENABLE)

Queries involving the archive table can only run on the accelerator and also need

to have special register CURRENT GET_ACCEL_ARCHIVE set to YES

Remember that updates to the current data need to be reflected in the accelerated

tables in order to come up with current results:

– Either reload the current table into the accelerator before such queries

– Or, use the incremental update feature with Change Data Capture (CDC) to

keep the data in the accelerated tables current

Certain restrictions apply while data is stored on the accelerator (e.g. no schema

changes)

HPSS with DB2 Archive Transparency – Considerations



48

DB2 11 provides an integrated and transparent method to deal with frequently

updated current data as well as large amounts of read-only archive data

For more advanced journaling of data changes system versioning temporal tables

provide additional functionality

– Auditing capabilities added

– DB2 11 added transparent query enhancements (time machine, views)

IDAA High Performance Storage Saver also provides a method to deal with

current and archive data

– Can be combined with DB2‘s Archive Transparency

Archiving in DB2 – Summary



49

2015 01 26_5212_2__transparent_archiving_with

Data & Analytics

copyright ibm corporation

data archiving

archived data

inactivecold data

common problem performance

standard ibm benchmarks

potential future products

storage configuration