introduction to purexml in db2 9 for z/os

IBM Software Group | DB2 Information Management Software

1

Introduction topureXML in DB2 9 for z/OSDecember 2008

Guogen (Gene) Zhang, [email protected]

IBM Software Group

®


2

IBM Silicon Valley Lab in San Jose, CA

The first IBM lab dedicated to software development, built 30 years ago.


3

Our Mission

• Build the world’s most scalable XML database system that is easy to use.


4

DB2 for z/OS XML Design Principles

• Learn from relational scalability, leveraging mature infra-structure in DB2

• Provide application developers “revolutionary” new tools (query languages) in processing XML data productively and efficiently

• Provide DBAs familiar objects to administrate for XML data easily with familiar tools– Table spaces, indexes, buffer pools etc.

– Locking

– Performance monitoring


5

Agenda

• What is XML and where is XML used?

• Overview of DB2 XML

– SQL/XML feature in DB2 for z/OS V8

– what you can do with pureXML in DB2 9 for z/OS

– Language interfaces and tools

• Usage scenarios and business value of pureXML

• DBA’s tasks for XML

• Performance and scalability

�

6

XML document

<?xml version =“1.0” encoding=“utf-8” ?>

<?xml-stylesheet type="text/xml" href="#style1"?>

<abc:process xmlns:abc=“http://abc.com”

abc:title=“How to crack open an egg”>

<abc:step>Strike egg against an edge.</abc:step>

<abc:step>

Pull apart the shells

<abc:warning>carefully</abc:warning>

</abc:step>

</abc:process>

Start Tag

End Tag

ElementContent

Attribute

Value

Namespace DeclarationXML Declaration

QName

XML = Extensible Markup Language

PI

Comment


7

An XML Purchase Order<?xml version="1.0" encoding="UTF-8"?><ipo:purchaseOrder

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:ipo="http://www.example.com/IPO" orderDate="1999-12-01">

<shipTo exportCode="1" xsi:type="ipo:UKAddress"><name>Helen Zoe</name><street>47 Eden Street</street><city>Cambridge</city><postcode>CB1 1JR</postcode>

</shipTo><billTo xsi:type="ipo:USAddress"><name>Robert Smith</name><street>8 Oak Avenue</street><city>Old Town</city><state>PA</state><zip>95819</zip>

</billTo>

<?xml version="1.0" encoding="UTF-8"?><ipo:purchaseOrder




</billTo>

<items><item partNum="833-AA"><productName>Lapis necklace</productName><quantity>1</quantity><USPrice>99.95</USPrice><comment>Want this for the

holidays!</comment><shipDate>1999-12-05</shipDate>

</item><item partNum="926-AA"><productName>Baby Monitor</productName><quantity>1</quantity><USPrice>39.98</USPrice><shipDate>1999-12-21</shipDate>

</item></items>

</ipo:purchaseOrder>




</item></items>



8

An XML Purchase Order<?xml version="1.0" encoding="UTF-8"?><ipo:purchaseOrder




</billTo>

<?xml version="1.0" encoding="UTF-8"?><ipo:purchaseOrder




</billTo>




</item></items>





</item></items>


<billTo xsi:type="ipo:USAddress">

<name>Robert Smith</name>

<street>8 Oak Avenue</street>

<city>Old Town</city>

<state>PA</state>

<zip>95819</zip>

</billTo>

Start Tag

End Tag

ElementAttribute

Content


9

XML Characteristics

• XML can represent flexibly structured data in text– Nesting

– Repeating

– Self-describing

• XML is a universal language to represent e-Business data and transactions.

• Platform-independent, and Unicode compliant

• Easy to understand and easy to process (with the right tools)


10

<book>

<authors>

<author id=“47”>John Doe</author>

<author id=“58”>Peter Pan</author>

</authors>

<title>Database systems</title>

<price>29</price>

<keywords>

<keyword>SQL</keyword>

<keyword>relational</keyword>

</keywords>

</book>

XML Parsing

Serialization

book

title price

author

authors

author

keywords

keyword keyword

id=47 id=58 SQL relational

Database

Systems29

John Doe Peter Pan

XML Parsing and Serialization


11

Simple Parsing v.s. Validation

<employee><name>Jason</name><job>Software Engineer</job></employee>

XML Document

XDM Instance

myschema.xsdSimple Parsing

(well-formednesschecking) Validating Parsing

with XML Schema

(optional)

XDM Instance


12

XML vs. Relational

<DEPARTMENT deptid="15" deptname="Sales"><EMPLOYEE>

<EMPNO>10</EMPNO><FIRSTNAME>CHRISTINE</FIRSTNAME><LASTNAME>SMITH</LASTNAME><PHONE>408-463-4963</PHONE>

<SALARY>52750.00</SALARY></EMPLOYEE><EMPLOYEE>

<EMPNO>27</EMPNO><FIRSTNAME>MICHAEL</FIRSTNAME><LASTNAME>THOMPSON</LASTNAME> <SALARY>41250.00</SALARY>

</EMPLOYEE></DEPARTMENT>

DepartmentDEPTID DEPTNAME

15 Sales

EmployeeDEPTID EMPNO FIRSTNAME LASTNAME PHONE SALARY

15 27 MICHAEL THOMPSON NULL 4125015 10 CHRISTINE SMITH 408-463-4963 52750

Relational XML

Set oriented Sequences (ordered!)

Structure Flexibly-structured

Strong schema Schema-variability

Strongly typed Optionally typed

Tabular data model XML data model

Flat Nested, hierarchical

"Null" Not there at all


13

Schema Evolution

<DEPARTMENT deptid="15" deptname="Sales"><EMPLOYEE>

<EMPNO>10</EMPNO><FIRSTNAME>CHRISTINE</FIRSTNAME><LASTNAME>SMITH</LASTNAME><PHONE>408-463-4963</PHONE>

<PHONE>415-010-1234</PHONE>

<SALARY>52750.00</SALARY></EMPLOYEE><EMPLOYEE>

<EMPNO>27</EMPNO><FIRSTNAME>MICHAEL</FIRSTNAME><LASTNAME>THOMPSON</LASTNAME><PHONE>406-463-1234</PHONE><SALARY>41250.00</SALARY>

</EMPLOYEE></DEPARTMENT>

PhoneEMPNO PHONE

27 406-463-1234

10 415-010-1234

10 408-463-4963

Requires:

• Normalization of existing data !• Change of applications

Costly!DepartmentDEPTID DEPTNAME

15 Sales

EmployeeDEPTID EMPNO FIRSTNAME LASTNAME PHONE SALARY

15 27 MICHAEL THOMPSON 406-463-1234 41250

15 10 CHRISTINE SMITH 408-463-4963 52750

“Employees are now allowed to have multiple phone numbers…”


14

Where is XML Used?

• Data interchange and Web development– HTML 4.0 => XHTML

– Web services, SOAP, SOA

– Web 2.0 AJAX

• Documentation and content

• Database development– Managing XML data

– Use XML as a flexible hierarchical data model

• …


15

Industry-specific XML Formats• To model complex and variable business data

• To integrate businesses and applications

……

945 + 477HL7 CDA 3

1867 + 196FpML 4.2:

619 + 2593 FIXML 4.4:

77319 + 625STAR

Elements +

Attributes

Health Level 7, Clinical Document Architecture

Standards for Technology in Automotive Retail (OAGIS)

Financial products Markup Language

Financial Information eXchange Protocol


16

In Relational: 485+ tables

In pureXML:create table T(ID int, trade XML);�

�

Generate a Relational Schema for FpML


17

pureXML in DB2 9

• SQL XML data type and native storage

• Designed specifically for XML– Supports XML hierarchical structure storage

– Native operations and languages: XPath, SQL/XML, (XQuery in the future)

• Not transforming into relational

• Not using objects or nested tables

• Not using LOBs

• Integrated with relational engine, with all the utilities and tools support


18

Agenda









�


19

SQL/XML Publishing Functions in V8

• Scalar functions– Formally called XML constructors– XMLELEMENT, XMLATTRIBUTES– XMLNAMESPACES– XMLFOREST– XMLCONCAT

• Aggregate function: XMLAGG• Transient XML data type – internal use only• Cast function

– Cast internal XML values to CLOB: XML2CLOB

• Easier to use• Highly optimized – good performance

XML Extender is available on V8, deprecated in V9, will be removed in the future.


20

V8 XML Functions Example

<Department name="Shipping"><emp>Lee</emp><emp>Martin</emp><emp>Oppenheimer</emp>

</Department>

SELECT XML2CLOB(XMLELEMENT(NAME "Department", XMLATTRIBUTES (e.dept AS "name" ),XMLAGG(XMLELEMENT(NAME "emp", e.lname)

ORDER BY e.lname )) ) AS "dept_list"

FROM employees e GROUP BY dept;


21

What You Can Do with pureXML

• Create tables with XML columns or alter table add XML columns

• Insert XML data, optionally validated against schemas

• Create indexes on XML data• Efficiently search XML data• Extract XML data• Decompose XML data into

relational data or create relational view

• Construct XML documents from relational and XML data

• Handle XML in all the utilities and tools

XML

DOC

XML Column

XMLIndex

XML

- Managing XML data the same way as relational data


22

What you can do with pureXML (1)

CREATE TABLE PurchaseOrders (ponumber varchar(10) not null,podate date not null,status char(1),XMLpo xml)IN MYDB.MYTS;

CREATE TABLE PO LIKE PurchaseOrders;

CREATE VIEW ValidPurchaseOrders asSELECT ponumber, podate, XMLpoFROM PurchaseOrdersWHERE status = ‘A’;

ALTER TABLE PurchaseOrdersADD revisedXMLpo xml;


23

XML Storage on Mature Infrastructure

Base Table

XMLColDocID …

B+treeB+tree

DocID index

Internal XML Table

B+treeB+tree

NodeID index

B+treeB+tree

XML index (user)

Each XMLData column is a VARBINARY, containing a subtree or a sequence of subtrees, with context path. Rows in XML table are freely movable, linked with a NodeID index.

A table with an XML column has a DocID column, used to link from the base table to the XML table.A DocID index is used for getting to base table rows from XML indexes.

XMLDataDOCID MIN_NODEID

Regular

Table space

1

2

3

1

2

2

3

02

02

0208

02

(DB2_GENERATED_DOCID_FOR_XML)


24

Storing XML Trees - Tree Packing

Node1 (3) Node 2 (p)

Node6 Node7 (1) Node8

Node1Node1

Node2Node2 Node7Node7

Node3Node3 Node4Node4 Node5Node5 Node8Node8

Node6Node6

Node 2 (3) Node3 Node4

Node5

Each node contains local node id, length and optional number of children.

Proxy nodes are used as placeholder for subtrees in a separate record.

It supports traversal using firstChild,nextSibling, or nextNode.

RecHdr contains context path information for the record – absolute ID, path, in-scope namespaces

All names use stringIDs.

Each node contains local node id, length and optional number of children.

Proxy nodes are used as placeholder for subtrees in a separate record.

It supports traversal using firstChild,nextSibling, or nextNode.

RecHdr contains context path information for the record – absolute ID, path, in-scope namespaces

All names use stringIDs.

Rec Hdr

Rec Hdr

Node0Node002

02

02

02 04 06

04 06

02

Node0 (1)rid1

rid2


25

StringIDs to Optimize Storage and Processing

dept

name

employee

phoneid=901

John Doe

office

408-555-1212 344

name

employee

phoneid=902

Peter Pan

office

408-555-9918 216

0

1

4

25=901

John Doe

3

408-555-1212 344

1

4

25=902

Peter Pan

3

408-555-9918 216

0 dept4 employee1 name5 id2 phone3 office

String table

Tag names encoded

as unique integers


26

XML Objects for Non-partitioned Base Table

BASE Table

DOCID

INDEX

Non-Partioned Base TS

(simple, segmented, PBG)

Table for XMLCol1

NODEID

INDEX

XML

Index

PBG TS for XMLCol1

PBG TS for XMLCol2

Cols:DOCID

MIN_NODEIDXMLDATA

Cols:

DOCIDXMLCol1XMLCol2

Table for XMLCol1

NODEID

INDEX

XML

Index

Cols:DOCID

MIN_NODEIDXMLDATA

Default: Maxpartitions – 256, SEGSIZE 4, DSSIZE 4G


27

XML Objects for Partitioned Base Table

Cols:

DOCIDXMLCOL1XMLCOL2

BASE Table

Part1

Partitioned Base TS

2 Parts, Table has 2

XML Coumns

XML

Index

(NPI)

Partitioned TS for XMLCol1

(NPI)

Cols:

DOCIDXMLCOL1XMLCOL2

BASE Table

Part2

Part1 Part2

NodeID

INDEX

(NPI)

Partitioned TS for XMLCol2

DOCID

INDEX

DOCIDMIN_NODEIDXMLDATA


XML

Index

(NPI)

Part1 Part2



NodeID

INDEX

(NPI)

DSSIZE depends on base table page size (critical: max num parts)


28

What you can do with pureXML (2)EXEC SQL BEGIN DECLARE SECTION;

SQL TYPE IS XML AS CLOB(1M) xmlPo;EXEC SQL END DECLARE SECTION;

INSERT INTO PurchaseOrders VALUES (‘200300001’,CURRENT DATE, ‘A’, :xmlPo);

INSERT INTO PurchaseOrders VALUES(‘200300002’, CURRENT DATE, ‘A’, XMLPARSE(DOCUMENT :vchar PRESERVE WHITESPACE) );

INSERT into PurchaseOrders VALUES( '200300001', CURRENT DATE, 'A', XMLPARSE(DOCUMENT DSN_XMLValidate(:lobPo,’SYSXSR.myPOSchema’)) );

UPDATE PurchaseOrders SET XMLpo = :XMPpo_newWHERE ponumber = ‘12345’;

DELETE FROM PurchaseOrders WHERE ponumber = ‘12345’ ORXMLEXISTS(‘/purchaseOrder/items/item[shipDate < “2000-01-01”]’ PASSING

XMLPO);


29

- ipo:purchaseOrder orderDate="1999-12-01"- shipTo exportCode="1" xsi:type="ipo:UKAddress"

-name: Helen Zoe-street: 47 Eden Street-city: Cambridge-postcode: CB1 1JR

- billTo xsi:type="ipo:USAddress"-name: Robert Smith-street:8 Oak Avenue-city: Old Town-state: PA-zip: 95819

- items-item partNum="833-AA"

-productName: Lapis necklace-quantity: 1-USPrice: 99.95-comment: Want this for the holidays!-shipDate: 1999-12-05

-item partNum="926-AA"-productName: Baby Monitor-quantity: 1-USPrice: 39.98-shipDate: 1999-12-21

A Data RecordView and

XPath Examples

ShipTo name: /ipo:purchaseOrder/shipTo/name

BillTo: /ipo:purchaseOrder/billTo

Product names:/ipo:purchaseOrder/items/item/productName

Ship date of the first item:/ipo:purchaseOrder/items/item[1]/shipDate

Item info with product “Baby Monitor”:/ipo:purchaseOrder/items/item[productName = “Baby Monitor”]

Total amount:fn:sum(/ipo:purchaseOrder/items/item/xs:decimal(USPrice))

Sold price of product “Baby Monitor”:/ipo:purchaseOrder/items/item[productName = “Baby Monitor”]/USPrice


30

What you can do with pureXML (3)SELECT XMLpo INTO :xmlPoFROM PurchaseOrdersWHERE ponumber = ‘200300001’;

CREATE INDEX IDX1 ON PurchaseOrders(XMLPO) Generate Keys Using XMLPATTERN ‘//items/item/desc’ as SQL VARCHAR(100);

SELECT XMLQUERY(‘declare namespace ipo="http://www.example.com/IPO"; /ipo:purchaseOrder/items/item/quantity’ PASSING XMLpo)

FROM PurchaseOrders;

SELECT XMLPOFROM PurchaseOrdersWHERE XMLEXISTS(‘//items/item[desc = “Shoe”]’ PASSING XMLpo);

SELECT XMLPOFROM PurchaseOrders, ProductWHERE XMLEXISTS(‘//items/item[desc = $n]’ PASSING XMLpo,

Product.name as “n”);This is a join between XML and a regular column, it can join XML documents also


31

XML Indexes• XPath value index: index values

of elements or attributes inside a document.

• Index entries include:(key value, DocID, NodeID, RIDx)

• Support string (VARCHAR) or numeric (DECFLOAT) key type

CREATE INDEX IX1 ON PurchaseOrders(XMLPO) Generate Keys Using XMLPATTERN ‘/purchaseOrder/items/item/desc’as SQL VARCHAR(100);

<?xml version="1.0"?>

<purchaseOrder orderDate="1999-10-20">

<shipTo country="US">

<name>Alice Smith</name>

. . .

</shipTo>

<billTo country="US">


. . .

</billTo>

<comment>Hurry, my lawn is going wild!</comment>

<items>

<item partNum="872-AA">

<desc>Lawnmower</desc>

<quantity>1</quantity>

<USPrice>148.95</USPrice>

<comment>Confirm this is electric</comment>

</item>


<desc>Baby Monitor</desc>



<shipDate>2003-05-21</shipDate>

</item>

</items>

</purchaseOrder>

<?xml version="1.0"?>

<purchaseOrder orderDate="1999-10-20">

<shipTo country="US">

<name>Alice Smith</name>

. . .

</shipTo>

<billTo country="US">


. . .

</billTo>

<comment>Hurry, my lawn is going wild!</comment>

<items>


<desc>Lawnmower</desc>



<comment>Confirm this is electric</comment>

</item>


<desc>Baby Monitor</desc>



<shipDate>2003-05-21</shipDate>

</item>

</items>

</purchaseOrder>This index can be used for predicate:

XMLEXISTS(‘/purchaseOrder/items/item[desc = “Baby Monitor”]’ passing XMLPO)


XMLExists(‘/Catalog/Categories/Product[RegPrice > 100]’ passing catalog) with index on ‘/Catalog/Categories/Product/RegPrice’ as SQL DECFLOAT

DocID list access “DX”unique DocID list from an XML index, then access the base table and XML table.

XMLExists(‘/Catalog/Categories/Product[RegPrice > 100 and Discount > 0.1]’ … )With indexes on:‘//RegPrice’ as SQL DECFLOAT and ‘//Discount’ as SQL DECFLOAT

DocID ANDing/ORing“DX/DI/DU” union or intersect (unique) DocID lists from XML indexes, then access the base table and XML table.

Base algorithm: given a document, scan and evaluate XPath

DocScan “R”(QuickXScan)

DescriptionAccess Methods

New Access Methods

33

What you can do with pureXML (4)-- DML (construct XML from relational)

SELECT XMLDOCUMENT(XMLELEMENT(NAME “hr:Department", XMLNAMESPACES(‘http://example.com/hr’ as “hr”),XMLATTRIBUTES (e.dept AS "name" ),XMLCOMMENT(‘names in alphabetical order’),XMLAGG(XMLELEMENT(NAME “hr:emp", e.lname)

ORDER BY e.lname )) ) AS "dept_list”

FROM employees e GROUP BY dept; <?xml version=“1.0” encoding=“UTF-8”>

<hr:Department xmlns:hr=“http://example.com/hr”name="Shipping"><hr:emp>Lee</hr:emp><hr:emp>Martin</hr:emp><hr:emp>Oppenheimer</hr:emp>

</hr:Department>

<?xml version=“1.0” encoding=“UTF-8”><hr:Department xmlns:hr=“http://example.com/hr”

name="Shipping"><hr:emp>Lee</hr:emp><hr:emp>Martin</hr:emp><hr:emp>Oppenheimer</hr:emp>

</hr:Department>

Can construct XHTML for web pages.

Shipping

Shipping

ShippingLeeSean

AccountingJohnsonMichael

OppenheimerVicky

MartinChristine

deptlnamefirstname


34XML

Another XML Publishing ExampleSELECT

XMLSERIALIZE

( XMLELEMENT

( NAME "DEPT",

XMLATTRIBUTES

( D.DEPTNO AS "DEPTNO",

D.DEPTNAME AS "NAME" ) ,

( SELECT XMLAGG

( XMLELEMENT

( NAME "PROJ",

XMLATTRIBUTES

(P.PROJNO AS "PROJNO",

P.PROJNAME AS "NAME" ) ,

( SELECT XMLAGG

(XMLELEMENT

( NAME "EMP",

XMLATTRIBUTES

(E.EMPNO AS "EMPNO",

E.FIRSTNME || ' ' || E.LASTNAME AS "EMPNAME" )

)

)

FROM DBA015.EMPPROJACT EP,

DBA015.EMP E

WHERE EP.PROJNO = P.PROJNO AND

EP.EMPNO = E.EMPNO

)

)

)

FROM DBA015.PROJ P

WHERE P.DEPTNO = D.DEPTNO

)

)

) AS CLOB

FROM DBA015.DEPT D

WHERE D.DEPTNO = 'D01'

Query Result: (formatted for easy viewing)

<DEPT DEPTNO="D01" NAME="DEVELOPMENT CENTER">

<PROJ PROJNO="AD3100" NAME="ADMIN SERVICES">

<EMP EMPNO="000010" EMPNAME="BRIAN BARTAK">

</EMP>

</PROJ>

<PROJ PROJNO="MA2100" NAME="WELD LINE AUTOMATION">

<EMP EMPNO="000010" EMPNAME="BRIAN BARTAK">

</EMP>

<EMP EMPNO="000110" EMPNAME="VINCENZO LUCCHESI">

</EMP>

</PROJ>

</DEPT>

XML


35


SELECT XMLDocument(XMLElement(NAME "invoice",XMLAttributes( '12345' as "invoiceNo'),XMLQuery ('/purchaseOrder/billTo' PASSING xmlpo),XMLElement(NAME “purchaseOrderNo”,

PO.ponumber)XMLElement(NAME "amount",XMLQuery('fn:sum(/purchaseOrder/items/item/xs:decimal(USPrice))'PASSING xmlpo) )

) )FROM PurchaseOrders PO,WHERE PO.ponumber = ‘200300001’;


36

Aggregate/concatenate documents

• From a same row:SELECT XMLELEMENT(NAME “doc”,

XMLCONCAT(xd1, xd2, xd3) )FROM T1, T2 WHERE …

• From different rows:SELECT XMLELEMENT(NAME “doc”,

XMLAGG(xd) )FROM T1, T2 WHERE …

• xd1, xd2, xd3, xd … can be any XML expressions


37

What you can do with pureXML (6)Leveraging the Power of SQL

CREATE VIEW ORDER_VIEW ASSELECT PO.POID, X.*FROM PurchaseOrders PO,

XMLTABLE( '//item' PASSING PO.XMLPOCOLUMNS "orderDate" DATE PATH '../../@orderDate',

"shipTo City" VARCHAR(20) PATH '../../shipTo/city',"shipTo State" CHAR(2) PATH '../../shipTo/state',"Part #" CHAR(6) PATH '@partnum',"Product Name" CHAR(20) PATH 'productName',"Quantity" INTEGER PATH 'quantity',"US Price" DECIMAL(9,2) PATH 'USPrice',"Ship Date" DATE PATH 'shipDate',"Comment" VARCHAR(60) PATH 'comment' ) AS X;

SELECT "Product Name", "shipTo State",SUM("US Price" * "Quantity") AS TOTAL_SALE

FROM ORDER_VIEWGROUP BY "Product Name", "shipTo State";

SELECT "shipTo City", "shipTo State",RANK() OVER(ORDER BY SUM("Quantity")) AS SALES_RANK

FROM ORDER_VIEWWHERE "Product Name" = 'Baby Monitor'GROUP BY "shipTo State", "shipTo City"ORDER BY SALES_RANK;

XMLTable function was delivered via PTF UK33493

Can join with other tables.


38

XMLTABLE: Table view of XML DataSELECT X.*FROM PurchaseOrders PO,

XMLTABLE( '//item' PASSING PO.XMLPOCOLUMNS "Seqno" FOR ORDINALITY,"Part #" CHAR(6) PATH '@partnum',"Product Name" CHAR(20) PATH 'productName',"Quantity" INTEGER PATH 'quantity',"US Price" DECIMAL(9,2) PATH 'USPrice',"Ship Date" DATE PATH 'shipDate',"Comment" VARCHAR(60) PATH 'comment' ) AS

X;

1999-12-2139.981Baby Monitor926-AA2

Want this for the holidays!

1999-12-0599.951Lapis necklace

833-AA1

CommentShip DateUS PriceQuantityProduct Name

Part #Seqno


39


SELECT 1000 *XMLCAST( XMLQUERY('$d//*:ConversionRateResult' PASSINGXMLPARSE (DOCUMENT

DB2XML.SOAPHTTPNV('http://www.webservicex.net/CurrencyConvertor.asmx','http://www.webserviceX.NET/ConversionRate','<?xml version="1.0" encoding="utf-8"?>

<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:xsd="http://www.w3.org/2001/XMLSchema"xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body>

<ConversionRate xmlns="http://www.webserviceX.NET/"><FromCurrency>EUR</FromCurrency><ToCurrency>USD</ToCurrency>

</ConversionRate></soap:Body>

</soap:Envelope>')) AS "d") AS DECIMAL(10,5))FROM SYSIBM.SYSDUMMYU#

Consume Web Services from SQL:How much is EUR1000 worth in USD?


40


• XML Schema Repository (XSR), register schemas for– Schema validation– Annotated schema decomposition

• Remote DRDA support• Host language support: C/C++, COBOL, PL/I, Assembly, Java, .Net

• Utilities support for XML:– LOAD/UNLOAD record text data or file references– CHECK DATA– REORG– COPY– RECOVER– REBUILD– …


41

Encoding for XML

• Textual XML Data Encoding– Internally-encoded (within XML data itself)

• Encoding Declaration option or Byte Order Mark within XML data

• Default: UTF-8

• Applies to binary variables (DB2 detects encoding)

– Externally-encoded

• Character host variables CCSID (override internal encoding)

• DB2 uses UTF-8 to handle XML data. Character data in XML

always stored as UTF-8 in XML column.

• During insert, DB2 converts XML data to UTF-8 from the

corresponding CCSID.

• During select, DB2 converts XML in UTF-8 to host var encoding

locally, or sends serialized XML data in UTF-8 format to remote

requester.


42

XML Schema Support• XML Schema adds constraints on XML data (W3C Standard).• Register a schema in XML Schema Repository (XSR)

– A set of DB2 provided user tables, and stored procedures

• Schema validation (type annotation not kept)INSERT into PurchaseOrdersVALUES( '200300001', CURRENT DATE, 'A', SYSFUN.DSN_XMLValidate(:lobPO,’SYSXSR.ORDERSCHEMA’));

• Annotated schema-based decomposition – store using tables. (XDBDECOMPXML stored proc)E.g. orderID ->PORDER.ORDERID<attribute name="orderID" type=“xs:string”

db2-xdb:rowSet = “PORDER”db2-xdb:column= “ORDERID” />

annotations


43

Example: Use CLP to Register XML Schema

• Register schema

REGISTER XMLSCHEMA http://www.n1.com/order.xsdFROM file://C:/xmlschema/order.xsdAS SYSXSR.ORDERSCHEMAADD http://www.n1.com/lineitem.xsdFROM file://C:/xmlschema/lineitem.xsdADD http://www.n1.com/parts.xsdFROM file://C:/xmlschema/parts.xsdCOMPLETE [ENABLE DECOMPOSITION];

• Remove schema

REMOVE XMLSCHEMA SYSXSR.ORDERSCHEMA;

SQL Identifier SchemaLocation

SchemaLocation


44

Application Interfaces

• XML type is supported in– Java (JDBC, SQLJ), ODBC,

– C/C++, COBOL, PL/I, Assembly

– .NET

• Applications use:

– XML as CLOB(n), XML as CLOB_FILE

– XML as DBCLOB(n), XML as DBCLOB_FILE

– XML as BLOB(n), XML as BLOB_FILE

– All character or binary string types are supported

• XMLParse and XMLSerialize apply (implicitly or explicitly)


45

Java JDBC Example

Use standard interface:

PreparedStatement pstmt = connection.prepareStatement("INSERT INTO PurchaseOders VALUES(?, ?)"); // second column: XML type

…InputStream fin = new FileInputStream(file);pstmt.setBinaryStream( 2, fin, flen );pstmt.execute();

Statement s = connection.createStatement();ResultSet rs = s.executeQuery ("select ponumber, xmlpo from purchaseOrders");while (rs.next()) {int po_no = rs.getInt ("ponumber");String spo= rs.getString(2);System.out.println (spo); // uninterpreted flat xml text}

Or use com.ibm.db2.jcc.DB2Xml interface


46

COBOL Application with Embedded SQL Example

EXEC SQL BEGIN DECLARE SECTION END-EXEC.01 xmlBuff USAGE IS SQL TYPE IS XML as CLOB(5K).

EXEC SQL END DECLARE SECTION END-EXEC.

EXEC SQL SELECT xmlCol INTO :xmlBuf from myTable where id = '001'.

After Translation

01 xmlBuff.

49 xmlBuff-LENGTH PIC 9(9) COMP.

49 xmlBuff-DATA PIC X(length).

If the length of the host variables is greater than 32K, translation becomes

01 xmlBuff.

02 xmlBuff-LENGTH PIC 9(9) COMP.

02 xmlBuff-DATA.

49 filler PIC X(32767).

49 filler PIC X(32767).

:

49 filler PIC X(length - n * 32767).


47

Use Host Vars and Parameter Markers

• XML parameter marker: CAST(? AS XML)• Inside XPath:

XMLQUERY(‘/purchaseorder/items/item[desc = $x]’PASSING XMLCOL, cast(? AS VARCHAR(20)) as “x”)

XMLEXISTS(‘/purchaseorder/items/item[desc = $x]’PASSING XMLCOL, :hv as “x”)

XMLQUERY(‘/purchaseorder/items/item[desc = $x]’PASSING CAST(? AS XML), cast(? AS VARCHAR(20)) as “x”)

• Especially important for performance for short queries or OLTP applications.


48

Validate a doc – Example: COBOL using LOB• 1) XML in applications' memory (hostvar), if it's UTF-8, use BLOB hostvar is the best (use CLOB or DBCLOB if it's

not UTF-8):EXEC SQL INCLUDE COBIIVAR END-EXEC.

EXEC SQL INCLUDE COBSVAR END-EXEC.

EXEC SQL INCLUDE SQLCA END-EXEC.

...

01 BLOB-VAR USAGE IS SQL TYPE IS BLOB(16M).

...

EXEC SQL

INSERT INTO T VALUES (XMLPARSE(DOCUMENT DSN_XMLVALIDATE(:BLOB-VAR, 'SYSXSR.MYSCHEMA')) )

END-EXEC.

...

• 2) XML in a file, use LOB file reference variable is best. Assuming XML in UTF-8:...

01 BLOBFR1 SQL TYPE IS BLOB-FILE.

... *** example file is in HFS

MOVE '/u/li637/blob10a.txt' TO BLOBFR1-NAME.

MOVE 20 TO BLOBFR1-NAME-LENGTH.

MOVE SQL-FILE-READ TO BLOBFR1-FILE-OPTION.

EXEC SQL

INSERT INTO T VALUES (XMLPARSE(DOCUMENT DSN_XMLVALIDATE(:BLOBFR1, 'SYSXSR.MYSCHEMA')) )

END-EXEC.

• 3) Use LOB locator to hold XML value…


49

Development Tools

• Tool choices:– IBM Data Studio(http://www14.software.ibm.com/webapp/download/search.jsp?go=y&rs=swg-ids)– Rational Data Architect– Rational Application Developer– .NET– QMF– SPUFI– CLP (Command Line Processor)

• They can help to perform:– Schema registration, validation– Annotation for decomposition– Mapping relational to XML schema for XML generation– XML editor, schema editor (in Data Studio)


50

V9 XML Feature Summary• First-class XML type, native storage of XQuery Data Model (XDM)

• Complete SQL/XML constructor functions– XMLDOCUMENT, XMLTEXT, XMLCOMMENT, XMLPI

– From V8: XMLELEMENT, XMLATTRIBUTES, XMLNAMESPACES, XMLFOREST, XMLCONCAT, XMLAGG

• XMLPARSE, XMLSERIALIZE, XMLCAST

• XML indexes– use for XMLEXISTS and XMLTABLE

• Other SQL/XML functions with XPath– XMLEXISTS, XMLQUERY, XMLTABLE

• XML Schema repository, Validation UDF, and decomposition

• DRDA (distributed support) and application interfaces

• Utilities and tools

XMLTABLE and XMLCAST: available through PK51571, PK51572, PK51573


51

DB2 for z/OS XML Unique Features

• Mature optimized storage infrastructure, compact storage with optional high-ratio compression– As little as 0.3 of original document size

• Synergy with z/OS XML system services for highly efficient parsing, achieving unparalleled XML insertion and load performance

• Schema validation exploiting latest XML validating parser (XLXP-C);

• Efficient XML indexes for scalability; Patent pending highly efficient XPath algorithm for query performance;

• Partitioned table space and data sharing support;• Redirection of XML processing to the zIIP specialty

engine from DRDA workload;• 100% redirection of XML parsing to the zIIP or zAAP

specialty engine in the near future.


52

Agenda









�


53

Usage Scenarios: Directly Processing XML

Financial Services XML Standards

�XML for data in motion - storing what is exchanged

–SEPA/UNIFI, ACORD, FIXML, FpML, MIMSO, XBRL, …

–DJXDM, HR-XML, HL7, ARTS, HIPAA, NewsML, XForms, …

–Insurance policy, contract, purchase order, emails, etc.

X9NACHAECCHO

UNIFI (ISO 20022)


54

• Data analysis of intermittent process status– Immediate access to arriving data via XML tools and SQL

• Data warehouse

• Auditing

• ...

XML in DB2

Data Warehouse

Operational Data Store

Diverse Data Sources

XML

Cleansing

Extracting Or

Mapping

Usage Scenarios: Business Monitoring


55

Output XSLT

Feeds

JSON

Any text formatincluding XML, PDF

• Flexibility in Web service input and output formats • Alleviates ‘top-down’ Web service format requirements

Usage Scenarios: Generating web pages: XHTML,...


56

• Versatile schemas and enable end-user customizable applications.

–very difficult to reflect in a relational schema

• Object persistence (single XML column v.s. many tables)

–avoiding mapping of complex XML into many relational tables

• Sparse attribute values (null v.s. absence)

–Product features (e.g. car feature)

Usage Scenarios: Relational Schema vs. XML


57

DB2 as Web Services Consumer …

• Implementation–DB2 server-side routines (SOAP UDFs)– Light-weight native implementation–Can be called inside an SQL statement or from within other server-side routines

• Process flow–Receive input parms from SQL statement– Compose HTTP/SOAP or HTTPS/SOAP request – Invoke TCPIP socket call and send Web Service request– Receive reply from Web Service Provider

• Validate HTTP headers • Strip SOAP envelope and return SOAP Body

• Tooling support for DB2 in RAD –Create a SQL UDF specifically for a WS operation from WSDL–Creates ‘wrapper’ SQL UDF to hide XML/relational mapping.


58

… and DB2 as Web Service Provider

• Tooling integrated in IBM Data Studio– Query builder

• Integrated Query Editor for SQL and XQuery

• Graphical SQL Builder

– Stored Procedure builder

• Integrated debugger for Java and SQL PL procedures

• Run and test procedures with DB2 and IDS

• Rapid generation of Web services from database operations using intelligent

defaults

• No programming required

• Assembles a “ready-to-deploy” Data Web Services (DWS) Web application

• Integrated deploy and test environment

• Automated Web service artifact creation

– WSDL documents

• Web service exploration and test tools

• Support for DB2 pureXML and XSLT


59

DataStudio Tooling for Data Web Services

1. Develop Statements

2. Create Service3. Drag ‘n drop

Service assembly

4. Deploy Service

5. Test Service


60

• Usually hundreds-thousands of different tax forms

�Schema Diversity

• Typically not every field in a form is used

�Sparse Data

• Many forms change every year

�Schema Evolution

��A case for XML !

Tax Forms


61

XML: Avoids sparsity. Proper data labeling. 2 columns, not 1000. Transformable. Extensible. Simplifies mapping.

Generic columns � XMLCurrent relational storage,inefficient, anonymous columns, requires complex mappings in the application

New XML format:

…

No…NULL123.23NULLNULLNULL

NULL…NULL99.99NULLNULL12

NULL…YesNULLNULL276NULL

NULL…NULLNULL11/23/05NULL134

col1000…col5col4col3col2col1

<form><wages>134</wages><date>11/23/05</date>

</form>


62

XML and the Web before DB2 pureXML

Client Application Server

XML

XML ObjectsMapping Mapping

Relational schema

Relational Database

• XML to object mapping and object to relational mapping

– Very costly

– Complex

– Inflexible


63

DB2 pureXML and the Web

Client

XML

DB2 pureXML

relationalXMLSOA Gateway

• End-to-End XML– No expensive object mapping

– Pass Thru XML from/to database

• SOA-Gateway

– Device/application to handle network protocols, security, reliability, performance

– Easy to manage

• Simple pre- and post-processing of XML – e.g. via XSLT


64

XML as front to backend systems

XML

DB2 pureXML

relationalXML

XML

XML

Backend 1

Backend 2

Backend 3

Backend n

Interface Tables 1

Interface Tables 2

Interface Tables 3

Interface Tables n

Physical tables or logical views


65

XML-Enabled Databases:Two Main Options

XML

DOC

Extract

selected

elements/attr.

Side Tables

CLOB/Varchar

XML DOC

XML DOC

XML DOC

XML

DOC

Varchar or clobcolumn

Fixed

MappingShredder

(regular tables forfaster lookup)

(regular relational tables)

“Decomposition”

Shredding


66

DB2 pureXML Advantages

• Directly store XML, no decomp/comp, normalize/de-normalize

• Eliminates database schema evolution bottleneck

• Declarative language, reduce complexity, dramatically improve application development productivity

• Native processing, high performance

• Unparalleled reliability, availability, scalability

Up to 10

times


67

Business Value of pureXML

• Speed up Application Development– Reduced system and development complexity– Improved developer productivity

• Greater Business Agility– Easily accommodate changes to data and schemas– Update applications rapidly and reduce maintenance costs

• Improved Business Insight– Access to information in otherwise unexploited documents– Unprecedented application performance

• Consolidating converged information on System z– Reduce floor space, power consumption, cooling cost– Consolidate information resources and reduce admin cost


68

Agenda









�


69

A few Things that are New related to XML

1. Implicitly created objects – DBA cannot create explicitly

2. XML indexes: XPath and keys

3. XML Schema registration

4. XMLDATA contains StringIDs in a catalog table SYSIBM.SYSXMLSTRINGS (dictionary)• UNLOAD FROMCOPY restricted, DSN1COPY

5. A new XML lock type, ID '35'x in traces• IFCID 20, 21, 107, 150, 172, and 196.

6. XML keyword in some utilities


70

System Configurations

• Basic XML parsing requires z/OS XMLSS: z/OS 1.8 or z/OS 1.7 with APAR OA16303

• XML schemas requires IBM 31-bit SDK for z/OS, Java 2 Technology Edition, V5 (5655-N98), SDK5. And Java stored procedure setup.

• Zparms for virtual storage: XMLVALA and XMLVALS– Default: 200MB and 10GB.– Also LOBVALA and LOBVALS impact bind-in and bind-out of

XML

• Buffer pool for XML tables (default BP16K0), authorization for users who create or alter tables with XML columns.– DEFAULT BUFFER POOL FOR USER XML DATA ===>

BP16K0 BP16K0 - BP16K9


71

Overview of utilities for XML

• No new special utilities for XML.

• DB2 utilities support the XML data type and the related database objects – The XML data type in LOAD and UNLOAD, with file reference support

– Support for the XML table spaces and tables used to store XML column values

– Support for auxiliary relationships used to connect base tables to XML tables

– Support for the base table space DocID index

– Support for the XML table space NodeID index and XML indexes

• No partition level checking for XML PBG UTS(APAR pk49033)

• DSN1COPY copied data set containing XML objects cannot be moved to another system due to StringIDs (dictionary specific to each system).


72

DBA Considerations

• Database administrators should treat XML database

objects as they do in LOB database objects.

• Like LOB objects, XML objects contain data stored

outside the base table space.

– XML table spaces and index spaces must be consistent also with

their related base table

• All the utilities either support or tolerate XML, with

some specific XML keyword support.

• In addition, XML has more considerations, such as

table space size limit, compression, and indexes.


73

Table Space Size Consideration

• Basic XML storage is about 0.3 (strip whitespace w/ compression) to 1.5 (preserve ws w/o compression) of original doc size

• An XML table space always use 16KB pages.– For non range-partitioned base table spaces, PBG table space is

used for XML.

• Range-partitioned base table spaces: XML partitioning follows base table partitioning.

• The number of rows to fit into a relational partition is limited by the number of documents to fit into an XML partition.– For example, 4K doc size, 32GB partition can roughly store 8M

documents (or 7M to be safe).


74

REPORT TABLESPACESET - OutputContains both a LOB column and an XML column: POTABLE(I INT, CHARPO CLOB, PO XML).(New XML text is shown in red)

TABLESPACE SET REPORT:

TABLESPACE : DBMYPO.TSMYPOTABLE : ADMF001.POTABLEINDEXSPACE : DBMYPO.IRDOCIDPINDEX : ADMF001.I_DOCIDPOTABLE

LOB TABLESPACE SET REPORT:TABLESPACE : DBMYPO.TSMYPOBASE TABLE : ADMF001.POTABLECOLUMN : CHARPO

LOB TABLESPACE : DBMYPO.TSLOBPOAUX TABLE : ADMF001.TBLOBPOAUX INDEXSPACE : DBMYPO.IXLOBPOAUX INDEX : ADMF001.IXLOBPO

XML TABLESPACE SET REPORT:

TABLESPACE : DBMYPO.TSMYPOBASE TABLE : ADMF001.POTABLECOLUMN : PO

XML TABLESPACE : DBMYPO.XPOT0000XML TABLE : ADMF001.XPOTABLEXML NODEID INDEXSPACE: DBMYPO.IRNODEIDXML NODEID INDEX : ADMF001.I_NODEIDXPOTABLEXML INDEXSPACE : DBMYPO.XMLIDXXML INDEX : ADMF001.XMLIDX

TABLESPACE : DBMYPO.TSMYPOBASE TABLE : ADMF001.POTABLECOLUMN : POXML TABLESPACE : DBMYPO.XPOT0000XML TABLE : ADMF001.XPOTABLEXML NODEID INDEXSPACE: DBMYPO.IRNODEIDXML NODEID INDEX : ADMF001.I_NODEIDXPOTABLEXML INDEXSPACE : DBMYPO.XMLIDXXML INDEX : ADMF001.XMLIDX


75

LISTDEF – define a list of related objects

• Example:

LISTDEF listname INCLUDE TABLESPACESTABLESPACE tsname [RI] [ALL | BASE | LOB | XML]

• Valid specifications:– BASE (non-LOB and non-XML objects)

– LOB (LOB objects)

– XML (XML objects)

– ALL (BASE, LOB, and XML objects)

– TABLESPACES (related table spaces)

– INDEXSPACES (related index spaces)

– RI (related by referential constraints, including informational referential constraints)

– Initial object can be: DB, TS, IS, Table, Index


76

UNLOAD XML Data

• To unload XML data directly to output records, specify XML as the field type.– non-delimited format: a 2-byte length will precede the value of

the XML.

– For delimited output, no length field is present.

– Limit to 32K in length

• To unload XML data to separate files:– Specify CHAR(n)/VARCHAR(n) BLOBF, CLOBF or DBCLOBF

for file names

– Use the template control statement to create the XML output fileand filename

• UNLOAD FROMCOPY is restricted


77

File References – UNLOAD to HFS

SAMPLE.UFILEREF.SYSREC:ß@ /u/sample/clobf/BI1OFEHQÑ@ /u/sample/clobf/BI1OFEH0`@ /u/sample/clobf/BI1OFEH1i@ /u/sample/clobf/BI1OFEINr@ /u/sample/clobf/BI1OFEIO

SAMPLE.UFILEREF.SYSREC:ß@ /u/sample/clobf/BI1OFEHQÑ@ /u/sample/clobf/BI1OFEH0`@ /u/sample/clobf/BI1OFEH1i@ /u/sample/clobf/BI1OFEINr@ /u/sample/clobf/BI1OFEIO

$ cd /u/sample/clobf$ lsBI1OFEH0 BI1OFEH1 BI1OFEHQ BI1OFEIN BI1OFEIO

$ cd /u/sample/clobf$ lsBI1OFEH0 BI1OFEH1 BI1OFEHQ BI1OFEIN BI1OFEIO

//CRTMNT EXEC PGM=BPXBATCH, // PARM='sh mkdir /u/sample; chmod 777 /u/sample;// chown sysadm /u/sample'

//STEP1 EXEC PGM=BPXBATCH, // PARM='sh mkdir /u/sample/clobf‘...TEMPLATE TCLOBF DSN /u/sample/clobf

DIR(5) DSNTYPE(HFS)UNLOAD TABLESPACE SAMPLEDB.SAMPLETS

PUNCHDDN SYSPUNCH UNLDDN SYSRECFROM TABLE ADMF001.SAMPLETB( MYCOL1 POSITION(*) DECIMAL(5,2) ,MYXML1 POSITION(*) VARCHAR CLOBF TCLOBF)

//CRTMNT EXEC PGM=BPXBATCH, // PARM='sh mkdir /u/sample; chmod 777 /u/sample;// chown sysadm /u/sample'

//STEP1 EXEC PGM=BPXBATCH, // PARM='sh mkdir /u/sample/clobf‘...TEMPLATE TCLOBF DSN /u/sample/clobf

DIR(5) DSNTYPE(HFS)UNLOAD TABLESPACE SAMPLEDB.SAMPLETS

PUNCHDDN SYSPUNCH UNLDDN SYSRECFROM TABLE ADMF001.SAMPLETB( MYCOL1 POSITION(*) DECIMAL(5,2) ,MYXML1 POSITION(*) VARCHAR CLOBF TCLOBF)


78

LOAD XML Data

• LOAD uses internal INSERT for XML data – honors LOG(NO).

• To load XML directly from input records, specify XML as the field type.– LOAD DATA INDDN(INFILE) LOG NO RESUME(NO) FORMAT DELIMITED

INTO TABLE PURCHASEORDERS

– LOAD DATA INDDN(INFILE) LOG NO RESUME(NO)… XMLPO POSITION(20)

XML PRESERVE WHITESPACEINTO TABLE PURCHASEORDERS

• To load XML from files, specify CHAR or VARCHAR along with either BLOBF, CLOBF or DBCLOBF.

• Schema validation not supported for LOAD.

• XML compression takes effect after first REORG, not on initial LOAD. Same for FREEPAGE, PCTFREE.


79

File References – LOAD from HFS

LOAD DATA INDDN SYSREC LOG NO RESUME YESEBCDIC CCSID(00037,00000,00000)SORTKEYS 10 INTO TABLE "ADMF001"."SAMPLETB“WHEN(00001:00002) = X'0003‘( "DSN_NULL_IND_00001" POSITION( 00003) CHAR(1), "MYCOL1"

POSITION( 00004:00006) DECIMAL PACKEDNULLIF(DSN_NULL_IND_00001)=X'FF'

, "DSN_NULL_IND_00002" POSITION( 00007) CHAR(1), "MYXML1“

POSITION( 00008) VARCHAR CLOBFNULLIF(DSN_NULL_IND_00002)=X'FF'

)

LOAD DATA INDDN SYSREC LOG NO RESUME YESEBCDIC CCSID(00037,00000,00000)SORTKEYS 10 INTO TABLE "ADMF001"."SAMPLETB“WHEN(00001:00002) = X'0003‘( "DSN_NULL_IND_00001" POSITION( 00003) CHAR(1), "MYCOL1"

POSITION( 00004:00006) DECIMAL PACKEDNULLIF(DSN_NULL_IND_00001)=X'FF'

, "DSN_NULL_IND_00002" POSITION( 00007) CHAR(1), "MYXML1“

POSITION( 00008) VARCHAR CLOBFNULLIF(DSN_NULL_IND_00002)=X'FF'

)


80

Operation and Recovery

• To recover base table space, take image copies of all related objects

– Use LISTDEF to define a list of related objects

– (QUIESCE not needed in V9)

• COPYTOCOPY may be used to replicate image copies of XML objects.

• MERGECOPY may be used to merge incremental copies of XML table

spaces.

• Point in Time recovery (RECOVER TORBA, TOLOGPOINT)

– All related objects, including XML objects must be recovered to a consistent

point in time

• Optional: CHECK utilities to validate base table spaces with XML

columns, XML indexes and related XML table spaces.

• If there is an availability issue with one object in the related set, availability

of the others may be impacted.


81

Diagnosing Problem Related to XML Objects

• Identify XML tables and their related objects

– Run REPORT TABLESPACESET

• Validate that the auxiliary index is consistent with the underlying

table spaces

– Run CHECK INDEX on all indexes, DocID, NodeID and XML value indexes

• Validate the logical connection between the base table and XML

table.

– Run CHECK DATA against the base table space.

• Use Repair to diagnose problem related to base table spaces

with XML columns and their DocID index

– Use REPAIR LOCATE KEY to locate a row using DocID key in the DocID index

• Use Repair to diagnose problem related to XML table spaces and

their NodeID index or XML Value Index

– Use REPAIR LOCATE RID to locate a row using a RID.


82

Checking data integrity

XML table space

�CHECK INDEX on DOCID, NODEID, XML indexes

�CHECK DATA on base tablespace

ƒSCOPE AUXONLY

ƒAUXERROR REPORT

ƒAUXERROR INVALIDATE

CHECK INDEX(2)

NODEIDIndex

Base table space

XML table space

Cols:DOCIDNODEID

XML Record Value

XMLIndex

DOCIDIndex

2 - CHECK INDEX

3 - CHECK DATA

1 - CHECK INDEX

Cols:DOCIDXMLCOL


83

Performance Monitoring and Tuning

• There are no special changes in DB2 Performance Expert to support XML other than minor points - such as new XML locks.

• XML performance problem can be analyzed through accounting traces and performance traces.

• There is a new LOAD MODULE for XML: DSNNXML in ADMF address space

• XML indexes have the same consideration as other indexes.

• The REORG utility should be used to maintain order and free space.

• Run RUNSTATS for statistics to help pick XML indexes.


84

Checking query plan

CREATE TABLE ACORD.REQUEST (ID BIGINT NOT NULL PRIMARY KEY,REQUESTXML XML,RESPONSEXML XML) IN DATABASE DBACORD

CREATE INDEX ACORD.ACORDINDEX1 ON ACORD.REQUEST(REQUESTXML)GENERATE KEYS USING XMLPATTERN'declare default element namespace "http://ACORD.org/Standards/Life/2";/TXLife/TXLifeRequest/TransRefGUID' as SQL VARCHAR(24)

CREATE INDEX ACORD.ACORDINDEX2 ON ACORD.REQUEST(REQUESTXML)GENERATE KEYS USING XMLPATTERN'declare default element namespace "http://ACORD.org/Standards/Life/2";/TXLife/TXLifeRequest/OLifE/Holding/Policy/@id' AS SQL VARCHAR(9)


85

Query plan (cont’ed)

+-------------------------------------------------------------------------------------+| PLANNO | ACCESSTYPE | MATCHCOLS | ACCESSCREATOR | ACCESSNAME | MIXOPSEQ |+-------------------------------------------------------------------------------------+

1_| 1 | M | 0 | | | 0 |2_| 1 | DX | 1 | ACORD | ACORDINDEX2 | 1 |3_| 1 | DX | 1 | ACORD | ACORDINDEX1 | 2 |4_| 1 | DI | 0 | | | 3 |+-------------------------------------------------------------------------------------+

Find participant information

about a policy.

EXPLAIN PLAN SET QUERYNO = 101 FORSELECT

XMLQuery('declare default element namespace "http://ACORD.org/Standards/Life/2";/TXLife/TXLifeRequest/OLifE/Holding/Policy/Life/Coverage/LifeParticipant'PASSING R.REQUESTXML),

XMLQuery('declare default element namespace "http://ACORD.org/Standards/Life/2";/TXLife/TXLifeRequest/OLifE/Party[@id =/TXLife/TXLifeRequest/OLifE/Holding/Policy/Life/Coverage/LifeParticipant/@PartyID ] '

PASSING R.REQUESTXML)FROM ACORD.REQUEST R

WHERE XMLExists('declare default element namespace"http://ACORD.org/Standards/Life/2";

/TXLife/TXLifeRequest[TransRefGUID="2004-1217-141016-000012"]/OLifE[Holding/Policy/@id="POLICY12"]'

PASSING R.REQUESTXML)


86

Steps in a picture

XML IDX1XML IDX1 XML IDX2XML IDX2

DOCID list 1 DOCID list 2

INTERSECT

DOCID list

DOCID IDXDOCID IDX

RID list

Base Table NODEID IDXNODEID IDX1 2

34

56

Base Table

XMLColDocID …

B+treeB+tree

DocID index

Internal XML Table

B+treeB+tree

NodeID index

B+treeB+tree

XML index (user)


1

2

3

1

2

2

3

02

02

0208

02

Base Table

XMLColDocID …

B+treeB+tree

DocID index

Internal XML Table

B+treeB+tree

NodeID index

B+treeB+tree

XML index (user)


1

2

3

1

2

2

3

02

02

0208

02


87

Agenda








• Performance and scalability�


88

Performance and Scalability

• XML storage leverages mature optimized storage infrastructure. Compact format, compressed well.

• Next generation parsers: z/OS XML System Services and XLXP-C.– Validation is more expensive than well-formednesschecking only

• Highly efficient XPath streaming algorithm• Support partitioned table spaces and data sharing.

• Initial sweet spot: a large number of small documents.


89

Customer Sample Performance Test

0.300.3451285256980.230.3422868781RELUpdate

0.681.202282072721.041.475054178CLOBUpdate

1557987445250262XMLUpdate

399701766033874376RELRetr

111075309170191228CLOBRetr

5251479815824148XMLRetr2

5148472222106128XMLRetr1

0.400.4366485313960.370.2629145998RELInsert

0.811.0432715128600.931.0211554255CLOBInsert

264821337510715260XML Insert

EL

XML/

CPU

XML/ElapsedCPU

El

XML/

CPU

XML/ElapsedCPU

DB2 / ZIIPApplication

1 XML column1 CLOB column3 relational tablesNo indexes


90

Storage for UNIFI Messages

0

20

40

60

80

100

120

140

160

180

Storage (KB)

Original Docs DB2XML Strip WS DB2XML Pres WS

Uncompress (KB)

Compress (KB)

96 sample documentsStrip WS: Strip WhitespacesPres WS: Preserve Whitespaces

67%

70%

58%52% 71%


91

Table Compression

0

100

200

300

400

500

Time in second

EL CPU EL CPU

insert Xscan

Non compressed XML table Compressed XML table


92

Insert Performance (Batch)

Measurement in March 2007, z9 DS8300, Single thread, Docs in EBCDIC

0

50

100

150

200

250

300

1K x 1000000 10K x 100000 100K x 10000 1M x 1000 10M x 100

Doc Size x Number

Time (Sec)

Elapsed CPU

3.9 millions 10K

docs per hour or

1100 docs/sec


93

Insert Performance – compare w/ CLOB

(average of 1K to 10M document insert performance)

100% 100%111% 116%

71.36%64%

0%

20%

40%

60%

80%

100%

120%

140%

Elapsed CPU

XML XML w/ One index CLOB


94

Fetch Performance (Batch)

Measurement in March 2007, z9 DS8300, Single thread, Docs in EBCDIC

0

10

20

30

40

50

60

70

80

1K x 1000000 10K x 100000 100K x 10000 1M x 1000

Doc size x Number

Time (sec)

Elapsed CPU

9.3 millions 10K

docs per hour or

2580 docs/sec


95

XML Insert v.s. Validation v.s. Decomposition

0

2

4

6

8

10

12

Insert Insert with

Validation

Decomposition

4K doc Single thread 3000 repeat

Time in second

Elapsed

CPU


96

XML and Specialty Engines

• DB2 zIIP support applies to XML processing – DRDA zIIP– Utility index zIIP

• Non validation parser zAAP support – 100% of XMLSS under TCB mode is eligible

– z/OS 1.9 and APAR OA20308 for 1.8 and 1.7– Observed..

• 13% to 30% using small XML insert • Up to 50% using large XML load

• PK50575 for zAAP accounting support

• zIIP support for z/OS XML SS– z/OS 1.10, APAR OA23828 for z/OS 1.8, 1.9

• Validation parser support in z/OS 1.10 XML System Services


97

0

2

4

6

8

10

12

14

16

CPU time - ms for insert,

- sec for load

Order_insert (1-

2K)

CustAcc_insert (4-

20KB)

LOAD (2.5MB)

XMLSS usage in insert and load

zAAP eligble CP

z/OS XML Specialty Engine Support

14%32%

48%


98

LOAD Testing

21%57209Average

36%641141 25M200LOAD7

20%64 2541 25M200LOAD6

20%8233022.5M2,000LOAD5

20%82 32922.5M2,000LOAD4

29%389314K-20K300,000LOAD3

17%3115224K-20K300,000LOAD2

17%4019144K-20K300,000LOAD1

Redirection

percentage

CPU

time in

zAAP

CPU time

in

general

CP(sec)

Number of

User XML

Indexes

Size of XML

documents

(bytes)

Number

of Rows

Job

Processor IBM System z9 Enterprise Class (z9 EC) LPAR configuration: 4 General Purpose CPs, 1 zAAP, 1 zIIP, all dedicated Memory: 24GB memory Storage IBM DS8300 Operating system z/OS Version 1.9 DB2 DB2 9 Feb 2008 PTF level


99

TPoX Insert Test

18.23%32.30%XML System Service CPU usage in LPAR

356.15309.09Internal Throughput rate

226.9236.3Number of transactions per second (10 inserts /tx)

7.5867.909DB2 class1 average CPU time (ms, per commit)

88.50177.565DB2 class1 average elapsed time (ms, per commit)

22692363XML inserts per second (average)

2020 Threads Number of concurrent threads

63.71%76.45%LPAR CPU utilization

z/OS 1.9

(Case-B)

z/OS 1.8

(Case-A)TPoX Mass insert2

Processor IBM System z9 Enterprise Class (z9 EC) LPAR configuration: 3 dedicated General Purpose CPs (no zIIP no zAAP) Memory: 24GB memory Storage IBM DS8300 Case–AOperating system z/OS Version 1.8 DB2 DB2 9 June 2007 PTF levelCase –BOperating system z/OS Version 1.9 DB2 DB2 9 Feb 2008 PTF level


100

TPoX Mixed Transaction Test (1/2)

Processor IBM System z9 Enterprise Class (z9 EC) LPAR configuration: 3 General Purpose CPs, dedicated (no zAAP or zIIP) Memory: 24GB memory Storage IBM DS8300 Operating system z/OS Version 1.9 DB2 DB2 9 Feb 2008 PTF levelThreads 35

20Insert Insert_order

20Insert Insert_custacc

12QueryGet_security_price (6)

12QueryAccount_summary (5)

12QueryCustomer_profile (3)

12QueryGet_security (2)

12Query Get _order (1)

Transaction Weight Type of transaction Transaction name


101

TPoX Mixed Transaction Test (2/2)

3.5% z/OS XML System Services CPU consumption

2068 tpsInternal Throughput Rate

58.3% CPU utilization

1207 tpsTransactions per second

0.02 second Insert_order

0.03 second Insert_custacc

0.02 second Get_security_price

0.03 second Account_summary

0.03 second Customer_profile

0.03 second Get_security

0.03 second Get _order

Average Transaction

Response time

Transaction


102

Future Directions in DB2 for z/OS*

• Basic XQuery support in XMLQuery, XMLTable, XMLExists– FLWOR and constructors– More XML built-in types, XQuery functions

• XML column type modifier with XML schemas and automatic validation

• Trigger support• XML support in SQL PL stored procedures and UDFs• Sub-document update• Binary XML support for bind-in and bind-out• Schema validation inside the engine• XSLT support• …* All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.


103

References on DB2 z/OS pureXML

• Introductory whitepaper: Leveraging DB2 9 for z/OS pureXML Technology (2nd Edition)http://www.ibm.com/developerworks/wikis/download/attachments/1824/Leveraging_DB29_for_zOS_whitepaper_v2.pdf

• Detailed introduction presentation: ftp://ftp.software.ibm.com/software/data/db2zos/presentations/2007/misc/purexml.pdf

• DB2 for z/OS XML Guide: http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/topic/com.ibm.db29.doc.xml/bknxspsh.htm

• DB2 9 and z/OS XML System Services Synergy Update:http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101227

• IBM developerWorks DB2 XML (LUW): http://www.ibm.com/developerworks/db2/zones/xml/index.html

• Send emails to: [email protected]


104

DB2 pureXML ResourcesWeb sitewww.ibm.com/software/data/db2/xml/

Wikiwww.ibm.com/developerworks/wikis/display/db2xml/

Forumwww.ibm.com/developerworks/forums/forum.jspa?forumID=1423/

Team Blogwww.ibm.com/developerworks/blogs/page/purexml

ChannelDB2 User Groupwww.channeldb2.ning.com/group/pureXML


105

Backup


106

Implicitly Created Objects

NM: G,TB: XML,IS: G,ST: SYSDEFLT

NM: G,TB: XML,IS: G,ST: from Base

NM: G,TB: XML,IS: G,ST: from Base

NM: G,TB: XML,IS: G,ST: from BaseBP: Deflt/BP0

NodeIDindex

NM: given,TB: Base,IS: G,ST: given or SYSDEFLT

NM: G,TB: Base,IS: G,ST: SYSDEFLT

NM: G,DB: DSNDB04,TS: G,ST: from Base

NM: given,DB: DSNDB04,TS: given,ST: from TS

IN tablespace

NM: given,TB: Base,IS: G,ST: given or SYSDEFLT

NM: G,TB: Base,IS: G,ST: SYSDEFLT

NM: G,DB: Base,TS: G, ST:SYSDEFLT

NM: given,DB: G,TS: G, ST:SYSDEFLT

none

NM: given,

TB: Base,IS: G,ST: given or DB

NM: G, TB: Base,IS: G,ST: from DB

NM: G,DB: Base,TS: G,ST: from Base

NM: given,DB: given,TS: G,ST: from DB

IN DATABASE database

NM: given,

TB: Base,IS: G,ST: given or DBBP: given or Deflt

NM: G, TB: Base,IS: G,ST: from DBBP: Deflt/BP0

NM: G,DB: Base,TS: G,ST: from BaseBP: Deflt/BP16K0

NM: given,DB: given,TS: given,ST: from DBBP: given or Deflt

IN database.tablespace

XML IndexDocID indexXML tableBase tableCreate Table option

NM: Name, TS: Table Space, ST: Stogroup, BP: Buffer pool, TB: Table, IS: Index Space, G: Generated.


107

DSSIZE for XML Table Space (PTF PK50692)

• PBG XML DSSIZE = 4GB (base is simple, SEG, or PBG)

• PBR XML DSSIZE based on base DSSIZE and page size

• Rationale: not to limit base table max number of partitions.

SQL Ref


108

XML Related Locks• New XML lock type, ID ’35’x in instrumentation

Nones lock, release at commit

s page/row lockSELECT RR, RS

Nones lock, release on next fetch

s page/row lock on rowset, release on next fetch

SELECT UR, CS-CDN, CS-CDY with Multirow fetch and dynamic scrolling

Nones lock, release at close cursor

s page/row lock, release on next row fetch

SELECT CS-CDY workfile

Nones lock, release at next row fetch

s page/row lock, release on next row fetch

SELECT CS-CDY no workfile

Nones lock, release at next row fetch

NoneSELECT UR, CS-CDN

x page lock, release at commit

x lock, release at commit

u->x, s->x, x stays xUPDATE/DELETE

x page lock, release at commit

x lock, release at commit

x page/row lockINSERT

XML Table space Page Lock

XML LockBase Page/Row Lock (Business as usual)

SQL

Improvement under wayPK28265

page latch (and optional P-Lock)

page latch (and optional P-Lock)


109

Query Predicate Must Match Index Data Type<customerinfo cid=“1004”>

<name>Matt Foreman</name><phone>905-555-4789</phone>

</customerinfo>

XMLEXISTS(‘$i/customerinfo[@cid = 1004]’…)

XMLEXISTS(‘$i/customerinfo[@cid = “1004”]’ …)

…using xmlpattern '/customerinfo/@cid' as SQL DECFLOAT;

…using xmlpattern '/customerinfo/@cid' as SQL VARCHAR(10);

Index definition→

↓Predicate

It’s a

number

It’s a

string

Index used Index not used

What about XMLEXISTS(‘$i/customerinfo[@cid = “1004”]/name’ …) ?And XMLEXISTS(‘$i/customerinfo[@cid = 1004]/name’ …) ?


110

XML Index should “contain” query predicates

• Option 1: create index idx1 on customer(info) generate key usingxmlpattern '/customerinfo/phone' as SQL varchar(40);

• Option 2: create index idx2 on customer(info) generate key using

xmlpattern '//phone' as SQL varchar(40);

• Option 3: create index idx1 on customer(info) generate key using

xmlpattern '/customerinfo/*' as SQL varchar(40);

� Assume we expect predicates on “phone”.There are several ways to index this, including:

create table customer(info XML);<customerinfo cid=“1004”>

<name>Matt Foreman</name><phone>905-555-4789</phone>

</customerinfo>


111

Query predicate examples

• Consider these index/predicate pairs:

…using xmlpattern'//phone‘

as sql varchar(35);

EXISTS(‘$i/customerinfo[* = “905-555-4789”]’ … )

XMLEXISTS( ‘$i/customerinfo[phone = “905-555-4789” ]’ …)

EXISTS(‘$i//phone[. = “905-555-4789”]’ …)

…using xmlpattern'/customerinfo/*‘

as sql varchar(35);

…using xmlpattern '/customerinfo/phone‘

as sql varchar(35);

Index definition→

↓Predicate

<customerinfo cid=“1004”><name>Matt Foreman</name><phone>905-555-4789</phone>

</customerinfo>

IBM Software Group

© 2007 IBM Corporation112

Important Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.

IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR

• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE.

introduction to purexml in db2 9 for z/os

Documents

software development

xml performance

processing xml data

relational relational

ibm lab

selfdescribing xml

xml purchase order xmlns

zos xml design principles