chapter 10: xml
DESCRIPTION
Chapter 10: XML. The world of XML. The Data. Semistructured data instance = a large graph. The indexing problem. The storage problem Store the graph in a relational DBMS Develop a new database storage structure The indexing problem: Input: large, irregular data graph - PowerPoint PPT PresentationTRANSCRIPT
Chapter 10: XML
The world of XML
The Data
Semistructured data instance = a large graph
The indexing problem
• The storage problem– Store the graph in a relational DBMS– Develop a new database storage structure
• The indexing problem:– Input: large, irregular data graph– Output: index structure for evaluating (regular) path
expressions, e.g.bib.paper.author.firstname
XSet: a simple index for XML
• Part of the Ninja project at Berkeley• Example XML data:
XSet: a simple index for XML
Each node = a hashtableEach entry = list of pointers to data nodes (not shown)
• SELECT X FROM part.name X -yes• SELECT X FROM part.supplier.name X -yes• SELECT X FROM part.*.subpart.name X -maybe• SELECT X FROM *.supplier.name X -maybe
Region Algebras
• structured text = text with tags (like XML)• data = sequence of characters [c1c2c3 …]• region = interval in the text
– representation (x,y) = [cx,cx+1, … cy]– example: <section> … </section>
• region set = a set of regions– example all <section> regions (may be nested)
• region algebra = operators on region set, s1 op s2s1 op s2
• s1 intersect s2 = {r | r s1, r s2}• s1 included s2 = {r | rs1, r’ s2, r r’}• s1 including s2 = {r | r s1, r’ s2, r r’}• s1 parent s2 = {r | r s1, r’ s2, r is a parent of r’}• s1 child s2 = {r | r s1, r’ s2, r is child of r’}
Region Algebras
part.name name child (part child root)part.supplier.name name child (supplier child (part child root))*.supplier.name name child supplierpart.*.subpart.name name child (subpart included (part child root))
Region expressions correspond to simple XPath expressions
•s1 child s2 = {r | r s1, r’ s2, r is child of r’}
Efficient computation of Region Algebra Operators
Example: s1 included s2s1 = {(x1,x1'), (x2,x2'), …}s2 = {(y1,y1'), (y2,y2'), …}(i.e. assume each consists of disjoint regions)
Algorithm:if xi < yj then i := i + 1if xi' > yj' then j := j + 1otherwise: print (xi,xi'), do i := i + 1
Can do in sub-linear time when one region is very small
Storage structures for region algebras
• Every node is characterised by an integer pair (x,y)• This means we have a 2-d space• Any 2-d space data structure can be used
• If you use a (pre-order,post-order) numbering you get triangular filling of 2-d(to be discussed later)
Alternative mappings
• Mapping the structure to the relational world– The Edge approach– The Attribute approach– The Universal Table approach– The Normalized Universal approach– The Monet/XML approach– The Dataguide approach
• Mapping values– Separate value tables– Inlining
• Shredding
Dataguide approach
• Developed in the context of Lore, Lorel (Stanford Univ)• Predecessor of the Monet/XML model• Observation:
– queries in the graph-representation take a limited form– they are partial walks from the root to an object of
interest– this behaviour was stressed by the query language
Lorel, i.e. an SQL-based query language based on processing regular expressions
SELECT X
FROM (Bib.*.author).(lastname|firstname).Abiteboul X
DataGuides
Definitiongiven a semistructured data instance DB, a DataGuide for DB is a graph G s.t.:- every path in DB also occurs in G- every path in G occurs in DB- every path in G is unique
Dataguides
Example:
DataGuides
• Multiple DataGuides for the same data:
DataGuides
DefinitionLet w, w’ be two words (I.e word queries) and G a graphw G w’ if w(G) = w’(G)
DefinitionG is a strong dataguide for a database DB if G is the same as DB
Example:- G1 is a strong dataguide- G2 is not strong
person.project !DB dept.projectperson.project !G2 dept.project
DataGuides
• Constructing the strong DataGuide G:Nodes(G)={{root}}Edges(G)=while changes do
choose s in Nodes(G), a in Labelsadd s’={y|x in s, (x -a->y) in Edges(DB)} to Nodes(G)add (x -a->y) to Edges(G)
• Use hash table for Nodes(G)• This is precisely the powerset automaton construction.
DataGuides
• How large are the dataguides ?– if DB is a tree, then size(G) <= size(DB)
• why? answer: every node is in exactly one extent of G• here: dataguide = XSet
– How many nodes does the strong dataguide have for this DB ? 20 nodes (least common
multiple of 4 and 5)
Dataguides usually fail on data with cyclic schemas, like:
Monet XML approachMonet XML approach
Monet XML approach
Monet XML approach
Monet XML approach
Monet XML approach
• Querying the XML world
Querying and Transforming XML Data
• Standard XML querying/translation languages– XPath
• Simple language consisting of path expressions
– XSLT• Simple language designed for translation from XML to XML
and XML to HTML
– XQuery• An XML query language with a rich set of features
• Wide variety of other languages have been proposed, and some served as basis for the Xquery standard– XML-QL, Quilt, XQL, …
XPath
• XPath is used to address (select) parts of documents using path expressions
• A path expression is a sequence of steps separated by “/”– Think of file names in a directory hierarchy
• Result of path expression: set of values that along with their containing elements/attributes match the specified path
• E.g. /bank-2/customer/name evaluated on the bank-2 data we saw earlier returns <name>Joe</name><name>Mary</name>
• E.g. /bank-2/customer/name/text( ) returns the same names, but without the enclosing tags
XPath (Cont.)
• The initial “/” denotes root of the document (above the top-level tag)• Path expressions are evaluated left to right
– Each step operates on the set of instances produced by the previous step• Selection predicates may follow any step in a path, in [ ]
– E.g. /bank-2/account[balance > 400] • returns account elements with a balance value greater than 400• /bank-2/account[balance] returns account elements containing a balance
subelement• Attributes are accessed using “@”
– E.g. /bank-2/account[balance > 400]/@account-number• returns the account numbers of those accounts with balance > 400
– IDREF attributes are not dereferenced automatically (more on this later)
Functions in XPath
• XPath provides several functions– The function count() at the end of a path counts the number of
elements in the set generated by the path• E.g. /bank-2/account[customer/count() > 2]
– Returns accounts with > 2 customers– Also function for testing position (1, 2, ..) of node w.r.t. siblings
• Boolean connectives and and or and function not() can be used in predicates
• IDREFs can be referenced using function id()– id() can also be applied to sets of references such as IDREFS and
even to strings containing multiple references separated by blanks– E.g. /bank-2/account/id(@owner)
• returns all customers referred to from the owners attribute of account elements.
More XPath Features• Operator “|” used to implement union
– E.g. /bank-2/account/id(@owner) | /bank-2/loan/id(@borrower)• gives customers with either accounts or loans• However, “|” cannot be nested inside other operators.
• “//” can be used to skip multiple levels of nodes – E.g. /bank-2//name
• finds any name element anywhere under the /bank-2 element, regardless of the element in which it is contained.
• A step in the path can go to (13 variations in the standard):parents, siblings, ancestors and descendants
of the nodes generated by the previous step, not just to the children– “//”, described above, is a short from for specifying “all descendants”– “..” specifies the parent.
Pathfinder
• Xpath is essential for the implementation of an Xquery processor. It is strongly related to the data structures and its primitives.
• A state-of-the-art implementation is MonetDB/Pathfinder developed by Uni. Konstantz, Twente University, CWI
Pathfinder Uni Konstantz
Pathfinder
Pathfinder
Pathfinder
Pathfinder
Pathfinder
Pathfinder
pathfinder
Pathfinder
Staircase join
Staircase join
Pathfinder
Pathfinder
Pathfinder
Pathfinder
XQuery
XQuery
• XQuery is a general purpose query language for XML data
• Currently being standardized by the World Wide Web Consortium (W3C)– The textbook description is based on a March 2001 draft of the standard.
The final version may differ, but major features likely to stay unchanged.
• Alpha version of XQuery engine– Galax http://db.bell-labs.com/galax/– IPSI-IQ– Xpath visualized http://www.vbxml.com/xpathvisualizer/ – MonetDB/Pathfinder– Xhive
• XQuery is derived from the Quilt query language, which itself borrows from SQL, XQL and XML-QL
XQuery
• XQuery uses a for … let … where .. return … syntax
• for SQL from where SQL where return SQL select let allows temporary variables, and has no equivalent in SQL
• Variables make it possible to keep the state of processing around and severely complicates optimization
FLWR Syntax in XQuery
• For clause uses XPath expressions, and variables in the for- clause ranges over values in the set returned by Xpath
• XPath is used to address (select) parts of documents using path expressions• A path expression is a sequence of steps separated by “/”• Result of path expression: set of values that along with their containing
elements/attributes match the specified path
• E.g. /bank-2/customer/name evaluated on the bank-2 data we saw earlier returns
<name>Joe</name><name>Mary</name>
• E.g. /bank-2/customer/name/text( ) returns the same names, but without the enclosing tags
XPath
• XPath is used to address (select) parts of documents using path expressions• A path expression is a sequence of steps separated by “/”
– Think of file names in a directory hierarchy• Result of path expression: set of values that along with their containing
elements/attributes match the specified path
• E.g. /bank-2/customer/name evaluated on the bank-2 data we saw earlier returns <name>Joe</name><name>Mary</name>
• E.g. /bank-2/customer/name/text( ) returns the same names, but without the enclosing tags
XPath (Cont.)
• The initial “/” denotes root of the document (above the top-level tag)• Path expressions are evaluated left to right
– Each step operates on the set of instances produced by the previous step
• Selection predicates may follow any step in a path, in [ ]– E.g. /bank-2/account[balance > 400]
• returns account elements with a balance value greater than 400• /bank-2/account[balance] returns account elements containing a balance subelement
• Attributes are accessed using “@”– E.g. /bank-2/account[balance > 400]/@account-number
• returns the account numbers of those accounts with balance > 400– IDREF attributes are not dereferenced automatically (more on this later)
Functions in XPath
• XPath provides several functions– The function count() at the end of a path counts the number of elements
in the set generated by the path• E.g. /bank-2/account[customer/count() > 2]
– Returns accounts with > 2 customers– Also function for testing position (1, 2, ..) of node w.r.t. siblings
• Boolean connectives and and or and function not() can be used in predicates• IDREFs can be referenced using function id()
– id() can also be applied to sets of references such as IDREFS and even to strings containing multiple references separated by blanks
– E.g. /bank-2/account/id(@owner) • returns all customers referred to from the owners attribute of account
elements.
More XPath Features• Operator “|” used to implement union
– E.g. /bank-2/account/id(@owner) | /bank-2/loan/id(@borrower)• gives customers with either accounts or loans• However, “|” cannot be nested inside other operators.
• “//” can be used to skip multiple levels of nodes – E.g. /bank-2//name
• finds any name element anywhere under the /bank-2 element, regardless of the element in which it is contained.
• A step in the path can go to (13 variations in the standard):parents, siblings, ancestors and descendants
of the nodes generated by the previous step, not just to the children– “//”, described above, is a short from for specifying “all descendants”– “..” specifies the parent.
FLWR Syntax in XQuery
• Simple FLWR expression in XQuery – find all accounts with balance > 400, with each result enclosed in an
<account-number> .. </account-number> tag for $x in /bank-2/account let $acctno := $x/@account-number where $x/balance > 400 return <account-number> $acctno </account-number>
• Let clause not really needed in this query, and selection can be done In XPath. Query can be written as:
for $x in /bank-2/account[balance>400]return <account-number> $X/@account-number
</account-number>
Path Expressions and Functions
• Path expressions are used to bind variables in the for clause, but can also be used in other places– E.g. path expressions can be used in let clause, to bind variables to
results of path expressions• The function distinct( ) can be used to removed duplicates in path
expression results• The function document(name) returns root of named document
– E.g. document(“bank-2.xml”)/bank-2/account• Aggregate functions such as sum( ) and count( ) can be applied to path
expression results• XQuery does not support groupby, but the same effect can be got by
nested queries, with nested FLWR expressions within a return clause – More on nested queries later
Joins• Joins are specified in a manner very similar to SQL
for $b in /bank/account, $c in /bank/customer, $d in /bank/depositor
where $a/account-number = $d/account-number and $c/customer-name = $d/customer-name
return <cust-acct> $c $a </cust-acct>• The same query can be expressed with the selections specified as
XPath selections: for $a in /bank/account $c in /bank/customer
$d in /bank/depositor[ account-number =$a/account-number and customer-name = $c/customer-name] return <cust-acct> $c $a</cust-acct>
Changing Nesting Structure• The following query converts data from the flat structure for bank information
into the nested structure used in bank-1 <bank-1>
for $c in /bank/customer return
<customer> $c/* for $d in /bank/depositor[customer-name = $c/customer-name], $a in /bank/account[account-number=$d/account-number] return $a
</customer> </bank-1>
• $c/* denotes all the children of the node to which $c is bound, without the enclosing top-level tag
• Exercise for reader: write a nested query to find sum of accountbalances, grouped by branch.
XQuery Path Expressions
• $c/text() gives text content of an element without any subelements/tags
• XQuery path expressions support the “–>” operator for dereferencing IDREFs– Equivalent to the id( ) function of XPath, but simpler to
use– Can be applied to a set of IDREFs to get a set of results
Sorting in XQuery • Sortby clause can be used at the end of any expression. E.g. to return
customers sorted by name for $c in /bank/customer return <customer> $c/* </customer> sortby(name)
• Can sort at multiple levels of nesting (sort by customer-name, and by account-number within each customer)
<bank-1> for $c in /bank/customer return
<customer> $c/* for $d in /bank/depositor[customer-name=$c/customer-name], $a in /bank/account[account-number=$d/account-number]
return <account> $a/* </account> sortby(account-number)</customer> sortby(customer-name)
</bank-1>
Functions and Other XQuery Features
• User defined functions with the type system of XMLSchema function balances(xsd:string $c) returns list(xsd:numeric) { for $d in /bank/depositor[customer-name = $c], $a in /bank/account[account-number=$d/account-number] return $a/balance
}• Types are optional for function parameters and return values• Universal and existential quantification in where clause predicates
– some $e in path satisfies P – every $e in path satisfies P
• XQuery also supports If-then-else clauses
• Xmark http://www.xml-benchmark.org
• Used in most experiments on Xpath and Xquery evaluation
• Old figures on hand-compiled queries for the dataguide approach can be found in
• http://www.cwi.nl/~mk/xmarkArchive/Reports/Monet_report/monet_report.html
Xmark
XMark
Monet XML approachMonet XML approach
XMark
• Q1 Return the name of the person with ID ‘personal’
FOR $b IN /site/people/person[@id=‘personal’]RETURN $b/name/text()
Query optimizer challenges
• Mapping Xquery to a RBDMS should be able – to deal with ordered tables– to skip sub-documents– to perform dynamic type casting– to avoid unnecessary construction of string
intermediates– to recognize join-paths for fast access– to balance fragmentation and reconstruction cose
Xmark answers
• Q2: Return the initial increases of all open auctions.– This query evaluates the cost of array look-ups. Note that this
query may actually be harder to evaluate than it looks; especially relational back-ends may have to struggle with rather complex aggregations to select the bidder element with index 1.
FOR $b IN document("auction.xml")/site/open_auctions/open_auctionRETURN <increase> $b/bidder[1]/increase/text() </increase>
XMark
• Q3: Return the IDs of all open auctions whose current increase is at least twice as high as the initial increase.– This is a more complex application of index lookups. In the case
of a relational DBMS, the query can take advantage of set-valued aggregates on the index attribute to accelerate the execution.
FOR $b IN document("auction.xml")/site/open_auctions/open_auctionWHERE $b/bidder[0]/increase/text() *2 <= $b/bidder[last()]/increase/text()RETURN <increase first=$b/bidder[0]/increase/text() last=$b/bidder[last()]/increase/text()/>
Xmark result
• Q4: List the reserves of those open auctions where a certain person issued a bid before another person– This time, we stress the textual nature of XML documents by
querying the tag order in the source document
FOR $b IN document("auction.xml")/site/open_auctions/open_auctionWHERE $b/bidder/personref[id="person18829"] BEFORE $b/bidder/personref[id="person10487"]RETURN <history> $b/initial/text() </history>
Xmark answers
• Q5: How many sold items cost more than 40?– Strings are the generic data type in XML documents. Queries that
interpret strings will often need to cast strings to another data type that carries more semantics. This query challenges the DBMS in terms of the casting primitives it provides. Especially, if there is no additional schema information or just a DTD at hand, casts are likely to occur frequently.
COUNT (FOR $i document("auction.xml")/site/closed_auctions/closed_auction WHERE $i/price/text() >= 40 RETURN $i/price)
Xmark results
Q6: How many items are listed on all continents? • Regular path expressions are a fundamental building block of virtually
every query language for XML or semi-structured data. These queries investigate how well the query processor can optimize path expressions and prune traversals of irrelevant parts of the tree.
FOR $b IN document("auction.xml")/site/regionsRETURN COUNT ($b//item)
Xmark results
Q7: How many pieces of prose are in our database?• A good evaluation engine should realize that there is no need to
traverse the complete document tree to evaluate such expressions.Also note that the COUNT aggregation does not require a complete traversal of the tree. Just the cardinality of the respective relation is queried. Note that the tag <email> does not exist in the database document.
FOR $p IN document("auction.xml")/siteRETURN count($p//description) + count($p//annotation) +
count($p//email);
Xmark results
Q8: List the names of persons and the number of items they bought. (joins person, closed\_auction)
• References are an integral part of XML as they allow richer relationships than just hierarchical element structures. This query defines horizontal traversals with increasing complexity. A good query optimizer should take advantage of the cardinalities of the sets to be joined.
FOR $p IN document("auction.xml")/site/people/personLET $a := FOR $t IN
document("auction.xml")/site/closed_auctions/closed_auction WHERE $t/buyer/@person = $p/@id RETURN $tRETURN <item person=$p/name/text()> COUNT ($a) </item>
Xmark results
Q9: List the names of persons and the names of the items they bought in Europe. (joins person, closed_auction, item)
• References are an integral part of XML as they allow richer relationships than just hierarchical element structures. These queries define horizontal traversals with increasing complexity. A good query optimizer should take advantage of the cardinalities of the sets to be joined.
FOR $p IN document("auction.xml")/site/people/personLET $a := FOR $t IN
document("auction.xml")/site/closed_auctions/closed_auction LET $n := FOR $t2 IN document("auction.xml")/site/regions/europe/item WHERE $t/itemref/@item = $t2/@id RETURN $t2 WHERE $p/@id = $t/buyer/@person RETURN <item> $n/name/text() </item>RETURN <person name=$p/name/text()> $a </person>
Xmark results
Q10: List all persons according to their interest; use french markup in the result.
• Constructing new elements may put the storage engine under stress especially in the context of creating materialized document views. The following query reverses the structure of person records by grouping them according to the interest profile of a person. Large parts of the person records are repeatedly reconstructed. To avoid simple copying of the original database we translate the mark-up into french.
FOR $i IN DISTINCT document("auction.xml")/site/people/person/profile/interest/@category
LET $p := FOR $t IN document("auction.xml")/site/people/person WHERE $t/profile/interest/@category = $i RETURN <personne> <statistiques> <sexe> $t/gender/text() </sexe>, <age> $t/age/text() </age>, <education> $t/education/text()</education>, <revenu> $t/income/text() </revenu> </statistiques>,
<coordonnees> <nom> $t/name/text() </nom>, <rue> $t/street/text() </rue>, <ville> $t/city/text() </ville>, <pays> $t/country/text() </pays>, <reseau> <courrier> $t/email/text() </courrier>, <pagePerso> $t/homepage/text()</pagePerso> </reseau>, </coordonnees> <cartePaiement> $t/creditcard/text()</cartePaiement> </personne>RETURN <categorie> <id> $i </id>, $p </categorie>
Xmark results
Q11: For each person, list the number of items currently on sale whose price does not exceed 0.02\% of the person's income
• This query tests the database's ability to handle large (intermediate) results. This time, joins are on the basis of values. The difference between these queries and the reference chasing queries Q8 and Q9 is that references are specified in the DTD and may be optimized with logical OIDs for example. The two queries Q11 and Q12 cascade in the size of the result set and provide various optimization opportunities.
FOR $p IN document("auction.xml")/site/people/personLET $c := FOR $i IN
document("auction.xml")/site/open_auctions/open_auction/initial WHERE $p/profile/@income > (5000 * $i/text()) RETURN $iRETURN <items name=$p/profile/@income> COUNT ($c) </items>
Xmark results
12: For each richer-than-average person, list the number of items currently on sale whose price does not exceed 0.02% of the person's income
• This query tests the database's ability to handle large (intermediate) results. This time, joins are on the basis of values. The difference between these queries and the reference chasing queries Q8 and Q9 is that references are specified in the DTD and may be optimized with logical OIDs for example. The two queries Q11 and Q12 cascade in the size of the result set and provide various optimization opportunities. FOR $p IN document("auction.xml")/site/people/person
FOR $p IN document("auction.xml")/site/people/person LET $l := FOR $i IN
document("auction.xml")/site/open_auctions/open_auction/initial WHERE $p/profile/@income > (5000 * $i/text()) RETURN $iWHERE $p/profile/@income > 50000RETURN <items income=$p/profile/@income> COUNT ($l) </items>
Xmark results
Q13: List the names of items registered in Australia along with their descriptions.
• A key design for XML->DBMS mappings is to determine the fragmentation criteria. The complementary action is to reconstruct the original document from its broken-down representation. Query 13 tests for the ability of the database to reconstruct portions of theoriginal XML document.
• FOR $i IN document("auction.xml")/site/regions/australia/item RETURN <item name=$i/name/text()> $i/description </item>
Xmark results
Q14:Return the names of all items whose description contains the word `gold'.
• We continue to challenge the textual nature of XML documents; this time, we conduct a full-text search in the form of keyword search. Although full-text scanning could be studied in isolation we think that the interaction with structural mark-up is essential as the concepts are considered orthogonal; so query Q14 is restricted to a subset of the document by combining content and structure.
FOR $i IN document("auction.xml")/site//itemWHERE CONTAINS ($i/description,"gold")RETURN $i/name/text()
Xmark results
Q15: Print the keywords in emphasis in annotations of closed auctions.• We now try to quantify the costs of long path traversals that don't
include wildcards. We first descend deep into the tree (Query 15) and then return again (Query 16). Both queries only check for the existence of paths rather than selecting paths with predicates.
FOR $a IN document("auction.xml")/site/closed_auctions/closed_auction/annotation/description/parlist/listitem/parlist/listitem/text/emph/keyword/text()
RETURN <text> $a </text>
Xmark results
Q16: Return the IDs of those auctions that have one or more kweywords in emphasis.
FOR $a IN document("auction.xml")/site/closed_auctions/closed_auctionWHERE NOT EMPTY ($a/annotation/description/parlist/listitem/parlist/\ listitem/text/emph/keyword/text())RETURN <person id=$a/seller/@person />
Xmark results
Q17:Which persons don't have a homepage?• This is to test how well the query processors knows to deal with the
semi-structured aspect of XML data, especially elements that are declared optional in the DTD.
FOR $p IN document("auction.xml")/site/people/personWHERE EMPTY($p/homepage/text())RETURN <person name=$p/name/text()/>
Xmark results
Q18:Convert the currency of the reserve of all open auctions to another currency.
• This query puts the application of user defined functions (UDF) to the proof. In the XML world, UDFs are of particular importance because they allow the user to assign semantics to generic strings that go beyond type coercion.
FUNCTION CONVERT ($v){ RETURN 2.20371 * $v -- convert Dfl to Euros}
FOR $i IN document("auction.xml")/site/open_auctions/open_auction/RETURN CONVERT($i/reserve/text())
Query optimizer challenges
• Mapping Xquery to a RBDMS should be able – to deal with ordered tables– to skip sub-documents– to perform dynamic type casting– to avoid unnecessary construction of string
intermediates– to recognize join-paths for fast access– to balance fragmentation and reconstruction cose
Xmark results
Effect of loading 100Mb document into DBMS
Xmark results
Xmark results
Pathfinder/MonetDB 2004 implementation in seconds