hash tables and query execution march 1st, 2004. hash tables secondary storage hash tables are much...
DESCRIPTION
Assume 1 bucket (block) stores 2 keys + pointers h(e)=0 h(b)=h(f)=1 h(g)=2 h(a)=h(c)=3 Hash Table Example e b f g a cTRANSCRIPT
![Page 1: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/1.jpg)
Hash Tables and Query Execution
March 1st, 2004
![Page 2: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/2.jpg)
Hash Tables
• Secondary storage hash tables are much like main memory ones
• Recall basics:– There are n buckets– A hash function f(k) maps a key k to {0, 1, …, n-1}– Store in bucket f(k) a pointer to record with key k
• Secondary storage: bucket = block, use overflow blocks when needed
![Page 3: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/3.jpg)
• Assume 1 bucket (block) stores 2 keys + pointers
• h(e)=0• h(b)=h(f)=1• h(g)=2• h(a)=h(c)=3
Hash Table Example
e
b
f
g
a
c
0
1
2
3
![Page 4: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/4.jpg)
• Search for a:• Compute h(a)=3• Read bucket 3• 1 disk access
Searching in a Hash Table
e
b
f
g
a
c
0
1
2
3
![Page 5: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/5.jpg)
• Place in right bucket, if space• E.g. h(d)=2
Insertion in Hash Table
e
b
f
g
d
a
c
0
1
2
3
![Page 6: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/6.jpg)
• Create overflow block, if no space• E.g. h(k)=1
• More over-flow blocksmay be needed
Insertion in Hash Table
e
b
f
g
d
a
c
0
1
2
3
k
![Page 7: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/7.jpg)
Hash Table Performance
• Excellent, if no overflow blocks• Degrades considerably when number of
keys exceeds the number of buckets (I.e. many overflow blocks).
• Typically, we assume that a hash-lookup takes 1.2 I/Os.
![Page 8: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/8.jpg)
Where are we?
• File organizations: sorted, hashed, heaps.• Indexes: hash index, B+-tree• Indexes can be clustered or not.• Data can be stored in the index or not.
• Hence, when we access a relation, we can either scan or go through an index:– Called an access path.
![Page 9: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/9.jpg)
Current Issues in Indexing
• Multi-dimensional indexing:– how do we index regions in space?– Document collections?– Multi-dimensional sales data– How do we support nearest neighbor queries?
• Indexing is still a hot and unsolved problem!
![Page 10: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/10.jpg)
Query ExecutionQuery compiler
Execution engine
Index/record mgr.
Buffer manager
Storage manager
storage
User/Application
Queryupdate
Query executionplan
Record, indexrequests
Page commands
Read/writepages
![Page 11: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/11.jpg)
Query Execution Plans
Purchase Person
Buyer=name
City=‘seattle’ phone>’5430000’
buyer
(Simple Nested Loops)
SELECT S.snameFROM Purchase P, Person QWHERE P.buyer=Q.name AND Q.city=‘seattle’ AND Q.phone > ‘5430000’
Query Plan:• logical tree• implementation choice at every node• scheduling of operations.
(Table scan) (Index scan)
Some operators are from relationalalgebra, and others (e.g., scan, group)are not.
![Page 12: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/12.jpg)
The Leaves of the Plan: Scans
• Table scan: iterate through the records of the relation.
• Index scan: go to the index, from there get the records in the file (when would this be better?)
• Sorted scan: produce the relation in order. Implementation depends on relation size.
![Page 13: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/13.jpg)
How do we combine Operations?• The iterator model. Each operation is implemented
by 3 functions:– Open: sets up the data structures and performs
initializations– GetNext: returns the the next tuple of the result.– Close: ends the operations. Cleans up the data
structures.• Enables pipelining!• Contrast with data-driven materialize model.• Sometimes it’s the same (e.g., sorted scan).
![Page 14: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/14.jpg)
Implementing Relational Operations
• We will consider how to implement:– Selection ( ) Selects a subset of rows from relation.– Projection ( ) Deletes unwanted columns from
relation.– Join ( ) Allows us to combine two relations.– Set-difference Tuples in reln. 1, but not in reln. 2.– Union Tuples in reln. 1 and in reln. 2.– Aggregation (SUM, MIN, etc.) and GROUP BY
π
><
![Page 15: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/15.jpg)
Schema for Examples
• Purchase:– Each tuple is 40 bytes long, 100 tuples per page, 1000
pages (i.e., 100,000 tuples, 4MB for the entire relation).• Person:
– Each tuple is 50 bytes long, 80 tuples per page, 500 pages (i.e, 40,000 tuples, 2MB for the entire relation).
Purchase (buyer:string, seller: string, product: integer),
Person (name:string, city:string, phone: integer)
![Page 16: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/16.jpg)
Simple Selections• Of the form• With no index, unsorted: Must essentially scan the whole
relation; cost is M (#pages in R).• With an index on selection attribute: Use index to find
qualifying data entries, then retrieve corresponding data records. (Hash index useful only for equality selections.)
• Result size estimation: (Size of R) * reduction factor. More on this later.
SELECT *FROM Person RWHERE R.phone < ‘543%’
R attr valueop R. ( )
![Page 17: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/17.jpg)
Using an Index for Selections• Cost depends on #qualifying tuples, and clustering.
– Cost of finding qualifying data entries (typically small) plus cost of retrieving records.
– In example, assuming uniform distribution of phones, about 54% of tuples qualify (500 pages, 50000 tuples). With a clustered index, cost is little more than 500 I/Os; if unclustered, up to 50000 I/Os!
• Important refinement for unclustered indexes: 1. Find sort the rid’s of the qualifying data entries. 2. Fetch rids in order. This ensures that each data page is looked at
just once (though # of such pages likely to be higher than with clustering).
![Page 18: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/18.jpg)
Two Approaches to General Selections
• First approach: Find the most selective access path, retrieve tuples using it, and apply any remaining terms that don’t match the index:– Most selective access path: An index or file scan that
we estimate will require the fewest page I/Os.– Consider city=“seattle AND phone<“543%” :
• A hash index on city can be used; then, phone<“543%” must be checked for each retrieved tuple.
• Similarly, a b-tree index on phone could be used; city=“seattle” must then be checked.
![Page 19: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/19.jpg)
Intersection of Rids• Second approach
– Get sets of rids of data records using each matching index.
– Then intersect these sets of rids.– Retrieve the records and apply any remaining terms.
![Page 20: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/20.jpg)
Implementing Projection
• Two parts: (1) remove unwanted attributes, (2) remove duplicates from the result. • Refinements to duplicate removal:
– If an index on a relation contains all wanted attributes, then we can do an index-only scan.
– If the index contains a subset of the wanted attributes, you can remove duplicates locally.
SELECT DISTINCT R.name, R.phoneFROM Person R
![Page 21: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/21.jpg)
Equality Joins With One Join Column
• R S is a common operation. The cross product is too large. Hence, performing R S and then a selection is too inefficient.
• Assume: M pages in R, pR tuples per page, N pages in S, pS tuples per page.– In our examples, R is Person and S is Purchase.
• Cost metric: # of I/Os. We will ignore output costs.
SELECT *FROM Person R, Purchase SWHERE R.name=S.buyer
><×
![Page 22: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/22.jpg)
Discussion
• How would you implement join?
![Page 23: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/23.jpg)
Simple Nested Loops Join
• For each tuple in the outer relation R, we scan the entire inner relation S. – Cost: M + (pR * M) * N = 1000 + 100*1000*500 I/Os: 140
hours!• Page-oriented Nested Loops join: For each page of R, get
each page of S, and write out matching pairs of tuples <r, s>, where r is in R-page and S is in S-page.– Cost: M + M*N = 1000 + 1000*500 (1.4 hours)
For each tuple r in R dofor each tuple s in S do
if ri == sj then add <r, s> to result
![Page 24: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/24.jpg)
Index Nested Loops Join
• If there is an index on the join column of one relation (say S), can make it the inner. – Cost: M + ( (M*pR) * cost of finding matching S tuples)
• For each R tuple, cost of probing S index is about 1.2 for hash index, 2-4 for B+ tree. Cost of then finding S tuples depends on clustering.– Clustered index: 1 I/O (typical), unclustered: up to 1 I/O per
matching S tuple.
foreach tuple r in R doforeach tuple s in S where ri == sj do
add <r, s> to result
![Page 25: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/25.jpg)
Examples of Index Nested Loops• Hash-index on name of Person (as inner):
– Scan Purchase: 1000 page I/Os, 100*1000 tuples.– For each Person tuple: 1.2 I/Os to get data entry in index, plus 1
I/O to get (the exactly one) matching Person tuple. Total: 220,000 I/Os. (36 minutes)
• Hash-index on buyer of Purchase (as inner):– Scan Person: 500 page I/Os, 80*500 tuples.– For each Person tuple: 1.2 I/Os to find index page with data
entries, plus cost of retrieving matching Purchase tuples. Assuming uniform distribution, 2.5 purchases per buyer (100,000 / 40,000). Cost of retrieving them is 1 or 2.5 I/Os depending on clustering.
![Page 26: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/26.jpg)
Block Nested Loops Join• Use one page as an input buffer for scanning the
inner S, one page as the output buffer, and use all remaining pages to hold ``block’’ of outer R.– For each matching tuple r in R-block, s in S-page, add
<r, s> to result. Then read next R-block, scan S, etc.
. . .. . .
R & SHash table for block of R
(k < B-1 pages)
Input buffer for S Output buffer
. . .
Join Result
![Page 27: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/27.jpg)
Sort-Merge Join (R S)• Sort R and S on the join column, then scan them to
do a ``merge’’ on the join column.– Advance scan of R until current R-tuple >= current S
tuple, then advance scan of S until current S-tuple >= current R tuple; do this until current R tuple = current S tuple.
– At this point, all R tuples with same value and all S tuples with same value match; output <r, s> for all pairs of such tuples.
– Then resume scanning R and S.
><i=j
![Page 28: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/28.jpg)
Cost of Sort-Merge Join
• R is scanned once; each S group is scanned once per matching R tuple.
• Cost: M log M + N log N + (M+N)– The cost of scanning, M+N, could be M*N
(unlikely!)• With 35, 100 or 300 buffer pages, both
Person and Purchase can be sorted in 2 passes; total: 7500. (75 seconds).
![Page 29: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/29.jpg)
Hash-Join• Partition both relations
using hash fn h: R tuples in partition i will only match S tuples in partition i.
Read in a partition of R, hash it using h2 (<> h!). Scan matching partition of S, search for matches.
Partitionsof R & S
Input bufferfor Si
Hash table for partitionRi (k < B-1 pages)
B main memory buffersDisk
Output buffer
Disk
Join Result
hashfnh2
h2
B main memory buffers DiskDisk
Original Relation OUTPUT
2INPUT
1
hashfunction
h B-1
Partitions
1
2
B-1
. . .
![Page 30: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/30.jpg)
Cost of Hash-Join• In partitioning phase, read+write both relations; 2(M+N).
In matching phase, read both relations; M+N I/Os.• In our running example, this is a total of 4500 I/Os. (45
seconds!)• Sort-Merge Join vs. Hash Join:
– Given a minimum amount of memory both have a cost of 3(M+N) I/Os. Hash Join superior on this count if relation sizes differ greatly. Also, Hash Join shown to be highly parallelizable.
– Sort-Merge less sensitive to data skew; result is sorted.
![Page 31: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/31.jpg)
How are we doing?Nested loopjoin140 hours50 millionI/OsPage nestedloop1.4 hours500,000 I/OsIndex nestedloop36 minutes220,000 I/OsSort-mergejoin75 seconds7500 I/O’sHash join45 seconds4500 I/O’s
![Page 32: Hash Tables and Query Execution March 1st, 2004. Hash Tables Secondary storage hash tables are much like main memory ones Recall basics: –There are n](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b657f8b9ab0599afcd7/html5/thumbnails/32.jpg)
Double Pipelined Join (Tukwila)
Hash Join Partially pipelined: no output
until inner read Asymmetric (inner vs. outer) —
optimization requires source behavior knowledge
Double Pipelined Hash Join
Outputs data immediately Symmetric — requires less
source knowledge to optimize