1 introduction to query processing and query optimization techniques

77
1 Introduction to Query Processing and Query Optimization Techniques

Post on 21-Dec-2015

237 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: 1 Introduction to Query Processing and Query Optimization Techniques

1

Introduction to Query Processing and Query Optimization

Techniques

Page 2: 1 Introduction to Query Processing and Query Optimization Techniques

2

Introduction to Query Processing

Query optimization: The process of choosing a suitable execution

strategy for processing a query. Amongst all equivalent evaluation plans choose

the one with lowest cost. Cost is estimated using statistical information

from the database catalog e.g. number of tuples in each relation, size of tuples

Two internal representations of a query: Query Tree Query Graph

Page 3: 1 Introduction to Query Processing and Query Optimization Techniques

3

Basic Steps in Query Processing

Page 4: 1 Introduction to Query Processing and Query Optimization Techniques

4

Basic Steps in Query Processing

Select balance From account where balance < 2500 balance2500(balance(account))

is equivalent to

balance(balance2500(account))

Annotated relational algebra expression specifying detailed evaluation strategy is called an evaluation-plan. E.g., can use an index on balance to find accounts

with balance 2500, or can perform complete relation scan and discard

accounts with balance 2500

Page 5: 1 Introduction to Query Processing and Query Optimization Techniques

5

Measures of Query Cost

Cost is generally measured as total elapsed time for answering query. Many factors contribute to time cost disk accesses, CPU, or even network communication

Typically disk access is the predominant cost, and is also relatively easy to estimate.

For simplicity we just use number of block transfers from disk as the cost measure

Costs depends on the size of the buffer in main memory Having more memory reduces need for disk access

Page 6: 1 Introduction to Query Processing and Query Optimization Techniques

6

Translating SQL Queries into Relational Algebra

Query Block: The basic unit that can be translated into the

algebraic operators and optimized. A query block contains a single SELECT-FROM-

WHERE expression, as well as GROUP BY and HAVING clause if these are part of the block.

Nested queries within a query are identified as separate query blocks

Aggregate operators in SQL must be included in the extended algebra.

Page 7: 1 Introduction to Query Processing and Query Optimization Techniques

7

Translating SQL Queries into Relational Algebra

SELECT LNAME, FNAMEFROM EMPLOYEEWHERE SALARY > ( SELECT MAX (SALARY)

FROM EMPLOYEEWHERE DNO = 5);

SELECT MAX (SALARY)FROM EMPLOYEEWHERE DNO = 5

SELECT LNAME, FNAME

FROM EMPLOYEE

WHERE SALARY > C

πLNAME, FNAME (σSALARY>C(EMPLOYEE)) ℱMAX SALARY (σDNO=5 (EMPLOYEE))

Page 8: 1 Introduction to Query Processing and Query Optimization Techniques

8

Algorithms for External Sorting

Sorting is needed in Order by, join, union, intersection, distinct, …

For relations that fit in memory, techniques like quicksort can be used.

External sorting: Refers to sorting algorithms that are suitable for

large files of records stored on disk that do not fit entirely in main memory, such as most database files.

For relations that don’t fit in memory, external sort-merge is a good choice

Page 9: 1 Introduction to Query Processing and Query Optimization Techniques

9

External Sort-Merge

External sort-merge algorithm has two steps: Partial sort step, called runs. Merge step, merges the sorted runs.

Sort-Merge strategy: Starts by sorting small subfiles (runs) of the main file

and then merges the sorted runs, creating larger sorted subfiles that are merged in turn.

Sorting phase: Sorts nB pages at a time

nB = # of main memory pages buffer creates nR = b/nB initial sorted runs on disk

b = # of file blocks (pages) to be sorted

Sorting Cost = read b blocks + write b blocks = 2 b

Page 10: 1 Introduction to Query Processing and Query Optimization Techniques

10

External Sort-Merge

Example: nB = 5 blocks and file size b = 1024 blocks, then nR = (b/nB) = 1024/5 = 205 initial sorted runs

each of size 5 bocks (except the last run which will have 4 blocks)

nB = 2, b = 7, nR = b/nB = 4

run

Page 11: 1 Introduction to Query Processing and Query Optimization Techniques

11

External Sort-Merge

Sort Phase: creates nR sorted runs. i = 0 Repeat Read next nB blocks into RAMSort the in-memory blocksWrite sorted data to run file Ri

i = i + 1Until the end of the relation

nR = i

Page 12: 1 Introduction to Query Processing and Query Optimization Techniques

12

External Sort-Merge

Merging phase: The sorted runs are merged during one or more

passes. The degree of merging (dM) is the number of

runs that can be merged in each pass. dM = Min (nB-1, nR) nP = (logdM(nR))

nP: number of passes. In each pass,

One buffer block is needed to hold one block from each of the runs being merged, and

One block is needed for containing one block of the merged result.

Page 13: 1 Introduction to Query Processing and Query Optimization Techniques

13

External Sort-Merge

We assume (for now) that nR nB. Merge the runs (nR -way merge).

Use nR blocks of memory to buffer input runs, and 1 block to buffer output.

Merge nR Runs Step Read 1st block of each nR runs Ri into its buffer pageRepeat

Select 1st record (sort order) among nR buffer pagesWrite the record to the output buffer.

If the output buffer is full write it to diskDelete the record from its input buffer pageIf the buffer page becomes empty then read the next block (if

any) of the run into the bufferUntil all input buffer pages are empty

Page 14: 1 Introduction to Query Processing and Query Optimization Techniques

14

External Sort-Merge

Merge nB - 1 Runs Step If nR nB, several merge passes are required.

merge a group of contiguous nB - 1 runs using one buffer for output

A pass reduces the number of runs by a factor of nB - 1, and creates runs longer by the same factor. E.g. If nB = 11, and there are 90 runs, one pass

reduces the number of runs to 9, each 10 times the size of the initial runs

Repeated passes are performed till all runs have been merged into one

Page 15: 1 Introduction to Query Processing and Query Optimization Techniques

15

External Sort-Merge

Degree of merging (dM) # of runs that can be merged together in each pass =

min (nB - 1, nR) Number of passes nP = (logdM(nR))

In our example dM = 4 (four-way merging)

min (nB-1, nR) = min(5-1, 205) = 4 Number of passes nP = (logdM(nR)) = (log4(205)) = 4

First pass:– 205 initial sorted runs would be merged into 52 sorted runs

Second pass:– 52 sorted runs would be merged into 13

Third pass:– 13 sorted runs would be merged into 4

Fourth pass:– 4 sorted runs would be merged into 1

Page 16: 1 Introduction to Query Processing and Query Optimization Techniques

16

External Sort-Merge

Blocking factor bfr = 1 record, nB = 3, b = 12, nR = 4, dM = min(3-1, 4) = 2

Page 17: 1 Introduction to Query Processing and Query Optimization Techniques

17

External Sort-Merge

Page 18: 1 Introduction to Query Processing and Query Optimization Techniques

18

External Sort-Merge

External Sort-Merge: Cost Analysis Disk accesses for initial run creation (sort phase) as well as

in each merge pass is 2b reads every block once and writes it out once

Initial # of runs is nR = b/nB and # of runs decreases by a factor of nB - 1 in each merge pass, then the total # of merge passes is np = logdM

(nR)

In general, the cost performance of Merge-Sort is Cost = sort cost + merge cost Cost = 2b + 2b * np Cost = 2b + 2b * logdM nR

= 2b (logdM(nR) + 1)

Page 19: 1 Introduction to Query Processing and Query Optimization Techniques

19

External Sort-Merge

Page 20: 1 Introduction to Query Processing and Query Optimization Techniques

20

Catalog Information

File r: # of records in the file R: record size b: # of blocks in the file bfr: blocking factor

Index x: # of levels of a multilevel index bI1: # of first-level index blocks

Page 21: 1 Introduction to Query Processing and Query Optimization Techniques

21

Catalog Information

Attribute d: # of distinct values of an attribute sl (selectivity):

the ratio of the # of records satisfying the condition to the total # of records in the file.

s (selection cardinality) = sl * r average # of records that will satisfy an equality

condition on the attribute For a key attribute:

d = r, sl = 1/r, s = 1 For a nonkey attribute:

assuming that d distinct values are uniformly distributed among the records

the estimated sl = 1/d, s = r/d

Page 22: 1 Introduction to Query Processing and Query Optimization Techniques

22

File Scans

Types of scans File scan – search algorithms that locate and

retrieve records that fulfill a selection condition.

Index scan – search algorithms that use an indexselection condition must be on search-key of index.

Cost estimate C = # of disk blocks scanned

Page 23: 1 Introduction to Query Processing and Query Optimization Techniques

23

Algorithms for SELECT Operations

Implementing the SELECT Operation

Examples: (OP1): SSN='123456789' (EMP) (OP2): DNUMBER>5(DEPT) (OP3): DNO=5(EMP) (OP4): DNO=5 AND SALARY>30000 AND SEX=F(EMP) (OP5): ESSN=123456789 AND PNO=10(WORKS_ON)

Page 24: 1 Introduction to Query Processing and Query Optimization Techniques

24

Algorithms for Selection Operation

Search Methods for Simple Selection: S1 (linear search)

Retrieve every record in the file, and test whether its attribute values satisfy the selection condition.

If selection is on a nonkey attribute, C = b

If selection is equality on a key attribute, if record found, average cost C = b/2, else C = b

Page 25: 1 Introduction to Query Processing and Query Optimization Techniques

25

Algorithms for Selection Operation

S2 (binary search) Applicable if selection is an equality

comparison on the attribute on which file is ordered.

Assume that the blocks of a relation are stored contiguously

If selection is on a nonkey attribute: C = log2b: cost of locating the 1st tuple +

s/bfr - 1: # of blocks containing records that satisfy selection condition

If selection is equality on a key attribute: C = log2b, since s = 1, in this case

Page 26: 1 Introduction to Query Processing and Query Optimization Techniques

26

Selections Using Indices

S3 (primary or hash index on a key, equality) Retrieve a single record that satisfies the corresponding

equality condition If the selection condition involves an equality comparison on a

key attribute with a primary index (or a hash key), use the primary index (or the hash key) to retrieve the record.

Primary index: retrieve 1 more block than the # of index levels,C = x + 1:

Hash index:C = 1: for static or linear hashingC = 2: for extendable hashing

Page 27: 1 Introduction to Query Processing and Query Optimization Techniques

27

Selections Using Indices

S4 (primary index on a key, range selection) S4 Using a primary index to retrieve multiple

records: If the comparison condition is >, ≥, <, or ≤ on a key field with a

primary index, use the index to find the record satisfying the corresponding equality condition, then retrieve all subsequent records in the (ordered) file.

Assuming relation is sorted on AFor Av(r) use index to find 1st tuple = v and retrieve all

subsequent records.For Av(r) use index to find 1st tuple = v and retrieve all

preceding records. – OR just scan relation sequentially till 1st tuple v; do not use index

with average cost C = b/2

Average cost C = x + b/2

Page 28: 1 Introduction to Query Processing and Query Optimization Techniques

28

Selections Using Indices

S5 (clustered index on nonkey, equality) Retrieve multiple records. Records will be on consecutive blocks C = x + s/bfr

# of blocks containing records that satisfy selection condition

Page 29: 1 Introduction to Query Processing and Query Optimization Techniques

29

Selections Using Indices

S6-1 (secondary index B+-tree, equality) Retrieve a single record if the search-key is a

candidate key, C = x + 1

Retrieve multiple records if search-key is not a candidate key,C = x + s Can be very expensive!. Each record may be on a

different block , one block access for each retrieved record

Page 30: 1 Introduction to Query Processing and Query Optimization Techniques

30

Selections Using Indices

S6-2 (secondary index B+-tree, comparison) For Av(r) use index to find 1st index entry = v

and scan index sequentially from there, to find pointers to records.

For Av(r) just scan leaf pages of index finding pointers to records, till first entry v

If ½ records are assumed to satisfy the condition, then ½ first-level index blocks are accessed, plus ½ the file records via the indexC = x + bI1/2 + r/2

Page 31: 1 Introduction to Query Processing and Query Optimization Techniques

31

Complex Selections: 12…n(r)

S7 (conjunctive selection using one index) Select i and algorithms S1 through S6 that

results in the least cost for i(r).

Test other conditions on tuple after fetching it into memory buffer.

Cost of the algorithms chosen.

Page 32: 1 Introduction to Query Processing and Query Optimization Techniques

32

Complex Selections: 12…n(r)

S8 (conjunctive selection using composite index). If two or more attributes are involved in equality

conditions in the conjunctive condition and a composite index (or hash structure) exists on the combined field, we can use the index directly.

Use appropriate composite index if available using one the algorithms S3 (primary index), S5, or S6 (B+-tree, equality).

Page 33: 1 Introduction to Query Processing and Query Optimization Techniques

33

Complex Selections: 12…n(r)

S9 (conjunctive selection by intersection of record pointers) Requires indices with record pointers. Use corresponding index for each condition, and

take intersection of all the obtained sets of record pointers, then fetch records from file

If some conditions do not have appropriate indices, apply test in memory.

Cost is the sum of the costs of the individual index scan plus the cost of retrieving records from disk.

Page 34: 1 Introduction to Query Processing and Query Optimization Techniques

34

Complex Selections: 12… n(r)

S10 (disjunctive selection by union of identifiers) Applicable if all conditions have available indices.

Otherwise use linear scan. Use corresponding index for each condition, and

take union of all the obtained sets of record pointers. Then fetch records from file

READ “Examples of Using the Cost Functionst”

page 569--570.page 694-695

Page 35: 1 Introduction to Query Processing and Query Optimization Techniques

35

Duplicate Elimination

Duplicate elimination can be implemented via hashing or sorting. On sorting, duplicates will come adjacent to each

other, and all but one set of duplicates can be deleted.

Optimization: duplicates can be deleted during run generation as well as at intermediate merge steps in external sort-merge.

Cost is the same as the cost of sorting

Hashing is similar – duplicates will come into the same bucket.

Page 36: 1 Introduction to Query Processing and Query Optimization Techniques

36

Algorithms for PROJECT Operation

Algorithm for PROJECT operations (Figure 15.3b)

<attribute list>(R)

1. If <attribute list> has a key of relation R, extract all tuples from R with only the values for the attributes in <attribute list>.

2. If <attribute list> does NOT include a key of relation R, duplicated tuples must be removed from the results.

Page 37: 1 Introduction to Query Processing and Query Optimization Techniques

37

Algorithms for SET Operations

Algorithm for SET operations Set operations:

UNION, INTERSECTION, SET DIFFERENCE and CARTESIAN PRODUCT

CARTESIAN PRODUCT of relations R and S include all possible combinations of records from R and S. The attribute of the result include all attributes of R and S.

Cost analysis of CARTESIAN PRODUCT If R has n records and j attributes and S has m records

and k attributes, the result relation will have n*m records and j+k attributes.

CARTESIAN PRODUCT operation is very expensive and should be avoided if possible.

Page 38: 1 Introduction to Query Processing and Query Optimization Techniques

38

Set Operations

R S: (See Figure 18.3c) 1. Sort the two relations on the same attributes. 2. Scan and merge both sorted files concurrently,

whenever the same tuple exists in both relations, only one is kept in the merged results.

R S: (See Figure 18.3d) 1. Sort the two relations on the same attributes. 2. Scan and merge both sorted files concurrently,

keep in the merged results only those tuples that appear in both relations.

R – S: (See Figure 18.3e) keep in the merged results only those tuples that

appear in relation R but not in relation S.

Page 39: 1 Introduction to Query Processing and Query Optimization Techniques

39

b. Projectc. Union

Page 40: 1 Introduction to Query Processing and Query Optimization Techniques

40

d. Intersection e. Different

Page 41: 1 Introduction to Query Processing and Query Optimization Techniques

41

Aggregate Operations

The aggregate operations MIN, MAX, COUNT, AVERAGE, and SUM can be computed by scanning the whole records (the worst case)

If index exists on the attribute of MAX , MIN operation, then these operations can be done in a much more efficient way: select max/min (salary) from employee If an (ascending) index on SALARY exists for

the employee relation, then the optimizer could decide on traversing the index for the largest/smallest value, which would entail following the right/left most pointer in each index node from the root to a leaf.

Page 42: 1 Introduction to Query Processing and Query Optimization Techniques

42

Aggregate Operations

SUM, COUNT and AVG For a dense index (each record has one index entry):

Apply the associated computation to the values in the index. For a non-dense index:

Actual number of records associated with each index entry must be accounted for

With GROUP BY: the aggregate operator must be applied separately to each group of tuples. Use sorting or hashing on the group attributes to partition the

file into the appropriate groups; Computes the aggregate function for the tuples in each

group. What if we have Clustering index on the grouping attributes?

Page 43: 1 Introduction to Query Processing and Query Optimization Techniques

43

Combining Operations using Pipelining

Motivation A query is mapped into a sequence of operations. Each execution of an operation produces a

temporary result (Materialization). Generating & saving temporary files on disk is

time consuming.

Alternative: Avoid constructing temporary results as much as

possible. Pipeline the data through multiple operations

pass the result of a previous operator to the next without waiting to complete the previous operation.

Page 44: 1 Introduction to Query Processing and Query Optimization Techniques

44

Materialization

Materialized evaluation: evaluate one operation at a time, starting

at the lowest-level. Use intermediate results materialized into

temporary relations to evaluate next-level operations. )(2500 accountbalance

Page 45: 1 Introduction to Query Processing and Query Optimization Techniques

45

Pipelining

Pipelined evaluation evaluate several operations simultaneously,

passing the results of one operation on to the next. E.g., in previous expression tree, don’t store result

of

instead, pass tuples directly to the join. Similarly, don’t store result of join, pass tuples

directly to projection.

Much cheaper than materialization: no need to store a temporary relation to disk.

Pipelining may not always be possible e.g., sort, hash-join.

)(2500 accountbalance

Page 46: 1 Introduction to Query Processing and Query Optimization Techniques

46

Using Heuristics in Query Optimization

Process for heuristics optimization1. The parser of a high-level query generates an initial

internal representation;2. Apply heuristics rules to optimize the internal

representation.3. A query execution plan is generated to execute groups of

operations based on the access paths available on the files involved in the query.

The main heuristic is to apply first the operations that reduce the size of intermediate results. E.g., Apply SELECT and PROJECT operations before

applying the JOIN or other binary operations.

Page 47: 1 Introduction to Query Processing and Query Optimization Techniques

47

Using Heuristics in Query Optimization

Query tree: A tree data structure that corresponds to a relational

algebra expression. It represents the input relations of the query as leaf nodes of the tree, and represents the relational algebra operations as internal nodes.

An execution of the query tree consists of executing an internal node operation whenever its operands are available and then replacing that internal node by the relation that results from executing the operation.

Query graph: A graph data structure that corresponds to a relational

calculus expression. It does not indicate an order on which operations to perform first. There is only a single graph corresponding to each query.

Page 48: 1 Introduction to Query Processing and Query Optimization Techniques

48

Using Heuristics in Query Optimization

Example: For every project located in ‘Stafford’, retrieve the project

number, the controlling department number and the department manager’s last name, address and birthdate.

Relation algebra:

PNUMBER, DNUM, LNAME, ADDRESS, BDATE (((PLOCATION=‘STAFFORD’(PROJECT))

DNUM=DNUMBER (DEPARTMENT))

MGRSSN=SSN (EMPLOYEE))

SQL query:Q2: SELECT P.NUMBER,P.DNUM,E.LNAME, E.ADDRESS,

E.BDATE FROM PROJECT AS P,DEPARTMENT AS D, EMPLOYEE AS E WHERE P.DNUM=D.DNUMBER AND D.MGRSSN=E.SSN AND P.PLOCATION=‘STAFFORD’;

Page 49: 1 Introduction to Query Processing and Query Optimization Techniques

49

Using Heuristics in Query Optimization

Page 50: 1 Introduction to Query Processing and Query Optimization Techniques

50

Using Heuristics in Query Optimization

Page 51: 1 Introduction to Query Processing and Query Optimization Techniques

51

Using Heuristics in Query Optimization

Heuristic Optimization of Query Trees: The same query could correspond to many

different relational algebra expressions — and hence many different query trees.

The task of heuristic optimization of query trees is to find a final query tree that is efficient to execute.

Example:SELECT LNAME FROM EMPLOYEE, WORKS_ON,

PROJECT WHERE PNAME = ‘AQUARIUS’ AND PNMUBER=PNO AND ESSN=SSN AND BDATE > ‘1957-12-31’;

Page 52: 1 Introduction to Query Processing and Query Optimization Techniques

52

Using Heuristics

in Query Optimization

Page 53: 1 Introduction to Query Processing and Query Optimization Techniques

53

Using Heuristics in Query

Optimization

Page 54: 1 Introduction to Query Processing and Query Optimization Techniques

54

Using Heuristics in Query

Optimization

Page 55: 1 Introduction to Query Processing and Query Optimization Techniques

55

Transformation Rules for RA Operations

1. Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of individual operations:

c1 AND c2 AND ... AND cn(R) c1(c2(...(cn(R))...))

2. Commutativity of :The operation is commutative:

c1(c2(R)) c2(c1(R))

3. Cascade of :In a cascade (sequence) of operations, all but the last one can be ignored:

List1(List2(...(Listn(R))...)) = List1(R)

4. Commuting with :If the selection condition c involves only the attributes A1, ..., An in the projection list, the two operations can be commuted:

A1, A2, ..., An(c(R)) = c(A1, A2, ..., An(R))

Page 56: 1 Introduction to Query Processing and Query Optimization Techniques

56

Transformation Rules for RA Operations

5. Commutativity of ⋈ (or ):The ⋈ operation is commutative as is the operation:

R ⋈c S = S ⋈c R; R S = S R

6. Commuting with ⋈ (or ):If all the attributes in the selection condition c involve only the attributes of one of the relations being joined—say, R—the two operations can be commuted as follows:

c(R ⋈ S) (c(R)) ⋈ S

Alternatively, if the selection condition c can be written as (c1 and c2), where condition c1 involves only the attributes of R and condition c2 involves only the attributes of S, the operations commute as follows:

c(R ⋈ S) (c1(R)) ⋈ (c2(S))

Page 57: 1 Introduction to Query Processing and Query Optimization Techniques

57

Transformation Rules for RA Operations

7. Commuting with ⋈ (or ):Suppose that the projection list is L = {A1, ..., An, B1, ..., Bm}, where A1, ..., An are attributes of R and B1, ..., Bm are attributes of S. If the join condition c involves only attributes in L, the two operations can be commuted as follows:

L(R ⋈c S) (A1, ..., An(R)) ⋈c (B1, ..., Bm(S))

If the join condition c contains additional attributes not in L, these must be added to the projection list, and a final operation is needed.

Page 58: 1 Introduction to Query Processing and Query Optimization Techniques

58

Transformation Rules for RA Operations

8. Commutativity of set operations:The set operations and are commutative but – is not.

9. Associativity of ⋈, , , and :These four operations are individually associative; that is, if stands for any one of these 4 operations (throughout the expression), we have

(R S) T R (S T)

10.Commuting with set operations:The operation commutes with , , and –. If stands for any one of these 3 operations, we have

c(R S) (c(R)) (c(S))

Page 59: 1 Introduction to Query Processing and Query Optimization Techniques

59

Transformation Rules for RA Operations

11.The operation commutes with . L(R υ S) (L(R)) υ (L(S))

12.Converting a (sequence into ⋈:If the condition c of a thatfollows a corresponds to a join condition, convert the (sequence into a ⋈ as

(c(R S)) = (R ⋈c S)

Page 60: 1 Introduction to Query Processing and Query Optimization Techniques

60

Heuristic Algebraic Optimization Algorithm

1. Using rule 1, break up any select operations with conjunctive conditions into a cascade of select operations.

σc1 AND c2 AND … AND cn(R) ≡ σc1(σc2(…(σcn(R))…))2. Using rules 2, 4, 6, and 10 concerning the

commutativity of select with other operations, move each select operation as far down the query tree as is permitted by the attributes involved in the select condition.

3. Using rule 9 concerning associativity of binary operations, rearrange the leaf nodes of the tree: Position the leaf node relation with the most restrictive

σ operations so they are executed first, Make sure that the ordering of leaf nodes

does not cause CARTESIAN PRODUCT operations

Page 61: 1 Introduction to Query Processing and Query Optimization Techniques

61

Heuristic Algebraic Optimization Algorithm

4. Using Rule 12, combine a cartesian product operation with a subsequent select operation in the tree into a join operation.

Combine a with a subsequent σ in the tree into a ⋈

5. Using rules 3, 4, 7, and 11 concerning the cascading of project and the commuting of project with other operations, break down and move lists of projection attributes down the tree as far as possible by creating new project operations as needed.

6. Identify subtrees that represent groups of operations that can be executed by a single algorithm.

Page 62: 1 Introduction to Query Processing and Query Optimization Techniques

62

Summary of Heuristics for Algebraic Optimization

The main heuristic is to apply first the operations that reduce the size of intermediate results.

Perform select operations as early as possible to reduce the number of tuples and perform project operations as early as possible to reduce the number of attributes. (This is done by moving select and project operations as far down the tree as possible.)

The select and join operations that are most restrictive should be executed before other similar operations. (This is done by reordering the leaf nodes of the tree among themselves and adjusting the rest of the tree appropriately.)

Page 63: 1 Introduction to Query Processing and Query Optimization Techniques

63

Query Execution Plans

An execution plan for a relational algebra query consists of a combination of the relational algebra query tree and information about the access methods to be used for each relation as well as the methods to be used in computing the relational operators stored in the tree.

Materialized evaluation: the result of an operation is stored as a temporary relation.

Pipelined evaluation: as the result of an operator is produced, it is forwarded to the next operator in sequence.

Page 64: 1 Introduction to Query Processing and Query Optimization Techniques

64

Using Selectivity and Cost Estimates in Query Optimization

Cost-based query optimization: Estimate and compare the costs of executing a

query using different execution strategies and choose the strategy with the lowest cost estimate.

(Compare to heuristic query optimization)

Issues Cost function Number of execution strategies to be

considered

Page 65: 1 Introduction to Query Processing and Query Optimization Techniques

65

Using Selectivity and Cost Estimates in Query Optimization

Cost Components for Query Execution1. Access cost to secondary storage2. Storage cost3. Computation cost4. Memory usage cost5. Communication cost

Note: Different database systems may focus on different cost components.

Page 66: 1 Introduction to Query Processing and Query Optimization Techniques

66

Using Selectivity and Cost Estimates in Query Optimization

Catalog Information Used in Cost Functions Information about the size of a file

number of records (tuples) (r), record size (R), number of blocks (b) blocking factor (bfr)

Information about indexes and indexing attributes of a file Number of levels (x) of each multilevel index Number of first-level index blocks (bI1) Number of distinct values (d) of an attribute Selectivity (sl) of an attribute Selection cardinality (s) of an attribute. (s = sl * r)

Page 67: 1 Introduction to Query Processing and Query Optimization Techniques

67

Using Selectivity and Cost Estimates in Query Optimization

Examples of Cost Functions for SELECT S1. Linear search (brute force) approach

CS1a = b; For an equality condition on a key, CS1a = (b/2) if the

record is found; otherwise CS1a = b. S2. Binary search:

CS2 = log2b + (s/bfr) –1 For an equality condition on a unique (key) attribute,

CS2 =log2b S3. Using a primary index (S3a) or hash key (S3b) to

retrieve a single record CS3a = x + 1; CS3b = 1 for static or linear hashing; CS3b = 1 for extendible hashing;

Page 68: 1 Introduction to Query Processing and Query Optimization Techniques

68

Using Selectivity and Cost Estimates in Query Optimization

Examples of Cost Functions for SELECT (contd.) S4. Using an ordering index to retrieve multiple records:

For the comparison condition on a key field with an ordering index, CS4 = x + (b/2)

S5. Using a clustering index to retrieve multiple records: CS5 = x + ┌ (s/bfr) ┐

S6. Using a secondary (B+-tree) index: For an equality comparison, CS6a = x + s; For an comparison condition such as >, <, >=, or <=, CS6a = x + (bI1/2) + (r/2)

Page 69: 1 Introduction to Query Processing and Query Optimization Techniques

69

Using Selectivity and Cost Estimates in Query Optimization

Examples of Cost Functions for SELECT (contd.) S7. Conjunctive selection:

Use either S1 or one of the methods S2 to S6 to solve. For the latter case, use one condition to retrieve the

records and then check in the memory buffer whether each retrieved record satisfies the remaining conditions in the conjunction.

S8. Conjunctive selection using a composite index: Same as S3a, S5 or S6a, depending on the type of

index.

Page 70: 1 Introduction to Query Processing and Query Optimization Techniques

70

10. Semantic Query Optimization

Semantic Query Optimization: Uses constraints specified on the database schema in order to

modify one query into another query that is more efficient to execute.

Consider the following SQL query,SELECT E.LNAME, M.LNAMEFROM EMPLOYEE E MWHEREE.SUPERSSN=M.SSN AND E.SALARY>M.SALARY

Explanation: Suppose that we had a constraint on the database schema that

stated that no employee can earn more than his or her direct supervisor. If the semantic query optimizer checks for the existence of this constraint, it need not execute the query at all because it knows that the result of the query will be empty. Techniques known as theorem proving can be used for this purpose.

Page 71: 1 Introduction to Query Processing and Query Optimization Techniques

71

Example (1)

An un-optimized relational algebra expression: Name ( GPA 3.5 and Title = 'Ada Programming Language’ and

Students.SSN = Enrollment.SSN and Enrollment.Course_no =

Courses.Course_no (Students Enrollment Courses))

Page 72: 1 Introduction to Query Processing and Query Optimization Techniques

72

Example (2)

Initial query tree:

Name

GPA 3.5 and Title = 'Ada Programming Language’

and Students.SSN = Enrollment.SSN

and Enrollment.Course_no = Courses.Course_no

Courses

EnrollmentStudents

Page 73: 1 Introduction to Query Processing and Query Optimization Techniques

73

Example (3)

Perform selections as early as possible.

Name

Title = 'Ada Programming

Language’

CoursesEnrollment

Students

Enrollment.Course_no = Courses.Course_no

Students.SSN = Enrollment.SSN

GPA 3.5

Page 74: 1 Introduction to Query Processing and Query Optimization Techniques

74

Example (4)

Replace Cartesian products by joins.

Name

Title = 'Ada Programming Language’

CoursesEnrollment

Students

GPA 3.5

Page 75: 1 Introduction to Query Processing and Query Optimization Techniques

75

Example (5)

Perform more restrictive joins first.

Name

Title = 'Ada Programming

Language’

Courses

EnrollmentStudents

GPA 3.5

Page 76: 1 Introduction to Query Processing and Query Optimization Techniques

76

Example (6)

Project out useless attributes early.

Name

Title = 'Ada Programming

Language’

Courses

EnrollmentStudents

GPA 3.5

SSN, Name

SSN, Course_no Course_no

SSN

Page 77: 1 Introduction to Query Processing and Query Optimization Techniques

77

Example (7)

The optimized algebra expression is: Name (( SSN, Name ( GPA 3.5 (Students))) ((SSN,

Course_no(Enrollments)) (Course_no ( Title = 'Ada

Programming Language’ (Courses)))))

Projections and selections on the same relation are usually performed using the same scan of the relation.