© ibm corporation 2007 1 patrick bossman ibm silicon valley lab tuning with optimization service...

© IBM Corporation 2007

1

Patrick Bossman

IBM Silicon Valley Lab

Tuning with Optimization Service Center

Part II

®

Columbia, MDSeptember 12, 2007


2

Agenda

• Query tuning with OSC– Statistics Advisor

– Query annotation

– Query report

– Access plan graph

– Visual plan hint

• Demo as time permits


3

Tuning a query

• Steps to tuning a query– Identify statement

• Collect information regarding SQL– Review SQL

– Physical design of referenced objects

– Statistics

– Access path

– Mentally compute, is access path efficient?

– Could we do better?


4

Identify query

• Choose a query from an existing workload– Selecting query from a workload

• Use Tune Query input options– Copy / paste / manually type within “Tune a query”

option

– Grab static SQL from the catalog

– Choose from statement cache

– Retrieve from QMF


5

Tune query from workload

• Right click on selected query

OR

• Select query Query Tools


6

Select query from workload…


7

Press Query Tools button


8

Tune query input options

• Manual input (type, copy / paste)• Use catalog search options (static SQL)• Select SQL from snap statement cache• Select from QMF


9

Welcome page


10

Tune single query


11

Choose identify target query…


12

Query options

• Top of window, select query source– Top of window varies based on selected source

• Bottom of window – Show selected SQL

– Tuning options


13

Query selection window


14

Query source


15

Query text


16

Query tuning options


17

Query menu


18

Query menu


19

Query menu


20

Choose query source…


21

Text input…


22

Source: Statement cache…

• Improved statement cache interface!– Use predefined cache views sorted in popular use

sequence– Define and save your own display of the cache


23

Other cache improvements

• Design your own cache display view– Choose filter criteria

– Choose how to sort rows retrieved from cache

– Choose which columns to display

– After rows retrieved, you can re-sort the rows displayed on the screen also

• Vastly improved statement cache interface– One-time setup, repeating use


24Cache new view optionsFilter…


25Cache new view optionssort on…


26Cache new view optionsdisplay columns…


27

Source section of window

• Double left click to see SQL text• Left click on row to select SQL• Left-click on column header to sort retrieved

records by column


28

Source: Catalog

• Build catalog search criteria as a view


29

Package filter options• Package filter options

– Collection id, name, owner, …

– Can use equals, like, in, etc.


30

Package filter options


31

Plan filter options• Plan filter options

– Plan name, plan creator, etc.

– Can use equals, like, in, etc.


32

Plan filter options


33

Cost & Object filter options• Cost and Object filters

– Filters SQL within package / plan filter– Requires the static SQL be bound with explain yes.– DSN_STATEMNT_TABLE must already be populated.– Show only SQL with PROCSU > …– Show SQL which uses table…– Show SQL which uses index…

• Choice to return rows which qualify for ANY or ALL of the cost / object filter conditions


34

Cost & Object filter options


35

Access path filters• Access path filters

– Filters SQL within package / plan filter

– Requires the static SQL be bound with explain yes.

– PLAN_TABLE must already be populated.

• Show SQL which performs…– Tablespace scan

– Sort

– Non-matching index scan

– List prefetch

– Outer join

– ….


36

Access path filter options


37

Catalog SQL list

• Left click on row to select…


38

Source QMF / QMF HPO


39

QMF search options

• Searching for Bob’s QMF queries…


40

Source: File


41

Source: Categorized SQL


42

Source: Plan Table


43

Plan table sort options


44

Plan table display options


45

Plan table display options

• Plan table comments– OSC will map plan_table to SYSPACKSTMT / SYSSTMT– Explain row selected must map to currently bound version of

SQL to retrieve SQL statement text– Ease of use TIP: Make sure you also sort by BIND_TIME

DESC if you have history


46

After you’ve selected a query…

• Query options– Run SQL

– Categorize SQL

• Advisors– Run statistics advisor

• Tools– Query annotation

– Query graph

– Visual Plan Hint

– Query report


47

Query menu


48

Categorize query

• Categorize and store SQL you work with


49

Categorize query


50

Run query(just click run…)


51

Run query output


52

Statistics Advisor

• “Why” statistics advisor outside scope of this session – Covered somewhat in part I

– Covered as session in and of itself.

• Run Statistics Advisor• Statistics Advisor Options


53

FYI - Switching SQL…

SELECT X.DISCOUNT

, SUM(X.QUANTITY) AS SUMQ

, AVG(X.QUANTITY * X.PRICE) AS AVGP

FROM LITEM_UIV_NOGBY AS X

WHERE X.QUANTITY <= 5

AND X.DISCOUNT > 0.08

AND ORDERKEY >30000000

GROUP BY X.DISCOUNT


54

Generate targeted statistics…

• Identify interesting objects– Tables– Indexes

• Identify interesting columns, column groups– Predicate analysis

• Which column (groups) have predicates?

• Which column (groups) are indexed?

• Which statistics should I collect?– Column statistics– Correlation statistics– Frequencies– Histograms

– Statistic needed depends on predicate types (range, equal)


55

Run Statistics Advisor


56

Run Statistics Advisor


57

Statistics Advisor Options

• General options– Enable / disable certain analysis options

– Histogram predicate types

• Typical default list– In addition to DDL defined defaults

• Conflict thresholds


58

Statistics Advisor options


59

SA general options


60

SA general options


61

RUNSTATS collection parameters


62

Report type


63

Defaults and conflicts

• Typical default list– Default values typically skewed– If SA observes a “typical default” or a DDL defined default

value within a predicate, it will trigger frequent value collection

• Conflict thresholds– Statistics Advisor looks for inconsistent statistics– Could be inconsistent collection strategy

• Collect table / index statistics at different intervals• Collect statistics on indexes on same columns at different times

– Minor inconsistencies OK and can naturally occur even with consistent collection strategies

• Index cardinalities more accurate than column collected cardinalities (simulation)


64

Predefined defaults


65

Conflict threshold (1)


66

Conflict threshold (2)


67

Query Annotation


68

Query annotation

• Original SQL– Format SQL

– Annotate SQL with statistics

– Helpful highlighting

– Expand / collapse query sections


69

Original SQL

SELECT S_SUPPKEY, S_NAME , SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)) AS REVENUE FROM SYSADM.ORDER , SYSADM.LINEITEM , SYSADM.SUPPLIER WHERE L_ORDERKEY = O_ORDERKEY AND S_SUPPKEY = L_SUPPKEY AND O_ORDERDATE >= DATE('1993-10-01') AND O_ORDERDATE < DATE('1993-10-01') + 3 MONTH AND L_RETURNFLAG = 'R' AND S_NATIONKEY IN (1,2,3,4,5) AND S_SUPPKEY < 10000 GROUP BY S_SUPPKEY, S_NAME HAVING SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)) > 1000000 ORDER BY REVENUE DESC QUERYNO 3


70

FormattedSELECT SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAME , SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 -

SYSADM.LINEITEM.L_DISCOUNT ) ) AS REVENUEFROM SYSADM.SUPPLIER , SYSADM.ORDER , SYSADM.LINEITEMWHERE ( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 )GROUP BY SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAMEHAVING SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 -

SYSADM.LINEITEM.L_DISCOUNT ) ) > 1000000ORDER BY REVENUE DESC


71

AnnotatedSELECT SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAME , SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 - SYSADM.LINEITEM.L_DISCOUNT ) ) AS REVENUEFROM SYSADM.SUPPLIER CARDF=10,000 QUALIFIED_ROWS=1,526.2466 NPAGESF=402 , SYSADM.ORDER CARDF=1,500,000 QUALIFIED_ROWS=57,114.59 NPAGESF=42,417 , SYSADM.LINEITEM CARDF=4,254,339 QUALIFIED_ROWS=1,048,737.0 NPAGESF=141,743WHERE ( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY COLCARDF=1,063,455/1,500,00

MAX_FREQ=0.0001645%/0.000066666666FF=6.666666649834951E-7

AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY COLCARDF=10,000/10,000 MAX_FREQ=99.99%/

0.011917244958617542% FF=9.99999901978299E-5

AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' COLCARDF=4MAX_FREQ=50.75460606218732%

FF=0.24650996923446655

AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) COLCARDF=2,304 MAX_FREQ=0.0414% FF=0.30288922786712646 LOW2KEY=1992-01-02 HIGH2KEY=1998-

08-01 AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) COLCARDF=2,304

MAX_FREQ=0.0414% FF=0.7351873517036438 LOW2KEY=1992-01-02 HIGH2KEY=1998-

08-01 AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) COLCARDF=25

MAX_FREQ=4.21%FF=0.1526399850845337

AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 COLCARDF=10,000 MAX_FREQ=99.99% FF=0.9998999834060669 LOW2KEY=2 HIGH2KEY=9999

)GROUP BY SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAMEHAVING SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 - SYSADM.LINEITEM.L_DISCOUNT ) ) > 1000000 FF=0.3333333134651184ORDER BY REVENUE DESC


72

Annotated within OSC


73

Annotation table details

• Table annotations– CARDF – number of rows in the table

– NPAGESF – number of pages

– Qualified rows – optimizer estimated number of rows after local filtering (as though table is the outer table)

• Value– Poor join sequence often due to inaccurate filtering estimate at

the single table level

– Qualified rows shows optimizer estimate, is it accurate?

– CARDF, NPAGESF – useful to have this embedded as you’re looking at SQL and predicates


74

Annotation table close-upsSELECT cols…

FROM SYSADM.SUPPLIER

CARDF=10,000

QUAL_ROWS=1,526.2466

NPAGESF=402

, SYSADM.ORDER

CARDF=1,500,000

QUAL_ROWS=57,114.59

NPAGESF=42,417

, SYSADM.LINEITEM

CARDF=4,254,339

QUAL_ROWS=1,048,737.0

NPAGESF=141,743


75

Annotation predicate details

• Column information– COLCARDF – number of distinct values for a

column. For join predicate, both COLCARDF’s shown

• Show selectivity of each predicate

– Max Freq – Maximum frequency (if collected)• Skew statistics collected?

• Show most skewed frequency (indicator of degree of skew)


76

Annotation predicate details

• Column information (cont.)– FF (filter factor) – optimizer single predicate filter

factor estimate• Show optimizer filtering estimate• Is it accurate? (Compare to actual filtering)

– LOW2KEY / HIGH2KEY – shown for range predicates.

• Observe literal value with “domain” of actual values• Can spot stale columns statistics…• Eg. HIGH2KEY = 2006-11-01, you’ve got 2007 data in the

table…• WHERE SALES_DT BETWEEN 2007-01-01 AND 2007-

01-31


77

Annotation predicate close-up( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY

COLCARDF=1,063,455/1,500,00MAX_FREQ=0.0001645%/0.000066666666FF=6.666666649834951E-7

AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY COLCARDF=10,000/10,000 MAX_FREQ=99.99%/0.0119% FF=9.99999901978299E-5

AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' COLCARDF=4MAX_FREQ=50.75460606218732%

FF=0.24650996923446655

AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) COLCARDF=2,304 MAX_FREQ=0.0414%FF=0.30288922786712646 LKEY=1992-01-02 HKEY=1998-08-01

AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) COLCARDF=2,304 MAX_FREQ=0.0414%FF=0.7351873517036438 LOW2KEY=1992-01-02HIGH2KEY=1998-08-01

AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) COLCARDF=25 MAX_FREQ=4.21%FF=0.1526399850845337

AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 COLCARDF=10,000 MAX_FREQ=99.99% FF=0.9998999834060669 LOW2KEY=2 HIGH2KEY=9999


78

General annotation comments

• SQL analysis– Start at the table level

• Identify any tables where qualified rows estimate looks off

– Examine predicates on that table• Any individual predicates look too

– Optimistic?– Pessimistic?– Frequencies, histograms?

• After individual predicates are OK, any unaddressed correlation?

– KEYCARD, COLGROUP?

– Difficult to estimate predicate?• WHERE SUBSTR(C1,1,4) = ‘ABCD’


79

Query annotation transformed SQL

• Transformed SQL– All original SQL annotation benefits plus

– Show optimizer rewritten SQL• Predicate transitive closure

• Predicate transformations

• Distribution and pruning

• Merge / materialization


80

Query annotation transformed SQL

• Transformed SQL– Show SQL before / after distribution

– Before / after pruning


81

Untransformed UIV SQL

– Where predicates referencing the view


82

Transformed UIV SQL

Hey, where are the predicates?!?


83

Transformed UIV SQL

– Predicates distributed into each leg of the union all view

– Wonder what transformations optimizer did with your query?

– Wonder no more…


84Untransformed UIV5 union all legs

1

2

3

4

5


85Transformed UIV SQL3 union all legs

1

2

3


86

Query report

• Easy to read report which shows– Database / table space

– Table

– Index

– Indexed columns

– Column group cardinality

• Quick reference– Statistics on objects

– Statistics on indexed columns

– KEYCARD statistics mapped into index


87

Query Report


88

New SQL

SELECT <cols>FROM SYSIBM.SYSPACKSTMT AS A , SYSIBM.SYSPACKAGE AS CWHERE ( A.QUERYNO > -1 AND C.COLLID = 'TEST' AND C.NAME = 'PGM1' AND A.COLLID = C.COLLID AND A.LOCATION = C.LOCATION AND A.NAME = C.NAME AND A.VERSION = C.VERSION )ORDER BY <cols>


89

Query report table 1


90


• Looks like someone forgot to run RUNSTATS…

• Hey… this is MY database!


91

Statistics advisor says…

RUNSTATS TABLESPACE DSNDB06.SYSPKAGE

TABLE(SYSIBM.SYSPACKSTMT)

COLUMN(NAME,QUERYNO,VERSION,COLLID)

TABLE(SYSIBM.SYSPACKAGE)

COLUMN(NAME,VERSION,COLLID)

INDEX(SYSIBM.DSNKSX01 KEYCARD

,SYSIBM.DSNKKX02 KEYCARD

,SYSIBM.DSNKKX01 KEYCARD)

SHRLEVEL CHANGE REPORT YES


92


• Nice to see cascading MCARD’s– Only matching on 2 or 3 columns?

– Easy to see the KEYCARD bound…


93


• Easy to read index, indexed columns report.• Observations…

– Clusterratiof of DSNKKX01? 0.795 (reorg?)

– COLCARDF of CONTOKEN?• Not used as a predicate!


94

Access Plan Graph

• Visual explain– Ported from Visual Explain product

– Graphically display access path


95

Access plan graph


96

Access plan graph


97

Access plan graph


98

Query graph overview

• Reading query graph– Read from bottom up

– Left to right


99

Access plan graph


100

Access plan graph zoom


101

Query graph zoom reading

• Access index– Plan_table_IDX1

– Qualifies 2 rows

• Fetch from table– Plan table has 930 rows

– Approximately 1.6 survive

– Inner table is accessed approximately 1.6 times


102

Access plan graph zoom


103

Descriptor


104

Descriptor


105

Descriptor

• Displays detailed information– Descriptor changes as you navigate plan graph

– Click on index node for index information

– Click on table node, table information shown

– Double click node will pop-up independent descriptor window for node


106

Index descriptor

• Index node information– Index creator, name

– Statistics on index

– Uniqueness

– Clustering attribute

– …


107

Index scan descriptor

• Index scan information– Index cardinality

– Index leaf pages

– Matching predicates• Individual matching predicate filter factor

• Bounded matching predicate filter factor

– Total index filtering (matching and screening)

– Estimated scanned leaf pages

– Estimated index qualified rows


108

There’s more…

• Other information…– Identify matching, screening, S1, S2 predicates– Filter factors for predicates– Qualified partitions– Parallelism

• Key range • Page Range• Degree of parallelism

– Sort keys, sort key length– Estimated number of records at

• Index matching, total index filtering• Data access• Sort input / output• Join


109

Demo query graph as time permits


110

Textual explain view…


111

Textual explain view…

• OSC shows plan_table rows• Short / long format• … this is obviously a truncated view…


112

Visual Plan Hint

• Visual Plan Hint– Graphically alter the explain table

• Will show an overview– More details included as part of

– Optimization hints presentation• http://www.bwdb2ug.org/PDF/Optimization%20Hints.pdf


113Visual Plan HintVisual Plan Hint


114Change join sequenceChange join sequence


115Join connectors removedJoin connectors removed


116Move the tables aroundMove the tables around


117Create, connect nodesCreate, connect nodes


118Validate hintValidate hint

Hint


119Validate hint(will show screen shot)


120

Deploy hint

Input query level settings


121

Visual Plan Hint Summary

• Visual Plan Hint– Graphically alter the explain table

– Use of drop down lists reduces errors

– Validation of optimization hint built in

– Colorful compare of before / after access path, very nice


122

OSC Summary

• Optimization Service Center overview– Support for workloads, workload advisors– Monitoring allows for profiling of applications and exception

generated workloads– Consolidated inputs / outputs– Improved statement cache display (allow customized views)

• Improved query tuning tools– Statistics advisor– Annotation– Query report– Access plan graph– Visual plan hint


123

Patrick Bossman

[email protected]

Tuning with Optimization Service Center II

®

© ibm corporation 2007 1 patrick bossman ibm silicon valley lab tuning with optimization service...

Documents

ibm corporation

query source slide

cache slide

query query tools slide

query text slide

query menu slide

target query slide

query tuning options