In Search of Plan StabilityPart 1

Karen MortonSr. Technical Consultant

Now part of Accenture

karenmorton.blogspot.com

@karen_morton

karen.morton@enkitec.com

Topics

• Define and identify plan stability/instability issues• Using SQL Profiles and Patches to stabilize regressed plans• Introduction to SQL Plan Management (SPM)

Flexibility vs. Stability

• Plan Flexibility– Core feature of Cost-based Optimizer CBO• Ever-changing optimal plans as data changes

• Plan Stability– Desired feature for business-critical transactions• Plans remain constant regardless of data changes

• Both have Pros and Cons!– Oracle provides several features for both

Plan Flexibility: Good or Evil?

• The mission of the CBO is to compute an optimal plan for a given SQL and its data

• Plans have many (and complex) dependencies– Heuristics– Query predicates– Schema object statistics…and so on…

• The CBO is successful most of the time– Produces a sub-optimal plan some of the time

The Recurrent Nightmare

• A business critical processes experiences an occasional slow down– Same SQL usually runs quickly– Developers claim they haven't changed anything– Users claim they do the same thing every time– DBAs claim nothing has changed with the "system"– But…the business keeps reporting a problem to YOU!

Why Plans Perform Inconsistently (1)

• Concurrency issue with some other process(es)• State of the buffer cache• CPU starvation• Sudden changes in data volume• RAC node affinity• Inconsistent performance through database links

Why Plans Perform Inconsistently (2)

• Downgraded parallel execution• Crossing small-table threshold• Resource Manager directive• Plan "flips"• Many others…

Why do plans "flip"?

• Bind variable peeking (histograms)

• Outdated object statistics• Missing object statistics• New object statistics• New empty partitions• Database parameter changes

• System statistics changes• Incomplete set of hints• Dropped, invalid, invisible, or

new indexes• Index or table rebuild• Two or more plans with

similar cost

How to Mitigate Plan Flipping

• Solid object statistics• Default CBO parameters• Avoid global changes• Good SQL formulation

Flexibility and Stability

• Plan Flexibility– Cardinality Feedback (CF)– Adaptive Cursor Sharing (ACS)– SQL Tuning Advisor (STA) Profiles

• Plan Stability– CBO Hints– Stored Outlines (SO)– SQL Plan Management (SPM)

Manually created SQL Profiles and SQL Patches

Plan Stability implementation

• 8: CBO Hints• 9i: Stored Outline (SO)• 10g: SQL Profile • 11g: SQL Patch and SQL Plan Baseline (SPB)• 12c: SQL Plan Baseline

CBO Hints• Since Oracle 8• Reliable for the most part• Can be cumbersome• Requires SQL modification– There are some workarounds

• Strict Syntax– No syntax errors

• Does not require Oracle Tuning Pack

Stored Outline

• Since Oracle 9i• Deprecated in 11g– Use SQL Plan Management (SPM) instead

• Collection of CBO hints associated to one SQL• If one hint is ignored others would still apply• Does not require Oracle Tuning Pack

SQL Profile

• Since Oracle 10g• SQL Tuning Advisor (auto) or DBMS_SQLTUNE (manual)• Collection of one or more CBO hints associated to one SQL• If one hint is ignored others would still apply• Requires Oracle Tuning Pack

Limitation: Full set of hints needed to "guarantee" plan stability

SQL Profile Types (1)

• Based on Scaling CBO Hints (Flexibility)– OPT_ESTIMATE: fudge factor – TABLE_STATS: blocks, table rows – INDEX_STATS: blocks, index rows, keys, clustering factor– COLUMN_STATS: column length, NDV, nulls, min, max

OPT_ESTIMATE(@"SEL$5DA710D3", INDEX_FILTER, "F"@"SEL$1", IDX$$_1AA260002, SCALE_ROWS=8.883203639e-06)OPT_ESTIMATE(@"SEL$5DA710D3", INDEX_SKIP_SCAN, "F"@"SEL$1", IDX$$_1AA260002, SCALE_ROWS=8.883203639e-06)OPT_ESTIMATE(@"SEL$5DA710D3", JOIN, ("B"@"SEL$1", "A"@"SEL$1"), SCALE_ROWS=4.446153275)OPT_ESTIMATE(@"SEL$5DA710D3", JOIN, ("C"@"SEL$1", "A"@"SEL$1"), SCALE_ROWS=7.884506683)OPT_ESTIMATE(@"SEL$5DA710D3", TABLE, "C"@"SEL$1", SCALE_ROWS=11.39782103)

SQL Profile Types (2)

• Based on Plan Outline (Stability)/*+ BEGIN_OUTLINE_DATA USE_HASH(@"SEL$58A6D7F6" "S"@"SEL$1") LEADING(@"SEL$58A6D7F6" "C"@"SEL$1" "S"@"SEL$1") FULL(@"SEL$58A6D7F6" "S"@"SEL$1") FULL(@"SEL$58A6D7F6" "C"@"SEL$1") OUTLINE(@"SEL$1") OUTLINE(@"SEL$2") MERGE(@"SEL$1") OUTLINE_LEAF(@"SEL$58A6D7F6") ALL_ROWS OPT_PARAM('_fix_control' '8560951:1') OPT_PARAM('_b_tree_bitmap_plans' 'false') DB_VERSION('11.2.0.4') OPTIMIZER_FEATURES_ENABLE('11.2.0.4') IGNORE_OPTIM_EMBEDDED_HINTS END_OUTLINE_DATA*/

Basically, just a set of hints…

SELECT * FROM SPONSOR S JOIN CARRIER C ON C.OID = S.CARRIER_OID WHERE S.SPONSOR_TYP = 'EMPLOYER' AND (nvl(S.TEMPORARY_PROSPECT_IND, 0) = 0 or C.DISPLAY_TEMP_PROSPECTS_IND = 1) AND S.UPPER_SPONSOR_NM LIKE :1 ORDER BY S.UPPER_SPONSOR_NM ;

SQL Profile Example (1)

sql_id = afvwm281hms4n

Performance regression after upgrade to 11.2.0.4 from 11.2.0.1.Pre-upgrade response time 0-1 second.

SQL Profile Example (2)Plan hash value: 2184022130

---------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |---------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 34172 | 53M| | 40307 (1)| 00:08:04 || 1 | SORT ORDER BY | | 34172 | 53M| 89M| 40307 (1)| 00:08:04 ||* 2 | HASH JOIN | | 34172 | 53M| | 28571 (1)| 00:05:43 || 3 | TABLE ACCESS FULL| CARRIER | 764 | 355K| | 14 (0)| 00:00:01 ||* 4 | TABLE ACCESS FULL| SPONSOR | 34190 | 38M| | 28557 (1)| 00:05:43 |---------------------------------------------------------------------------------------

Predicate Information (identified by operation id):---------------------------------------------------

2 - access("C"."OID"="S"."CARRIER_OID") filter(NVL("S"."TEMPORARY_PROSPECT_IND",0)=0 OR "C"."DISPLAY_TEMP_PROSPECTS_IND"=1) 4 - filter("S"."UPPER_SPONSOR_NM" LIKE :1 AND "S"."SPONSOR_TYP"='EMPLOYER')

Execution plan (tables had histograms on all appropriate columns)Response time 15-30 seconds

SQL Profile Example (3)Plan hash value: 1255407507

--------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |--------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 34172 | 53M| | 18215 (1)| 00:03:39 || 1 | SORT ORDER BY | | 34172 | 53M| 89M| 18215 (1)| 00:03:39 ||* 2 | HASH JOIN | | 34172 | 53M| | 6480 (1)| 00:01:18 || 3 | TABLE ACCESS FULL | CARRIER | 764 | 355K| | 14 (0)| 00:00:01 ||* 4 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34190 | 38M| | 6466 (1)| 00:01:18 ||* 5 | INDEX RANGE SCAN | SPONSOR_NM_NUK | 6410 | | | 51 (0)| 00:00:01 |--------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):---------------------------------------------------

2 - access("C"."OID"="S"."CARRIER_OID") filter(NVL("S"."TEMPORARY_PROSPECT_IND",0)=0 OR "C"."DISPLAY_TEMP_PROSPECTS_IND"=1) 4 - filter("S"."SPONSOR_TYP"='EMPLOYER') 5 - access("S"."UPPER_SPONSOR_NM" LIKE :1) filter("S"."UPPER_SPONSOR_NM" LIKE :1)

1st optimization attempt: removed histogramsResponse time ~10 seconds

SQL Profile Example (4)select a.snap_id, to_char(b.begin_interval_time,'mm/dd/yyyy hh24:mi') snap_tm, a.instance_number, a.plan_hash_value,

a.executions_delta execs,round(a.disk_reads_delta/a.executions_delta,0) avg_pio,round(a.buffer_gets_delta/a.executions_delta,0) avg_lio,round(a.rows_processed_delta/a.executions_delta,0) avg_rows,round((a.elapsed_time_delta/1000000)/a.executions_delta,0) avg_time,round((a.cpu_time_delta/1000000)/a.executions_delta,0) avg_cpu,round((a.iowait_delta/1000000)/a.executions_delta,0) avg_iowait,round((a.clwait_delta/1000000)/a.executions_delta,0) avg_clwait

from dba_hist_sqlstat a, dba_hist_snapshot bwhere a.sql_id = '&sql_id'and a.plan_hash_value = nvl(&phv,a.plan_hash_value)and a.executions_delta > 0and a.elapsed_time_delta > 0and a.snap_id = b.snap_idand a.instance_number = b.instance_numberand a.dbid = b.dbidand b.snap_id in (select snap_id from dba_hist_snapshot where begin_interval_time >= trunc(sysdate) - &nodays)order by a.snap_id desc, a.instance_number, a.plan_hash_value ;

Checked AWR for "good" previous execution plans

SQL Profile Example (5) Snap# Snap Time Inst# Plan HV Execs Avg PIO Avg LIO Avg Rows Avg Time Avg CPU Avg IO------ ---------------- ------ ----------- ------ -------- -------- --------- --------- -------- ------- 67800 08/04/2014 20:45 3 1422290831 5 15 5288 7 0 0 0 67671 08/03/2014 12:30 6 1422290831 2 1 5286 6 0 0 0 67667 08/03/2014 11:30 2 1422290831 4 0 2636 1 0 0 0 67634 08/03/2014 03:15 3 1422290831 1 7 5307 5 0 0 0 67593 08/02/2014 17:00 6 1422290831 6 0 2640 2 0 0 0 67571 08/02/2014 11:30 2 1422290831 5 1 4204 1 0 0 0 67568 08/02/2014 10:45 6 1422290831 19 55 4602 28 0 0 0 67567 08/02/2014 10:30 6 1422290831 14 22 4942 10 0 0 0

-----------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 7009 (100)| || 1 | NESTED LOOPS | | 34107 | 53M| 7009 (1)| 00:01:25 || 2 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34168 | 37M| 3590 (1)| 00:00:44 || 3 | INDEX RANGE SCAN | SPONSOR_NM_NUK | 35614 | | 27 (0)| 00:00:01 || 4 | TABLE ACCESS BY INDEX ROWID| CARRIER | 1 | 476 | 1 (0)| 00:00:01 || 5 | INDEX UNIQUE SCAN | CARRIER_PK | 1 | | 1 (0)| 00:00:01 |-----------------------------------------------------------------------------------------------

Found plan_hash_value 1422290831 from prior to the upgrade

SQL Profile Example (6)declare

ar_profile_hints sys.sqlprof_attr;cl_sql_text clob;l_profile_name varchar2(30);

begin

select extractvalue(value(d), '/hint') as outline_hints bulk collect

into ar_profile_hintsfrom xmltable('/*/outline_data/hint'

passing ( select xmltype(other_xml) as xmlval from dba_hist_sql_plan where sql_id = '&&sql_id' and plan_hash_value = &&plan_hash_value and other_xml is not null ) ) d;

Created a SQL Profile using the historically good plan from AWR

Step 1:Get the set of hints that make up the good plan from AWR in dba_hist_sql_plan

SQL Profile Example (7)select sql_text, 'PROF_fix_for_&&sql_id'into cl_sql_text, l_profile_namefrom dba_hist_sqltextwhere sql_id = '&&sql_id';

dbms_sqltune.import_sql_profile(sql_text => cl_sql_text,profile => ar_profile_hints,category => 'DEFAULT',name => l_profile_name,force_match => falsereplace => true

);

end;/

Profile now in place named PROF_fix_for_afvwm281hms4n

Step 2:Get the SQL text for this SQL_ID from AWR in dba_hist_sqltext

Step 3:Attach the good plan from AWR to the SQL using dbms_sqltune.import_sql_profile

SQL Profile Example (8)-----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 7009 (100)| || 1 | NESTED LOOPS | | 34107 | 53M| 7009 (1)| 00:01:25 || 2 | TABLE ACCESS BY INDEX ROWID| SPONSOR | 34168 | 37M| 3590 (1)| 00:00:44 || 3 | INDEX RANGE SCAN | SPONSOR_UPPER_NM_NUK | 35614 | | 27 (0)| 00:00:01 || 4 | TABLE ACCESS BY INDEX ROWID| CARRIER | 1 | 476 | 1 (0)| 00:00:01 || 5 | INDEX UNIQUE SCAN | CARRIER_PK | 1 | | 1 (0)| 00:00:01 |-----------------------------------------------------------------------------------------------------

Note----- - SQL profile "PROF_fix_for_afvwm281hms4n" used for this statement

Execute the SQL again and verify the profile is being used

SQL Patch

• Since Oracle 11g• DBMS_SQLDIAG or SQL Repair Advisor– Available as part of Enterprise Edition

• Also DBMS_SQLDIAG_INTERNAL• Collection of one or more CBO hints associated to one SQL• If one hint is ignored others would still apply

Limitation: Can only be applied to outermost query block

begin sys.dbms_sqldiag_internal.i_create_patch( sql_text => 'select count(*), max(col1) from (select * from my_view where col2 = 99)', hint_text => 'gather_plan_statistics', name => 'gps_test');end;/

SQL Patch Example (1)

Easy way to add a hint to a SQL statement that cannot be touched directly.Hint is applied only at the outermost query block.

explain plan forselect count(*), max(col1)from (select * from my_view where col2 = 99);

select * from table(dbms_xplan.display(format=>'basic +note'));

-------------------------------------------------------------| Id | Operation | Name |-------------------------------------------------------------| 0 | SELECT STATEMENT | || 1 | SORT AGGREGATE | || 2 | TABLE ACCESS BY INDEX ROWID| MY_TABLE1 || 3 | INDEX RANGE SCAN | MY_TABLE1_IDX1 || 4 | TABLE ACCESS BY INDEX ROWID| MY_TABLE2 || 5 | INDEX UNIQUE SCAN | MY_TABLE2_PK |-------------------------------------------------------------

Note----- - SQL patch "gps_test" used for this statement

SQL Patch Example (2)Verify plan to see if SQL Patch was used.

Resources

• SQL Profiles– http://docs.oracle.com/cd/E11882_01/server.112/e41573/

sql_tune.htm#PFGRF02605– http://kerryosborne.oracle-guy.com/2009/04/oracle-sql-

profiles/• SQL Patches– https://blogs.oracle.com/optimizer/entry/how_can_i_hint_a– https://blogs.oracle.com/optimizer/entry/

additional_information_on_sql_patches

SQL Plan Management (SPM)

• One or more persistent, optimal plans per SQL• Plan Stability– Only known and accepted plans can be executed

• Plan Flexibility– Capture new plans and evaluate their performance "off-line"– New plans are acknowledged but not executed until "evolved"

GoalPlan stability with controlled flexibility

SQL Plan Baselines (1)

• A set of plans available to the CBO for a given SQL– Identified by SQL Handle and Signature • Hash function on SQL Text• dbms_sqltune.sqltext_to_signature

– View dba_sql_plan_baselines• enabled = YES• accepted = YES• reproduced = YES

SQL Plan Baselines (2)

• Stores plan_hash_value and verifies plan can be reproduced before using

• Provides stability at the expense of time to review and evolve new plans (automatic plan evolution available)

• Must match SQL text exactly– No option for force_matching (as available with SQL Profiles)

Summary

• Plan flexibility and plan stability need to be managed for optimal performance

• SQL Profiles and SQL Patches can be used to apply one or more hints in an attempt to influence or "lock in" a plan

• SQL Profiles can use force_matching to allow multiple SQL statements to use the same profile (if SQL differs only by its literals)

• SQL Plan Management is the next step in the evolution of balancing plan flexibility and plan stability

What's ahead in Part 2

• How SQL Plan Management (SPM) works• How SQL Plan Baselines are created• How SQL Plan Baselines are evolved/enabled for use

Thank You!