ukoug

AWR and ASH Deep Dive with Enterprise Manager 12c and BeyondKellyn Pot’VinConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramDecember, 2014

Oracle Confidential

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Confidential 4

Program Agenda

ASH/AWR Reports in EM12c

Compare ADDM

SQL Monitor

ASH/AWR Reporting from the Command Line

Search SQL, (Least Used Feature of EM)

ASH Queries

AWR Warehouse

When Someone Does’t Use AWR and ASH

Brief History

• ASH= Active Session History

• AWR= Automatic Workload Repository

• Introduced in Oracle 10g

• Evolution to statspack, requests for performance reporting improvements.

• “Always on” approach to performance metrics with requirement of non-locking collection process.

• Requires Tuning Management Pack License from Oracle.

Where We Will be Spending our Time Today…

Running ASH Report from EM

• ASH is by time, not snapshot.• Set start date and time.• End date and time• Generate report

HTML Format ASH

Main ASH Info

Top SQL, Top Sessions

Top SQL Details

Top Parallel, Top DB Files

How Often Are We Asked, “What Changed?”

• It ran fine last week, now it doesn’t!

• ETL loads have changed, but no one has released any new code!

• The DBA says there hasn’t been any parameter changes to this database, but I’m sure there have been.

• Compare ADDM Comparison Resolution

What is in an ADDM Compare?

• Report Includes Following:

– Clear Demonstration of impact of change.

– Recommendations to address issue.

– Identifies causes behind change, (with limitations.)

– Lists Regressed SQL, too!

– Tip: If Installing to database for first time, (simple installation, nothing to concern about, just pkg to fulfill views) you must have preferred credentials SET of install will fail!

Executing a Report from EM12c

• Choose focus period

• Choose to compare to an Offset, Baseline or Custom

A Comparison ADDM Report…

• Report Includes Following:• Clear Demonstration of impact of change.• Recommendations to address issue.• Identifies causes behind change, (with limitations.)• Lists Regressed SQL, too!

Tip: If Installing to database for first time, (simple installation, nothing to concern about, just pkg to fulfill views) you must have preferred credentials SET of install will fail!

Performing a Comparison

Yes, you can compare one snapshot against another snapshot in DIFFERENT database if you are using this in AWR Warehouse!

2-3pm, Compared on the 12th vs. the 13th

• Familiar interface with visual wait event comparisons.

• Average # of sessions during each period are displayed.

Detailed Report

High level data, highlight for analysis and recommendations

Regression SQL

Any performance degradation is noted with the down arrow icon:

Top Segments Causing IO Waits

• High Level data• Click on “Show Hot Object Breakdown” to see more detail.

Resource Comparisons

• Comparisons of Memory, CPU, IO and Interconnect.

• Memory Is there Virtual paging?

- Memory Base Period

- Memory Comparison Period

Dashboard Provided for CPU, Memory, IO and Interconnect

• Is something OTHER than Oracle the cause?

IO Bound Dashboard

• Base vs. comparison period

• Temp reads/writes specified

• Single block read latency

SQL Monitor in EM12c

Status of Statement

Wait Events

Degree of Parallelism

SQL_ID

SQL Text

Exadata and Offloading

• Drill down to specific statement within SQL Monitor will display offload efficiency per statement.

Full Detail of SQL Execution

View Report

SQL Monitor Report CLISET LONG 1000000

SET LONGCHUNKSIZE 1000000

SET LINESIZE 1000

SET PAGESIZE 0

SET TRIM ON

SET TRIMSPOOL ON

SET ECHO OFF

SET FEEDBACK OFF

SELECT DBMS_SQLTUNE.report_sql_monitor(

sql_id => '5vh6y3b7tnv8r',

type => 'TEXT',

report_level => 'ALL') AS report

FROM dual;

Text Output of SQL Monitor

One of the Best & Least Used Features in EM(10g, 11g, 12c)

Search SQL

Problem Query

We Have the SQL_ID, What Next?

4v2tsp8dz0nhn is our SQL_ID

Go to the EM Console, (Example is EM12c)

Search SQL Interface

• Choose AWR Snapshots, (change Time Period), AWR Baselines and put SQL_ID

Click on Search

• SQL_ID link for SQL Details

• Split up by tabs for Cursor, AWR, Baselines and SQL Tuning Sets

• Plan Hash Value

• Elapsed Time

AWR Data

Snapshot IDs

• Click on Snapshot ID and gather valuable data on resource usage during snapshot time or choose to view report.

AWR Report or Run ADDM Report

AWR and ASH from the CLI

All DBAs should know how to do this, (plus, Thor commands it!)

Running Reports, Command Line

$ORACLE_HOME/rdbms/admin/awrrpt.sql;

$ORACLE_HOME/rdbms/admin/ashrpt.sql;

$ORACLE_HOME/rdbms/admin/awrsqlrpt.sql;Less Known AWR Reports:

awrinfo.sql General AWR Info

awrddrpt.sql Comparison report between snapshots

awrblmig.sql Migrates pre-11g baseline data into 11g Baseline tables.

awrgrpt.sql RAC Aware AWR Report.

AWR Info Report

• Snapshot Interval Information

• Basic Info on Instances and Nodes

• No User or Application Schema info.

• Space Usage by SYSAUX

• WRH$ and Non- AWR Objects, ordered by size

• Snapshot info and if any errors.

• Advisor Tasks

• Use for sizing AWR Warehouse estimates.

AWR Info Report

ASH Report@$ORACLE_HOME/rdbms/admin/awrrpt.sql;

-Report Format: Text or HTML-Days to view snapshot IDs-Beginning and Ending Snapshot ID’s- Name of Report

ASH Report@$ORACLE_HOME/rdbms/admin/ashrpt.sql;

-Report Format: Text or HTML.-Timestamp to being report from.-Duration in minutes.-Name of report.

SQL_ID Specific AWR Report

“Interesting Part”

Finale!

Select * from table(dbms_xplan.display_awr(‘43mp3mjufgnkg’));

Querying ASH Data Directly

• More defined reporting

• No need to pull full report

• Detail on waits that are of interest

• Join to non-AWR objects

• Examples and Ideas…

V$ACTIVE_SESSION_HISTORY• SAMPLE_ID- This is a unique identifier within an ASH sample.

• SAMPLE_TIME- A unit of time used by Active Session History, (not to be confused with DB_TIME)

• USER_ID- Identifier for a user that’s executing the session.

• SESSION_ID- Same as the SID or Session ID and can be used to join to SID in other views/tables.

• SESSION_STATE- What was the state of the session when ASH recorded the sample.

• ON CPU/WAITING- The two session states in Active Session History. ON CPU is Active, vs. Waiting, which is self-explanatory.

• EVENT- Type of event that the session is currently active or waiting on.

• TIME_WAITED- How long the session has been waiting if waiting.

• WAIT_TIME- Confusing- but this is populated by any wait time if the session is currently active and for the previous waits.

• SQL_ID- The unique identifier for the SQL statement being executed.

• SQL_CHILD_NUMBER-The cursor child number.

Session Averagesselect

ROUND(RATIO_TO_REPORT(SUM(1)) OVER () * 100 ,2) PERCENTAGE, ash.session_type

SESS_TYPE,

session_state STATUS, decode(nvl(sql_id,'-1'),'-1','nonsql','sql') SQL_TYPE,

count(distinct to_char(session_id)|| to_char(session_serial#)) SESS_CNT from

v$active_session_history ash

sample_time > sysdate - 30/(24*60) and (( ash.session_state = 'ON CPU' )

or ( ash.session_type != 'BACKGROUND' ))

group by

ash.session_type,

ash.session_state, decode(nvl(sql_id,'-1'),'-1','nonsql','sql')

order by count(*)

Session Avg. Output

• Note the % of Background processes

Inspecting Whatselect * from (select

ash.SQL_ID , ash.SQL_PLAN_HASH_VALUE Plan_hash, aud.name type,

sum(decode(ash.session_state,'ON CPU',1,0)) "CPU",

sum(decode(ash.session_state,'WAITING',1,0)) "WAITING",

sum(decode(ash.session_state,'WAITING', decode(wait_class, 'User I/O',1,0),0))

"IO WAIT" ,

sum(decode(ash.session_state,'WAITING', decode(wait_class,'User I/O',1,0),0))

"IO" ,

sum(decode(ash.session_state,'WAITING', decode(wait_class, 'Concurrency',1,0)))

"CONCURRENCY" ,

sum(decode(ash.session_state,'WAITING', decode(wait_class, 'Application',1,0)))

"Application" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL“

from v$active_session_history ash, audit_actions aud

where SQL_ID is not NULL

and ash.sql_opcode=aud.action and ash.sample_time > sysdate - &minutes /( 60*24)

group by sql_id, SQL_PLAN_HASH_VALUE , aud.name

order by sum(decode(session_state,'ON CPU',1,1)) desc) where rownum < 5;

10 Min. View of Waits by SQL_ID

• Choose Time in Minutes To Review, (10 in our example)

• SQL_ID and Plan Hash Value Shown

• Waits for CPU, Wait, IO Wait and others.

Quantity of Events Occurred Over Small Amounts of TimeCol event for a50

select event, count(1)

from v$active_session_history

where sample_time between

to_date('21-FEB-14 01.43.00 PM','dd-MON-yy hh:mi:ssPM')

to_date('21-FEB-15 01.53.00 PM','dd-MON-yy hh:mi:ssPM')

group by event

order by event;

Results, Where to Focus?

Transaction Wait Detailselect

to_char(sample_time,'HH:MI') st, substr(event,0,20) event,

ash.session_id sid, mod(ash.p1,16) lm, ash.p2,ash.p3,

nvl(o.object_name,ash.current_obj#) objn,

substr(o.object_type,0,10) otype, CURRENT_FILE# fn,

CURRENT_BLOCK# blockn, ash.SQL_ID,

BLOCKING_SESSION bsid

from v$active_session_history ash, all_objects o

where event like 'enq: TX%'

and o.object_id (+)= ash.CURRENT_OBJ#

and sample_time > sysdate - 10/(60*24)

order by sample_time;

Transaction Lock Output

• What TX row locks are occurring!

Knowing What’s in the ASH Buffer

• Deters from making assumptions on what data is being queried.• Know your samples!

Wait Events Across Nodes

Query top 10 SQL_ID’s in the last 10 minutes?

SQL_ID and CPU Usage

IO Waits by Object from ASH

SQL Text with ASH

• SQL for most recent five minutes of sample data from ASH

SQL Results

• SQL_ID, SQL Text, Sample Time that Process was captured in.

Average Activity- Graphed

SELECT

to_char(to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS'),'DD-MON HH24:MI:SS') tm, samples npts,total/&samples aas,

substr(substr(substr(rpad('+',round((cpu*&v_bars)/&samples),'+') || rpad('-',round((waits*&v_bars)/&samples),'-') ||

rpad(' ',p.value * &v_bars,' '),0,(p.value * &v_bars)) || p.value || substr(rpad('+',round((cpu*&v_bars)/&samples),'+')

|| rpad('-',round((waits*&v_bars)/&samples),'-') || rpad(' ',p.value * &v_bars,' '),(p.value * &v_bars),10) ,0,30)

,0,&v_graph)graph,total,cpu, waits

FROM (

SELECT to_char(sample_time,'YYMMDD')tday, trunc(to_char(sample_time,'SSSSS')/&v_secs) tmod,

sum(decode(session_state,'ON CPU',1,decode(session_type,'BACKGROUND',0,1))) total, (max(sample_id) - min(sample_id) + 1 )

samples, sum(decode(session_state,'ON CPU' ,1,0) cpu,

sum(decode(session_type,'BACKGROUND',0,decode(session_state,'WAITING',1,0))) waits

FROM v$active_session_history

WHERE sample_time > sysdate - &v_hours/24

GROUP BY trunc(to_char(sample_time,'SSSSS')/&v_secs), to_char(sample_time,'YYMMDD')

UNION ALL

SELECT

to_char(sample_time,'YYMMDD')tday, trunc(to_char(sample_time,'SSSSS')/&v_secs) tmod, sum(decode(session_state,'ON

CPU',10,decode(session_type,'BACKGROUND',0,10))) total, (max(sample_id) - min(sample_id) + 1 ) samples,

sum(decode(session_state,'ON CPU' ,10,0))cpu,

sum(decode(session_type,'BACKGROUND',0,decode(session_state,'WAITING',10,0))) waits

FROM dba_hist_active_sess_history WHERE sample_time > sysdate - &v_hours/24 AND sample_time < (select min(sample_time)

FROM v$active_session_history)

GROUP BY trunc(to_char(sample_time,'SSSSS')/&v_secs), to_char(sample_time,'YYMMDD')) ash, v$parameter p

WHERE p.name='cpu_count‘ ORDER BY to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS');

Pivot the Wait Events

Digging into History

• DBA_HIST_ACTIVE_SESS_HISTORY

– SNAP_ID

– SAMPLE_ID

– SAMPLE_TIME

– SESSION_ID

– USER_ID

– SQL_ID

–WAIT_CLASS

– SESSION_STATE

– PGA_ALLOCATED

Process InformationSELECT * FROM (

SELECT /*+ PARALLEL(4) */

count(*) AS count,

user_id, program, module, sql_id

FROM SYS.DBA_HIST_ACTIVE_SESS_HISTORY

WHERE sample_time > TO_DATE('19-FEB-2014 03.00.00 PM','dd-MON-yy hh:mi:ss PM')

AND sample_time < TO_DATE('19-FEB-2014 08.00.00 PM','dd-MON-yy hh:mi:ss PM')

AND program LIKE 'oracle@%'

GROUP BY user_id, program, module, machine, sql_id

ORDER BY count(*) desc

WHERE rownum <= 20

Results of Process History

Tyler Muth ASH Mining Query

ASH Mining Output

Additional Options:• Physical Read Averages• Physical Writes, (Max/Averages)• Redo Info• Login Info• Hard Parsing, etc.

Best Practice When Querying ASH Data

• Keep it Simple and don’t reinvent the wheel.

• Samples are an alias for time, not for counts.

• Understand what is valuable and compare to packaged reports.

• Be aware on RAC of node specific data.

• Take care when querying Obj#, File# and Block#, (still issues in different versions…)

• Check the time that is available in buffer, don’t assume!

AWR/ASH Links/Blogs

• Tyler Muth: http://tylermuth.wordpress.com/

• Kyle Hailey, John Beresniewicz, Graham Wood: http://ashmasters.com/

• Mine- “For the Love of ASH and AWR” http://dbakevlar.com/2011/02/for-the-love-of-awr-and-ash/

• Karl Arao- http://karlarao.tiddlyspot.com/

• Guy Harrison- http://guyharrison.squarespace.com/opsgsamples/

Why an AWR Warehouse?

1. Centralized location offers space savings to retain AWR data in local SYSAUX tablespace.

2. Ability to offload resource demands when doing deep analysis and historical trending of AWR data on source database.

3. Centralizing the data, identified by Database identifier, host, allows analysis on more than one database without database links.

AWR Warehouse in EM12c• Automatic Workload Repository

(AWR) is the de factoperformance repository for Oracle databases since 10g

• Default retention period of 8 days prevents diagnosis of long term performance problems (“Compare performance during this quarter’s books close with last quarter’s”)

• Increasing AWR retention period increases overhead and cost in critical production environments

CRMFinance Supply Chain

Requirements

• AWR Warehouse Repository -11.2.0.4 or higher.

• Preferred Credentials set up for all targets involved.

• Discover the database you will use for your repository in the EM12c

• Pre-discover any source database before able to add.

• For a RAC target or AWR Warehouse, ensure you’ve set up a shared location for the ETL load files.

AWR Warehouse - Architecture• Central warehouse configured for

long term AWR data retention

• Historical and ongoing AWR snapshots collected from databases enabled for AWR warehouse

• ETL jobs moves snapshots from source databases into AWR warehouse

• Retention period configurable for weeks, months, years or forever (default)

Requirements

• AWR Warehouse Repository database must be 11.2.0.4 with patch or 12.1.0.2 with patch.

• Will support source databases 10.2.0.4 12.1.0.x

• EM12c 12.1.0.4 with

• Requires diagnostic and tuning pack. With these, limited EE license for use of AWR Warehouse Repository database.

• As long as no additional, (RAC, Dataguard, etc.) on AWR Warehouse repository database, the limited EE license will support.

AWR ETL Jobs

Oracle Confidential – Internal/Restricted/Highly Restricted 82

DBMS Job on Source Database to directory

Job in EM Job Service that pulls file from Source Target and then pushes to AWR Warehouse Target Directory Final DBMS Job Inserts data

into AWR Warehouse

All data identified by OLD/NEW DBID and with the EM_ID, (CAW_DBID_MAPPING in AWR WAREHOUSE)

Source Target

Enterprise Manager

AWR Warehouse

Source Database ETL Job

DBMS Scheduler Job Name: MGMT_CAW_EXTRACT

Exec Call: begin dbsnmp.mgmt_caw_extract.run_extract; end;

How Often: 3 Hour Intervals if “playing catch up”, otherwise, 24 hr interval.

EM12c EM Job

• Agent to agent push.

• No other interaction with EM12c outside of interface.

AWR Warehouse ETL Load Job

DBMS Scheduler Job Name: MGMT_CAW_LOAD

Exec Call: begin dbsnmp.mgmt_caw_load.run_master;

How Often: 5 Minute Intervals

Biggest Resource Demand from the “run_master”:

begin dbms_swrf_internal.move_to_awr(schname => :1); end;

New DBSNMP Objects

CAW_EXTRACT_PROPERTIES : Information on ETL job, dump location and intervals.

CAW_EXTRACT_METADATA : All data about extracts- times, failures, details.

Accessing

• First access, will be asked to set up AWR Warehouse to database.

• Request to add first source database to repository.

AWR Warehouse Interface• Warehouse dashboard tracking

ETL jobs

• All AWR features available on long term AWR data

• Performance page

• AWR report

• ASH analytics

• Compare Period ADDM

• Compare Period Report

• Integrated seamlessly into EM UI

• Zero runtime overhead on source Production databases

How Do You Know You are Using it?

• Upper right hand corner drop down

• Can switch between and if non-existent, console will inform the user.

AWR Warehouse Configurations

Along with AWR Warehouse Basics-

• Configuration Status

• Version of Repository Database

• Host Information

• Connect Info, (SID/Service Name, Port)

• Space Usage, Upload Interval and Retention

Databases and Reports

Considerable report options

Add or remove source databases to the warehouse.

Snapshots

View snapshot loads per day

High loads due to catch up, new source db’s.

Database Details

• Database, go to DB Details Page

• DB Type, DB Name, Version, Owner, Enabled

• Snapshot Information

Upload Information

• Add

• Highlight, Remove

• Highlight, View Errors

• Privileges to Manage Snapshots

Snapshot Information

• Oldest timestamp

• Newest timestamp

• Days of Snapshots

• Count

Database List

Clicking on Target Name will TAKE YOU TO DB Performance Home Page!

ETL Load Errors

Managing Snapshot Privileges

• Viewing Access

• Manage Snapshots

• Retention Time

• Removal of Snapshots

Centralized AWR Warehouse Objects

Simple and clean schema edition to the DBNSMP.

What Can I do with the AWR Warehouse?

Oracle Confidential – Internal/Restricted/Highly Restricted 101

100000

150000

200000

250000

300000

350000

400000

June 10, 2013

l_reads_s read_iops_max read_iops_direct read_iops_direct_max

100000

200000

300000

400000

500000

600000

June 10, 2014

l_reads_s read_iops_max read_iops_direct read_iops_direct_max

CPU Usage Issues

“Nothing’s changed in months…”

Using the AWR Warehouse for Capacity Planning

“Do we really need more partitioning and pruning? Our disk usage hasn’t increased this last year….”

CAW_DBID_MAPPING Table

Most important table for anyone querying the AWR Warehouse!

Querying the AWR Warehouse Tips

Update Scripts with DBID identified to filter.

• CAW_DBID_MAPPING on OLD_DBID/NEW_DBID=DBID

• CAW_DBID_MAPPING on TARGET_NAME=TARGET_NAME

How Current AWR Queries Change

from dba_hist_sys_time_model stm, dba_hist_snapshot s, gv$parameter p, dbsnmp.caw_dbid_mapping m

where stm.stat_name in ('DB CPU','background cpu time')

and LOWER(m.target_name)= '&dbname'

and s.dbid= m.new_dbid and s.snap_id = stm.snap_id

and s.dbid = stm.dbid and s.instance_number = stm.instance_number

and p.name = 'cpu_count' and p.inst_id = s.instance_number)

Answer Specific IT Questions

Updating Existing AWR Queries to Go Across Hosts

select * from (selectm.target_name,sum(CPU_TIME_DELTA),sum(DISK_READS_DELTA),count(*)fromDBA_HIST_SQLSTAT a, dba_hist_snapshot s, dba_hist_database_instance di, dbsnmp.caw_dbid_mapping mwhere di.host_name='&host'and di.dbid in m.new_dbidand m.new_dbid = a.dbidand a.snap_id = s.snap_idand s.begin_interval_time > sysdate -120group by m.target_nameorder bysum(CPU_TIME_DELTA) desc)

Graphing CPU Usage Per DB for One Host

AWR Warehouse Scripts at DBAKevlar.com

More scripts added as time goes by…

Want to Learn More?

Oracle Screenwatch: https://www.youtube.com/watch?v=StydMitHtuI

DBAKevlar Blog Posts:

http://dbakevlar.com/2014/06/awr-warehouse-in-em12c-rel-4/

http://dbakevlar.com/2014/06/awr-warehouse-in-em12c-rel-4-part-ii/

Scripts: http://dbakevlar.com/scripts/

Connect with me via Social Media:

ukoug

oracle andor

sole discretion of oracle

powerpoint template

safe harbor statements

safe harbor slides

safe harbor statementthe

safe harbor statement

picture slide ideal

Technology

2012 02 03 ukoug rman best practices

how the jdeveloper team test jdeveloper at ukoug'08

automatic storage management ukoug rac & ha … ·...

embracing database diversity: the new oracle / mysql dba - ...

ukoug applications conference 2013 - amosca 11123.pdf ·...

ukoug conference sales guide

ukoug tech13 bi_publisher_and_bi_em12c_dhananjay_papde_v1

ukoug 2011: practical mysql performance tuning

ukoug tech 15 - odi lifecycle and data governance

ukoug 2011: mysql architectures for oracle dba's

carl dudley – tradba ltd 1 the role of roles and...

monitoring oracle soa suite - ukoug tech15 2015

panaya ukoug apps13 presentation

running oracle ebs in the cloud (ukoug apps16 edition)

ukoug 2011 mysql_arch_for_orcl_dba

ukoug 2012 metadata management for oracle hyperion epm

oracle data pump carl dudley university of wolverhampton, uk...

oracle support update ukoug db sig

adf worst practices (ukoug tech2013)

goldengate cdr from ukoug 2017