HowtofindandfixyourJavaAPEXADFOBIEE.NETSQLPL/SQL
applicationperformanceproblem
CaryMillsapMethodRCorporationandAccentureEnkitecGroup
@CaryMillsap
DOUGJanuaryMeeting·Richardson,Texas5:00p–7:15pThursday28January2016
©2006,2016CaryMillsap
1
TMMeTHOD RTM
@CaryMillsap
Q WhatisthemostcommonOracleperformanceproblemyousee?”
3
“
@CaryMillsap
WhatisthemostcommonOracleperformanceproblemyousee?”
4
“
Assumingthatotherpeople’scommonproblemsmustbeyourproblem.
...
QA
@CaryMillsap 5
JavaAPEXADFOBIEE.NETSQLPL/SQL
@CaryMillsap
Whatisaperformance problem?
6
@CaryMillsap
Performance is notan attribute of a system.
8
@CaryMillsap 9
ID USERNAME OPERATION R SLR-- -------- --------- ------ --- 1 FCHANG OE BOOK 12.019 2.0 2 RSMITH OE SHIP 3.528 5.0 3 DJOHNSON OE PICK 1.211 5.0 4 FFORBES OE BOOK 0.716 2.0 5 FCHANG OE BOOK 1.917 2.0 6 LBUMONT PA MTCH 1.305 2.0
#define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)
@CaryMillsap 10
ID USERNAME OPERATION R SLR SPEED-- -------- --------- ------ --- ----- 1 FCHANG OE BOOK 12.019 2.0 SLOW 2 RSMITH OE SHIP 3.528 5.0 FAST 3 DJOHNSON OE PICK 1.211 5.0 FAST 4 FFORBES OE BOOK 0.716 2.0 FAST 5 FCHANG OE BOOK 1.917 2.0 FAST 6 LBUMONT PA MTCH 1.305 2.0 FAST
#define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)
@CaryMillsap
Performance is an attribute ofeach individual experience
with a system.
11
@CaryMillsap 12
TASK• id• name• ...
EXPERIENCE• id• task-id• user-id• ip-address• start-time• end-time• ERROR-code• WORK-done
SQL• ID• Task-id• ...
N
1
N
1
@CaryMillsap 13
<experience id="b3196c98-906d-4394-bc55-0339518a63b2" task-id="7" uid="238" ip="142.128.130.186" t0="2014-04-10T08:32:14.137886" t1="2014-04-10T08:32:17.891173" err="" work="3"/>
@CaryMillsap
has to finish quickly.”
clickbuttonlinkrow
queryreport
job
}{“My
14
Thisiswhatperformanceis.
@CaryMillsap
has to finish quickly.”
clickbuttonlinkrow
queryreport
job
}{“My
15
Aperformanceproblemiswhenitdoesn’t.
@CaryMillsap 16
“How long does it take?”
Responsetime(R)Durationfromservicerequesttoservicefulfillment.
Sanjay Nancy Ken Jorge
R
t0
t1
R=t1–t0
Twobigquestions...1. Howlongdidittake?2. Why?
@CaryMillsap 17
Twobigquestions...1. Howlongdidittake?2. Why?
“How long does it take?”
Responsetime(R)Durationfromservicerequesttoservicefulfillment.
Sanjay Nancy Ken Jorge
R
t0
t1
R=t1–t0
@CaryMillsap
1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.
19
MethodR
@CaryMillsap
1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.
20
MethodR
@CaryMillsap
1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.
21
MethodR
@CaryMillsap 22
MethodR
OP
TIMIZE ANYTHING
MeT H O D R
@CaryMillsap
1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.
23
MethodR
@CaryMillsap
1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.
24
MethodR
Howdoyoudothis,
whentheitisyourcode?
@CaryMillsap
EXADATAD ATA B A S EENTERPRISE EDITION
D ATA B A S ESTANDARD EDITION
D ATA B A S EEXPRESS EDITION
OracleextendedSQLtracingisafeatureofeveryOracleDatabase.
26
Oracle71992Oracle81997Oracle8i2000Oracle9i2001Oracle10g2004Oracle11g2007Oracle12c2013
@CaryMillsap
MeasuringOracleresponsetimes
27
@CaryMillsap 28
❶Activatetracing
❷Getthetracefile
❸Understanditsstory
@CaryMillsap
❶Activatetracing
❷Getthetracefile
❸Understanditsstory
29
@CaryMillsap 30
Thisisthehardestpart....Butonlythefirsttime.
Afterthat,youjustlather,rinse,repeat.
@CaryMillsap
https://app.com/apex/f?p=150:1:5547991082303::NO:::&P_TRACE=YES
31
Well,it’seasyinOracleAPEX.Todecideatruntimewhethertotraceyourcode...
@CaryMillsap
Othertechnologiesrequirealittlemorework.
First,thebasics.
32
@CaryMillsap 33
Todecideatcompiletimetotraceallyourcode...
dbms_session.session_trace_enable( waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS' );
--YourOEBOOKcode
dbms_session.session_trace_disable();
@CaryMillsap
if(should_trace('OEBOOK',dbms_random.value(0,1)){ dbms_session.session_trace_enable( waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS' );}
--YourOEBOOKcode
dbms_session.session_trace_disable();
34
Todecideatruntimewhethertotraceyourcode...
@CaryMillsap
subshould_trace(t,r){ selecttrace_proportionfromtrace_controlwheretask_name=:t; return(r<=trace_proportion);}
35
...whereshould_tracelookslikethis.
task_name trace_proportion
OEBOOK 0.05
OEPICK 0.02
OESHIP 1.00
OEINVOICE 0.01
should_trace("OEBOOK",0.00)→trueshould_trace("OEBOOK",0.01)→trueshould_trace("OEBOOK",0.02)→true...should_trace("OEBOOK",0.05)→trueshould_trace("OEBOOK",0.06)→falseshould_trace("OEBOOK",0.07)→falseshould_trace("OEBOOK",0.08)→false...should_trace("OEBOOK",1.00)→false
5%
95%
trace_control
@CaryMillsap
OracleDatabasehelpsyouimplement
runtimetracingdecisions...
...withouthavingtomakeyourdevelopersdotheifblockstuff.
36
@CaryMillsap
dbms_monitor.serv_mod_act_trace_enable( service_name =>'SYS$USERS', module_name =>'OEBOOK', action_name =>dbms_monitor.all_actions, waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS');
37
TheDBAdoesthis,atruntime.
Butthisworksonlyifyourcodesetsitsmodulenameto“OEBOOK”.
@CaryMillsap
Howyousetyourmodulenamevariesbytechnology.
SQLPL/SQLJavaADF.NETOBIEE
38
@CaryMillsap
--PL/SQLexample
dbms_application_info.set_module( module_name =>'OEBOOK', action_name =>sys_guid());
--YourOEBOOKcode
dbms_application_info.set_module( module_name =>null, action_name =>null);
39
SQLPL/SQLTosetyourcode’smoduleandactionnames...
@CaryMillsap
// JDBC 11g example
Stringmetrics[]=newString[OraCxn.END_TO_END_STATE_INDEX_MAX];metrics[END_TO_END_MODULE_INDEX]="OEBOOK";metrics[END_TO_END_ACTION_INDEX]=UUID.randomUUID().toString();conn.setEndToEndMetrics(metrics,(short)0);
//YourOEBOOKcode
metrics[END_TO_END_MODULE_INDEX]="";metrics[END_TO_END_ACTION_INDEX]="";conn.setEndToEndMetrics(metrics,(short)0);
40
JavaADFTosetyourcode’smoduleandactionnames...
@CaryMillsap
// JDBC 12c example
Properties p = new Properties();p.put("OCSID.MODULE", "OEBOOK");p.put("OCSID.ACTION", UUID.randomUUID().toString());connection.setClientInfo(p);
// Your OE BOOK code
p.put("OCSID.MODULE", "");p.put("OCSID.ACTION", "");connection.setClientInfo(p);
41
JavaADFTosetyourcode’smoduleandactionnames...
@CaryMillsap
//C#example
conn.ModuleName="OEBOOK";conn.ActionName=Guid.NewGuid().toString();
//YourOEBOOKcode
conn.ModuleName="";conn.ActionName="";
42
ODP.NETTosetyourcode’smoduleandactionnames...
@CaryMillsap 43
OBIEETosetyourcode’smoduleandactionnames...
@CaryMillsap
User’sRexperience
Oracletracefile
45
AppUser Oracle DB
time
Youwantthistobesmall
Youwantthistobesmall
@CaryMillsap
Anotherexperience
Anexperience
Notthetracefileyouwant
46
AppUser Oracle DB
time
@CaryMillsap
Anotherexperience
Anexperience
Youwantonetracefileperexperience
47
AppUser Oracle DB
time
@CaryMillsap
Thegoal:
Traceexactlyeachuserexperienceyoucareabout.
...Sothatyoucanseehowyourcodeconsumestimewhenitbehavesproperly,andwhenitmisbehaves.
48
@CaryMillsap
Thisiswhatyou’re
lookingatwhenyouusesystemwideaggregations.
50
AppUser Oracle DB
time
@CaryMillsap 51
❶Activatetracing
❷Getthetracefile
❸Understanditsstory
@CaryMillsap 52
Thisistheboringpart....Butit’saninexpensiveproblemtosolve.
@CaryMillsap
Somethingstoknow...
YourtracefileisontheOracleDatabaseserver(s),called:select tracefile from v$process where addr=( select paddr from v$session where sid=( select sid from v$mystat where rownum=1 ))
SessionswithDOP=kcancreatemorethan2k+1tracefiles.
53
@CaryMillsap
Therearelotsofwaystofetchthetracedata.FTP
SambaNFSmountportabledisk
USBthumbdriveOracleDatabasedirectoryobjects
MethodRTraceextensionforOracleSQLDeveloper
55
@CaryMillsap
Fetchingtracefilescanbeeasy.Youcanbuildtools,oryoucanbuythem.
56
Fn’m [ mifp_^ jli\f_g.
@CaryMillsap 57
❶Activatetracing
❷Getthetracefile
❸Understanditsstory
@CaryMillsap 60
AnOracletracefileisalogthatshows
whatyourcodedidinsidetheOracleDatabase.
@CaryMillsap
Somethingstoknow...
Oraclewritesatracelinewhenacall(db|os)finishes.
Therearetwoprimarylineformats:onefordbcalls,oneforoscalls.
EachcallisassociatedwithaSQLorPL/SQLstatementthroughacursorid.
Eachlinecontainsatimestamp(tim)andaduration(e|ela).
R≠∑(e|ela)becauseparentcalldurationsincludechildcalldurations.
61
@CaryMillsap 62
method-r.com
Formoredetails...
@CaryMillsap
Let’slookatsometracelines...
63
@CaryMillsap 64
begin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);
Oraclekernelcodepath
Thisisthekindofstuffyourcodecausesthe
Oraclekerneltodo.
@CaryMillsap 65
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='latch: cache buffers chains'…
WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
OracleextendedSQLtracedatabegin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);
Oraclekernelcodepath
Thisisthekindof
tracedatathekernelproduces.
@CaryMillsap 66
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='latch: cache buffers chains'…
WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
OracleextendedSQLtracedata
Ofcourse,youdon’tdirectlygettoseethekernelcode
path.
@CaryMillsap 67
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='latch: cache buffers chains'…
WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
OracleextendedSQLtracedata
...OrthathelpfulgridthatIdrewforyou.
@CaryMillsap 68
WAIT #42: nam='latch: library cache'…PARSE #42:c=10000,…WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='latch: cache buffers chains'…WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
OracleextendedSQLtracedata
Allyougettoseeisthis.
@CaryMillsap
WAIT #42: nam='latch: library cache'…PARSE #42:c=10000,…WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='latch: cache buffers chains'…WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…
69
OracleextendedSQLtracedataOraclekernelcodepathbegin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);
Youcanlearntoenvisionthekernel’scodepaththatmotivated
yourtracefile.
@CaryMillsap
Therearelotsofwaystosummarizeatracefile.tkprof
SQLDeveloper[Trace]ViewerTraceAnalyzer
tvdxstatxtraceOraSRP
MethodRProfiler
70
@CaryMillsap
Profilingtracefilescanbeeasy.Youcanbuildtools,oryoucanbuythem.
71
Fn’m [ mifp_^ jli\f_g.
@CaryMillsap
Whatyoucandowithtracefiles
72
@CaryMillsap 74
mrskew"r1-fixed.trc"
CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------SQL*Netmessagefromclient1,403.92794299.7%2,1610.6496660.0000000.927028FETCH3.0135490.2%2,1610.0013950.0000000.005000directpathreadtemp1.2590220.1%830.0151690.0032870.046968SQL*Netmoredatatoclient0.1412130.0%2,4600.0000570.0000050.001269SQL*Netmessagetoclient0.0079640.0%2,1610.0000040.0000010.000376--------------------------------------------------------------------------TOTAL(5)1,408.349690100.0%9,0260.1560330.0000000.927028
99.7%ofthetimeis2,161networkround-trips.
WhatSQLstatementscausetheround-trips?
@CaryMillsap 75
mrskew--group='($sqlid=~/^#/?"":"[".$sqlid."]")'--gl=SQLID--name='messagefromclient'"r1-fixed.trc"
SQLIDDURATION%CALLSMEANMINMAX--------------------------------------------------------------[7d0bv6ds85q1f]1,403.927942100.0%2,1610.6496660.0000000.927028--------------------------------------------------------------TOTAL(1)1,403.927942100.0%2,1610.6496660.0000000.927028
Justone.All2,161round-tripsareexecutedonbehalfofjustoneSQLstatement.
@CaryMillsap 76
mrskew--rc=p10.rc--name='SQL\*Netmessagefromclient'"r1-fixed.trc"
RANGE{min≤e<max}DURATION%CALLSMEANMINMAX----------------------------------------------------------------------------1.0.0000000.0000010.0000000.0%10.0000000.0000000.0000002.0.0000010.0000103.0.0000100.0001004.0.0001000.0010005.0.0010000.0100006.0.0100000.1000007.0.1000001.0000001,403.927942100.0%2,1600.6499670.5471100.9270288.1.00000010.0000009.10.000000100.00000010.100.0000001,000.00000011.1,000.000000+∞----------------------------------------------------------------------------TOTAL(11)1,403.927942100.0%2,1610.6496660.0000000.927028
Eachround-tripconsumesanaverageof.649967≈.650s.
Why?
@CaryMillsap
App Oracle DB
time
77
~.001s
~.001s
~.650s~.648s
EachSQL*Netmessagefromclientcall(~.650s)lookslikethis.
Ifround-tripnetworklatencyis~.002s,thenthisexperienceisspending~.648sintheJavacodeexecutedbetweendatabasecalls.
@CaryMillsap 78
mrskew--name=':dbcall'--select='$row'--slabel=ROWS--precision=0"r1-fixed.trc"
CALL-NAMEROWS%CALLSMEANMINMAX-------------------------------------FETCH216,017100.0%2,16110017100-------------------------------------TOTAL(1)216,017100.0%2,16110017100
Onefinalcheck...
Thetracefileshowsthattheapplication,atleast,isfetchinganaverageof100rowsperfetchcall(perround-trip).
ThishelpsexplaintheJava-sidelatency,butstill,.648stoprocessjust100rowsneedssomeexplaining.
@CaryMillsap 79
mrskew"r1-fixed.trc"
CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------SQL*Netmessagefromclient1,403.92794299.7%2,1610.6496660.0000000.927028FETCH3.0135490.2%2,1610.0013950.0000000.005000directpathreadtemp1.2590220.1%830.0151690.0032870.046968SQL*Netmoredatatoclient0.1412130.0%2,4600.0000570.0000050.001269SQL*Netmessagetoclient0.0079640.0%2,1610.0000040.0000010.000376--------------------------------------------------------------------------TOTAL(5)1,408.349690100.0%9,0260.1560330.0000000.927028
Nomatterhowlongyoutryto“fixthedatabase”here,you’regoingtoseeatmostonlya.3%differenceinresponsetime.
TheproblemhereisintheJava.
@CaryMillsap 81
mrskew"prd1_ora_9031.trc"
CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------PARSE735.42619778.9%6981.0536190.0000004.498316SQL*Netmessagefromclient104.76222911.2%1,3780.0760250.0003913.554818FETCH91.8000289.8%6800.1350000.0000000.506923dbfilesequentialread0.1046700.0%140.0074760.0010670.016408EXEC0.0839880.0%3490.0002410.0000000.002000gccrblock2-way0.0732330.0%960.0007630.0002800.001968gccurrentblock2-way0.0312980.0%470.0006660.0003610.001640gccurrentgrantbusy0.0280370.0%470.0005970.0001560.001508SQL*Netmoredatafromclient0.0258190.0%8370.0000310.0000000.002564CLOSE0.0189990.0%6980.0000270.0000000.00100012others0.0615760.0%1,6330.0000380.0000000.001687--------------------------------------------------------------------------TOTAL(22)932.416074100.0%6,4770.1439580.0000004.498316
PARSEcallsaccountfor78.9%oftheexperienceduration.
Thatisneverappropriate.
@CaryMillsap 82
mrskew--rc=p10.rc--name=parse"prd1_ora_9031.trc"
RANGE{min≤e<max}DURATION%CALLSMEANMINMAX--------------------------------------------------------------------------1.0.0000000.0000010.0000000.0%3070.0000000.0000000.0000002.0.0000010.0000103.0.0000100.0001004.0.0001000.0010000.0079920.0%80.0009990.0009990.0009995.0.0010000.0100000.0330000.0%330.0010000.0010000.0010006.0.0100000.1000007.0.1000001.0000008.1.00000010.000000735.385205100.0%3502.1011011.3337974.4983169.10.000000100.00000010.100.0000001,000.00000011.1,000.000000+∞--------------------------------------------------------------------------TOTAL(11)735.426197100.0%6981.0536190.0000004.498316
That’salotoftimespentparsing,andthesePARSEcallsarereallyexpensive.
@CaryMillsap 83
mrskew--name=parse--group='$sqlid'--gl=SQLID--sort=4nd"prd1_ora_9031.trc"
SQLIDDURATION%CALLSMEANMINMAX----------------------------------------------------------gkbss8w49204k4.1763630.6%3490.0119670.0000004.13537166kf30526wrgy3.1535210.4%13.1535213.1535213.1535213r3dhkb0z824v2.9115580.4%12.9115582.9115582.9115583tzra8a2a7pny1.6057570.2%11.6057571.6057571.6057572hycpfzdzsu983.1555200.4%13.1555203.1555203.1555206ppu3s1jszy3a2.2086650.3%12.2086652.2086652.20866566vkb784j9rcu1.9017110.3%11.9017111.9017111.9017115wamvs45j6nh41.4927730.2%11.4927731.4927731.492773dj1buvhxg7h191.4997720.2%11.4997721.4997721.49977241yrts4g94ghn1.6287530.2%11.6287531.6287531.628753340others711.69180496.8%3402.0932111.3337974.498316----------------------------------------------------------TOTAL(350)735.426197100.0%6981.0536190.0000004.498316
Onestatementwasparsed349times;atleast348ofthoseareunnecessary.*
Thereare350distinctSQLstatementsexecutedbythisreport....Whichisfunny,becauseyouknowthisreport,andyoudon’tremembertherebeingthatmany.*Actuallyall349areunnecessary,becausethetracefileshowsthatthere’sneveranEXECcallassociatedwithanyofthesePARSEcalls,butthat’sastoryforanotherday.
@CaryMillsap 84
mrskew--rc=distinct-texts.rc"prd1_ora_9031.trc"
SSQLIDDISTINCT-TEXTS%CALLSMEANMINMAX---------------------------------------------41518124977020.0%7011136423202577020.0%7011120477701237020.0%7011119285472397020.0%7011111389170666919.7%69111395741418510.3%349001---------------------------------------------TOTAL(6)350100.0%698101
Forthefirst5“sharedSQLid”valuesshownhere,thereare~70distinctstatementsthatcouldhavebeensharable.
Youshouldbeabletoreducetheparsecallcountfrom698to6,bywritingsharableSQLstatements,andpullingPARSEcallsoutofloops.
@CaryMillsap 85
mrskew"prd1_ora_9031.trc"
CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------PARSE735.42619778.9%6981.0536190.0000004.498316SQL*Netmessagefromclient104.76222911.2%1,3780.0760250.0003913.554818FETCH91.8000289.8%6800.1350000.0000000.506923dbfilesequentialread0.1046700.0%140.0074760.0010670.016408EXEC0.0839880.0%3490.0002410.0000000.002000gccrblock2-way0.0732330.0%960.0007630.0002800.001968gccurrentblock2-way0.0312980.0%470.0006660.0003610.001640gccurrentgrantbusy0.0280370.0%470.0005970.0001560.001508SQL*Netmoredatafromclient0.0258190.0%8370.0000310.0000000.002564CLOSE0.0189990.0%6980.0000270.0000000.00100012others0.0615760.0%1,6330.0000380.0000000.001687--------------------------------------------------------------------------TOTAL(22)932.416074100.0%6,4770.1439580.0000004.498316
Beforeyourbosswillletyou“fix”thiscode,youhavetopredictthebenefit.
Reducingtheparsecountfrom698to6shouldreduceparsingdurationfrom~735to~7,asavingsofabout730s.Responsetimeshouldimprovefrom~932sto~200s,justfromeliminatingthePARSEcallsonly.
@CaryMillsap
OPTI
MIZE ANYTHING
M eT H O D R
Youmighthaveknownthatyoushould“usebindvariables,”butyoucouldn’thavequantifiedtheRimpactontheexperiencewithoutthistracefile.
86
@CaryMillsap
BASELINE:foreachinvoicenumber{ cursor=parse(“select...whereinvoice_number=”+number); exec(cursor); loopovertheresultsettofetchalltherows;}
87
BAD
Thisishorrific:
• UsestoomuchCPUforPARSEcalls
• Serializationonlibrarycacheandsharedpoollatches
• Consumestoomuchmemoryinthelibrarycache
• Mayexecutetoomanynetworkround-trips
@CaryMillsap
BASELINE:BADforeachinvoicenumber{ cursor=parse(“select...whereinvoice_number=”+number); exec(cursor); loopovertheresultsettofetchalltherows;}
FIX1“Hey,let’susebindvariables”:foreachinvoicenumber{ cursor=parse(“select...whereinvoice_number=:a1”); exec(cursor,number); loopovertheresultsettofetchalltherows;}
88
STILLBAD
Alittlebetter,butstillreallyawful:
• UsestoomuchCPUforPARSEcalls
• Serializationonlibrarycachelatches
• Maybe,toomanynetworkround-trips
@CaryMillsap
FIX1“Hey,let’susebindvariables”:STILLBADforeachinvoicenumber{ cursor=parse(“select...whereinvoice_number=:a1”); exec(cursor,number); loopovertheresultsettofetchalltherows;}
FIX2:cursor=parse(“select...whereinvoice_number=:a1”);foreachinvoicenumber{ exec(cursor,number); loopovertheresultsettofetchalltherows;}
89
BETTER
Better(only1PARSEcallnow!),butstilllotsofnetworkround-trips.
@CaryMillsap
FIX2:BETTERcursor=parse(“select...whereinvoice_number=:a1”);foreachinvoicenumber{ exec(cursor,number); loopovertheresultsettofetchalltherows;}
FIX3:cursor=parse(“ select...whereinvoice_number in(selectinvoicenumberfromwhereveryourforeachwasgettingthem)”);exec(cursor);loopovertheresultsettofetchalltherows;
90
Now,only1PARSEcall,only1EXECcall,andtheminimumpossiblenumberofnetworkround-trips.**Unlessthere’sawaytoreturnfewerrows.
BETTERYET
@CaryMillsap
BadSQLBadPL/SQL
SlownetworkMissingindexesParsinginaloop
HotblockproblemsNotenoughmemoryDisklatencyproblemsRowlockingproblems
Row-at-a-timeprocessingBaddatastructurechoice
HardwaremisconfigurationsToomuchloadonthesystem
OSparameterssetinadequatelyOracleparameterssetinadequatelySQLreturnsmorerowsthanitshould
Databasebuffercachehot/coldproblemsOraclequeryoptimizerchoosingbadplans
ReportsrunwithpoorlylimitingparametervaluesInefficientcodebetweendatabasecallsintheapplication 92
Atracefileshowsyouwhereyourtimehasgone.Performanceproblemscannothidefromthat.
@CaryMillsap
Thereareonlytwopossiblerootcauses
foranyresponsetimeproblem:
❶Callcountistoobig.
❷Latencyistoobig.*
*Probablybecausesomeoneelse’scallcountsaretoobig.
93
#ProTip
@CaryMillsap 94
mrskew--top=10"prd1_ora_9031.trc"
CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------PARSE735.42619778.9%6981.0536190.0000004.498316SQL*Netmessagefromclient104.76222911.2%1,3780.0760250.0003913.554818FETCH91.8000289.8%6800.1350000.0000000.506923dbfilesequentialread0.1046700.0%140.0074760.0010670.016408EXEC0.0839880.0%3490.0002410.0000000.002000gccrblock2-way0.0732330.0%960.0007630.0002800.001968gccurrentblock2-way0.0312980.0%470.0006660.0003610.001640gccurrentgrantbusy0.0280370.0%470.0005970.0001560.001508SQL*Netmoredatafromclient0.0258190.0%8370.0000310.0000000.002564CLOSE0.0189990.0%6980.0000270.0000000.00100012others0.0615760.0%1,6330.0000380.0000000.001687--------------------------------------------------------------------------TOTAL(22)932.416074100.0%6,4770.1439580.0000004.498316
SeehowthereareonlytwowaystoreduceaDURATION?YouhavetheCALLScolumn,andtheMEANcolumn.
ProfileslikethismakeiteasytoseehowanythingyoudotomakesomethinggofastermusttranslatetoamanipulationofeitherCALLSorMEAN.
@CaryMillsap
Withagoodtracefile,youcanpredictthe
responsetimeimpactofaproposedchange.*
*Thisisnearlyimpossibletodowithsystemwideaggregatedstatistics.
95
#ProTip
@CaryMillsap
Yourcodedoesstuff.
IncludingsomestuffinsideOracle.
Thetimethisstufftakesisyouruser’sresponsetime.
Youcanseeexactlywhatitis.
It’snotthathard.
98
@CaryMillsap
RobynSands,etal.2010.ExpertOraclePractices.Apress
DetailedinformationaboutinstrumentingyourOracleapplicationcode.
CaryMillsap.2011.MasteringOracleTraceData.MethodRCorporation
Textbookfor1-daycoursethatteachesyouhowtomasterOracletracedata.
RonCrisco,etal.2011.ExpertPL/SQLPractices.Apress
DetailedinformationaboutinstrumentingyourOracleapplicationcode.
CaryMillsap,JeffHolt.2003.OptimizingOraclePerformance.O’Reilly
DetailedinformationaboutOracletracedataandwhattodowithit.
100
@CaryMillsap 101
method-r.com www.enkitec.commethod-r.com/facebook facebook.com/enkitec@MethodR @[email protected] [email protected]
Q&A