serialization control the lock, latch, & mutex...10 serialization control fundamental structural...

serialization control 1 ©2010 OraPub, Inc.

Craig A. ShallahamerOraPub, Inc.

[email protected]

OOrraaccllee

SSeerriiaalliizzaattiioonn CCoonnttrrooll((llaattcchheess && mmuutteexxeess))

The most recent version of this

presentation is on-line at www.orapub.com.Do an OraPub search

for “serial”.

serialization control 2

Craig A. Shallahamer [email protected]

SSeerriiaalliizzaattiioonn CCoonnttrrooll TThhee lloocckk,, llaattcchh,, && mmuutteexx


Email: [email protected] Twitter: @CShallahamer OraPub.Com: Everything starts here! LinkedIn: Connect and network with Craig and the OraPub Group.

Connect with Craig and OraPub.


Relevant background info...   Studied economics, mathematics and computer

science at Cal Polytechnic State University San Luis Obispo, California, USA.

  Started working with Oracle technology in 1989 as a Forms 2.3 developer on Oracle version 5.

  Soon after started performance firefighting daily.   Co-founded both Oracle’s Core Technology and

System Performance Groups.   Left Oracle to start OraPub, Inc. in 1998.   Authored 24+ technical papers and worked in 31

countries.   Author two books: Oracle Performance Firefighting

and Forecasting Oracle Performance.   Teaches performance analysis around the world.   Oracle ACE Director.   Blogs performance research: A Wider View


We exist because DBAs want to improve but can’t get the resources they need.

Focusing exclusively on Oracle systems

performance analysis

  Performance Blog

  Free Tools

  Free Presentations

  Free Papers

  Books

  Consulting

  Training

Resources

Fast-paced one hour seminars segmented into 8 to 10 digestible modules

for Oracle DBAs and Developers designed and presented by Craig Shallahamer


Latches and mutexes create an illusion of simultaneous

memory structure access.

In reality, when changing a memory structure, only one process can make a

change at a time.


What’s the point of this?

Get the visualization tool: OP search “visual”


Serialization is death.   Parallelism is key to high performance as it

enables all available resources to be used.   If there is excess capacity while performance is

not acceptable, then a parallelism issue exists.   Everyone gets involved; business, users, DBAs,

Oracle, operating system, and CPU vendors.   But at some point all parallelism effort converges

into a serial process. If those serial processes are not fast, parallelism effort is wasted!   We must ensure the contention in on the memory

structure, not with the serialization control. single process


Fundamental structural protection requirements.

  Relational Structure concurrency is controlled through locks, that is, enqueues. –  Application locks are under application developer control.

–  Data dictionary locks are under the control of Oracle kernel code developers.

  Memory structure access is under the control of Oracle kernel code architects. –  Buffers are controlled by pinning and sometimes latches.

–  Memory lists are controlled by latches and mutexes.


The need... Any more questions?

Oracle’s buffer cache


Oracle’s library cache


Oracle’s library cache


Conceptual Serialization

Control


General serialization control


Conceptual latch/mutex acquisition algorithm.

Did Get Latch/Mutex

?

Attempts > X ?

“back off”

Start

End

Y Y

N N


Many Processes Involved


How L|M are requested.

If get_latch/mutex(‘‘latch/mutex name’’,type, mode) {

release(‘‘latch/mutex name’’);

}

Piece of kernel code

related to the requested latch/mutex


Seeing wait time in perspective.

latch free


Looking at wait time and CPU time.


Looking at wait time and detailed CPU function time.

$ fulltime.sh 32873 45 PID: 32873 SID: 9 SERIAL: 13 USERNAME: OE2 at 18-Oct 12:13:49 CURRENT SQL: SELECT COUNT(*) FROM ( SELECT SUM(OBJECT_ID) FROM ORDERS UNION total time: 44.438 secs, CPU: 41.611 secs (93.64%), wait: 2.827 secs (6.36%) Time Time Component secs % ------------------------------------------------------------ ---------- ------- cpu : [.] kcbgtcr 29.714 66.87 cpu : [.] kdstf000010100001km 3.716 8.36 cpu : [.] lnxsum 3.541 7.97 cpu : [?] sum of funcs consuming less than 2% of CPU time 2.393 5.38 cpu : [.] kaf4reasrp0km 2.180 4.91 wait: latch: cache buffers chains 0.727 6.37 Samples remaining: 0 Gathering next 120 second sample...

get tool? OP search, “fulltime”


Latching related time.

  Time can be spent in these areas: –  Getting a latch (spin: CPU time. sleep: Queue/Wait) –  Holding a latch (kernel code: CPU time. OS calls,

blocks/locks: Queue/Wait) –  Releasing a latch (kernel code: CPU time)

  Key things to remember: –  Spinning consumes CPU time, so increasing the spin

count will consume more CPU time. –  Spinning does not post a wait event. –  Sleeping consumes no CPU time and does post the

wait event, latch free or latch: <latch name>.


Detecting and resolving L|M contention.

  Understand the general latch/mutex algorithm.   Detect significant latch/mutex contention.   Determine which latch/mutex.   Understand the related kernel code.   Understand:

–  Acquisition: Why is it being requested so often? –  Holding: Why is it being held so long?

  Come up with multiple resolution strategies.   Take appropriate action to resolve.


Understanding L|M.   L|M are control structures. Latches are actually a memory

address and mutexes are controlled through system calls.   Oracle’s memory structures must be safeguarded just like

application objects.   Oracle indirectly protects memory structures by controlling

kernel code execution that touches the memory structures. There are exceptions though.

  Each latch is related to one or more kernel code module/piece/section.

  To execute a specific kernel code module, you must first get the related latch.

  Therefore, memory structures are protected by the controlled execution of the kernel code.


Understanding L|M.   Ensure serial execution of Oracle kernel code.   Can also be used to pin memory structure(s).   Surrounds a section of Oracle code to ensure its execution is controlled.   Oracle continues to improve both latching (e.g., 10.2 uses less memory

per latch) and mutex.   They must be fast… very fast.   Implemented using very simple, low level, and if possible, operating

system dependent structures. (E.g., CAS operation)   Can be requested shared or exclusive (8i+). To get exclusive, there must

be no share holders.   Latches can be placed in one of up to 8 classes for enhanced spin

control.   While serialization is enforced, no ordering or queuing is involved when

requesting a L|M.


Oracle Latch Specifics


The latch way... P:MS1

Data. . .

C:MS1

Data. . .

C:MS1

Data. . .

C:MS1

Data. . .

Latch abc123

P:MS2

Data. . .

bucket #

bucket #


How multiple latches are used.   Multiple latches is another way of saying child latches.   To avoid possibly time consuming coordination, Oracle uses

only one latch per memory “structure.”   Oracle divides a large or popular memory structure into

smaller pieces…but one latch still covers a “single” memory structure piece/part…now there are just multiple memory structure pieces.

  For example: –  For each LRU latch, there is an associated LRU chain; one to many

relationship. –  For hash buffer chains, each latch covers/protects multiple hash

chains.

  A latch can cover multiple memory structures, but a memory structure can only be covered by a single latch.


Multiple latches provide increased granularity.

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

Option “A” Option “B”


Multiple latches = Multiple structures There are 5000 buffers in the Oracle buffer cache.

This single LRU contains 5000 buffers covered by 1 LRU latch.

These 5 LRUs contain 1000 buffers each and there is a total of 3 LRU latches.


Going by the numbers…

Item 8i 9i 10g 10g 11g 12c

Data Buffers 76,800 699,994 6,513 327,680 32,256 (8k)

800,569 (8k)

Hash Chain Buckets 153,600 1,400,000 16,384 1,048,576 65,536 2,097,152

CBC Latches 1,024 8,192 1,024 4,096 2,048 65,536

CBC Buckets/ CBC Latch 153 170 16 256 32 32


Watching sleeps using strace.   Background;

–  This process and the Oracle system (11gR2) is experiencing severe cache buffer chain latch contention.

–  Processes rarely sleeps more than once (p3=0). –  CPU utilization is only around 85% on a 4 core Linux box.

  Oracle uses the select system call to force the process to wait (that is, sleep) a specific time.

  Notice there only one timeout at a time, followed by other system calls. This correlates with the single sleep.

  As we expect, the “first” sleep time is set close to 10ms. In this case, Oracle sets the wait time to 10.01ms, but in previous releases I’ve seen 10.24ms.

  The actual sleep time is not the requested time. The select documentation states a busy system will not wait the exact specific time.

  Make sure /proc/sys/kernel/vsyscall64 is set to 0 or you won’t see all system calls.


Watching an 11g latch sleep. [oracle@fourcore]$ ps -eaf|grep oracleprod18. . .oracle 24477 24476 72 13:58 ? 01:22:03 oracleprod18 . . .. . .

[oracle@fourcore spin_classes]$ strace -rp 24477Process 24477 attached - interrupt to quit. . . 0.000034 select(0, [], [], [], {0, 10001}) = 0 (Timeout) 0.011137 gettimeofday({1264117983, 18827}, NULL) = 0. . . 0.000034 select(0, [], [], [], {0, 10001}) = 0 (Timeout) 0.010299 gettimeofday({1264117983, 112927}, NULL) = 0. . . 0.000030 select(0, [], [], [], {0, 10001}) = 0 (Timeout) 0.011160 gettimeofday({1264117983, 245021}, NULL) = 0. . . 0.000048 select(0, [], [], [], {0, 10001}) = 0 (Timeout) 0.010456 gettimeofday({1264117983, 271123}, NULL) = 0. . . 0.000084 select(0, [], [], [], {0, 10001}) = 0 (Timeout) 0.011098 gettimeofday({1264117983, 589336}, NULL) = 0. . .

11gR2 Linux


Watching a 12c latch sleep. $ strace -rp 32873Process 32873 attached - interrupt to quit 0.000000 clock_gettime(CLOCK_MONOTONIC, {81913, 30086683}) = 0 0.000067 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={217, 423946}, ru_stime={0, 122981}, ...}) = 0 0.000039 clock_gettime(CLOCK_MONOTONIC, {81913, 30163130}) = 0 0.000027 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={217, 423946}, ru_stime={0, 122981}, ...}) = 0 1.991878 clock_gettime(CLOCK_MONOTONIC, {81915, 22092045}) = 0 0.000058 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={218, 96844}, ru_stime={0, 122981}, ...}) = 0 0.000039 clock_gettime(CLOCK_MONOTONIC, {81915, 22165643}) = 0 0.000026 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={218, 96844}, ru_stime={0, 122981}, ...}) = 0 2.005875 clock_gettime(CLOCK_MONOTONIC, {81917, 28087969}) = 0 0.000056 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={218, 848729}, ru_stime={0, 122981}, ...}) = 0...... then bamb!... 0.000026 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={225, 612701}, ru_stime={0, 123981}, ...}) = 0 0.678239 clock_gettime(CLOCK_MONOTONIC, {81935, 710441832}) = 0 0.000056 gettimeofday({1382121114, 896822}, NULL) = 0 0.000036 semop(1933315, {{52, -1, 0}}, 1) = 0 0.010648 clock_gettime(CLOCK_MONOTONIC, {81935, 721164886}) = 0 0.000034 gettimeofday({1382121114, 907540}, NULL) = 0 0.000027 gettimeofday({1382121114, 907569}, NULL) = 0 0.000059 semctl(1933315, 46, SETVAL, 0x1) = 0 0.099170 semctl(1933315, 60, SETVAL, 0x7f4000000001) = 0 0.000064 clock_gettime(CLOCK_MONOTONIC, {81935, 820517786}) = 0 0.000033 gettimeofday({1382121115, 6892}, NULL) = 0 0.000029 semop(1933315, {{52, -1, 0}}, 1) = 0 0.009552 clock_gettime(CLOCK_MONOTONIC, {81935, 830133485}) = 0 0.000035 gettimeofday({1382121115, 16509}, NULL) = 0 0.000028 gettimeofday({1382121115, 16536}, NULL) = 0 0.256663 semctl(1933315, 60, SETVAL, 0x7f4000000001) = 0 0.003688 clock_gettime(CLOCK_MONOTONIC, {81936, 90562312}) = 0 0.000053 gettimeofday({1382121115, 276941}, NULL) = 0...

12.1 Linux


The _spin_count issue.

  Most latch intense systems are CPU saturated, so increasing _spin_count can make an already bad situation even worse.

  Increasing _spin_count can cause response time to increase because, while perhaps wait time decreases, the service time can increase.

  Not latch specific. What is needed is the ability to alter spinning for a specific latch. This is possible with latch classes…


Using latch classes. SQL> connect / as sysdbaConnected.SQL> @latchclass

Latch Contention Report

LATCH_NAME LATCH_CLASS CLASS_SPIN_COUNT---------------------------------------- ----------- ----------------process allocation 2 20000

1 row selected.

SQL> l 1 select kslltnam latch_name, 2 class_ksllt latch_class, 3 c.spin class_spin_count 4 from x$kslltr r, 5 x$ksllclass c 6 where c.indx = r.class_ksllt 7* and r.class_ksllt > 0SQL>

Default 11gR2, Linux


Making the latch specific class changes.

SQL> @latchclass Latch Contention Report

LATCH_NAME LATCH_CLASS CLASS_SPIN_COUNT--------------------------------- ----------- ----------------process allocation 2 20000cache buffers chains 1 250

2 rows selected.

[oracle@fourcore dbs]$ grep latch_class initprod18.ora_latch_classes = "150:1" # cbc latch (11.2) assigned to latch class 1_latch_class_1 = 250

In a test, the logical IO workload increased by 7% while the CPU utilization decreased!


SQL> @rtpctxRemember: This report must be run twice so both the initial andfinal values are available. If no output, press ENTER about 20 times.

Database: prod23 10-OCT 02:15pmReport: rtpctx.sql OSM by OraPub, Inc. Page 1 Response Time Activity (127 sec interval)

Avg Time Time WaitRT Component % TT % WT Waited (ms) (sec) Count(k)-------------------------------------- ------- ------- ----------- ----------- --------CPU consumption: Oracle SP + BG procs 98.28 0.00 0.000 1291.832 0latch: cache buffers chains 1.35 78.45 5.631 17.760 3control file parallel write 0.18 10.64 33.472 2.410 0latch free 0.08 4.81 6.527 1.090 0log file parallel write 0.03 1.86 5.122 0.420 0. . .

Instance level latch identification.

OSM: rtpctx.sql


Session level latch determination. SQL> @swswp latch%free Sess ID Wait Event P1 P2 P3 ----- ---------------------------- ------------ --------- ----- 1510 latch free 1628581220 150 0 4529 latch free 1628581220 150 0 2 rows selected. SQL> l 1 select sid, event, 2 p1, p2, p3 3 from v$session_wait 4 where event like '&input%' 5 and state = 'WAITING' 6* order by event,sid,p1,p2 SQL> select * from v$latchname where latch#=150; LATCH# Instance Workload Statistic HASH ---------- ---------------------------------------- ---------- 150 cache buffers chains 3563305585 1 row selected.

Any release

latch#


System level latch determination. SQL> @latch

Database: prod18 21-JAN 04:23pmReport: latch.sql OSM by OraPub, Inc. Page 1

Latch Contention Report

Wait TimeLatch Name (sec) %Impt Impact Gets(k) Sleeps (k)------------------------ ------- ------ ---------- ---------- ----------- cache buffers chains 935 99.7 550.01 386912 461 qmn task queue latch 0 0.3 1.42 0 0 simulator hash latch 16 0.0 0.06 38653 2 Real-time plan statistic 0 0.0 0.01 2 0 space background task la 0 0.0 0.00 1 0 call allocation 0 0.0 0.00 2 0

6 rows selected.

Impact = sleeps X (sleeps/gets) Works for all Oracle releases!


Mutex Specifics


What is a mutex?   A programming facility designed for

high concurrency thread control. Do a manual page on mutex…

  Very flexibly defined (that is, setup) serialization control structure. –  A mutex can protect one or more than

one memory structure or sub-structure. –  Each memory structure (parent or child)

can have its own mutex or share a mutex.

–  It can be stored within the memory structure or somewhere else. If part of the structure, if the structure is deallocated, then so is the mutex.

  Can be held shared or exclusive.   Are independent of latches, but uses

both latches and mutexes.

A

B C

E D

  There are Library Cache and Cursor Pin mutexes.

  Oracle 11.2.x had significant mutex changes. 12c seems to have stabilized.


Each mutex has two key variables. P:MS1

Holder ID Ref Count

Data. . .

C:MS1 Holder ID Ref Count

Data. . .


Data. . .


Data. . .

P:MS2 Holder ID Ref Count

Data. . .

bucket #

Holder ID Ref Count

bucket #

Holder ID Ref Count


Why use a mutex?   Smaller memory footprint. Only 16 bytes compared to 112

bytes (10gR2+) or 200 bytes (pre-10gR2).

  Faster because of less instructions. Getting a mutex takes around 30-35 instructions compared to 150-200.

  More granular decreasing false contention. Latches typically protect multiple objects. Protecting multiple objects can cause the latch itself to be the contention, not the object. Each object can have its own mutex.

  Fast pinning. Instead of creating 100 pins, one mutex is created with a reference count of 100. A cursor opening and closing test on 10gR2 ran an average of 11% faster.


Oracle and mutexes.   There are mutexes for finding objects in the LC and for cursor related

operations (building, updating stats, executing, etc.).

  Mutexes where introduced in 10.2, significant changes 11.2.0.1, 11.2.0.2, 11.2.0.3 and now in 12.1 maturity is apparent.

  Oracle continues to enhance mutex capability, control and options.

  _kks_use_mutex_pin will be set to true. Gone in 11gR2 and beyond.

  “kgx” latches are gone in 12c. Testing needed regarding CAS.

  While yield reduces the likelihood of a processes missing an opportunity to consume CPU, it also increases the likelihood of CPU starvation and “too long” CPU run queues. The late 11.2 changed fixed this.

  Late in 11g, Oracle allowed control over the a) “wait” method, b) sleep time, c) spin control, and d) how these work together. This appears to be working much better.


Seeing mutex contention - Oracle WORKLOAD REPOSITORY report for

DB Name DB Id Instance Inst Num Release RAC Host------------ ----------- ------------ -------- ----------- --- ------------PRD32 824431438 PRDB 1 11.2.0.1.0 NO cbsprod01

Snap Id Snap Time Sessions Curs/Sess --------- ------------------- -------- ---------Begin Snap: 47566 01-Sep 11:45:12 386 17.5 End Snap: 47567 01-Sep 12:00:16 295 16.8 Elapsed: 15.08 (mins) DB Time: 192.47 (mins). . .

Top 5 Timed Events Avg %Total~~~~~~~~~~~~~~~~~~ wait CallEvent Waits Time (s) (ms) Time Wait Class------------------------------ ------------ ----------- ------ ------ ----------CPU time 3,614 31.3cursor: pin S 1,781,320 1,918 1 16.6 Otherdb file sequential read 381,140 1,216 3 10.5 User I/Odb file scattered read 947,274 814 1 7.0 User I/Olatch: cache buffers chains 20,535 402 20 3.5 Concurrenc

. . .


Oracle’s general mutex algorithm. Function Fast_Get(mutex_addr,mode) { If mutex.holder := sid occurs Then Case mode: ‘‘X’’: If mutex.ref_count = 0 Then return TRUE Else

mutex.holder = clear return FALSE End-If ‘‘S’’: mutex.ref_count++ mutex.holder = clear return TRUE End Case Else return FALSE End-If } Function Spin_Get(mutex_addr,mode) { for i = 1 to 255 Loop If Fast_Get(mutex_addr,mode) Then return TRUE End If End Loop return FALSE }

Function Get_Mutex(mutex_addr,type,mode) return number { If type =‘‘nowait’’ If Fast_Get(mutex_addr) return TRUE Else return FALSE End If Else If Fast_Get(mutex_addr) Then return TRUE Else Loop If Spin_Get(mutex_addr,mode) Then return TRUE Else Register_Event(““cursor: *””) Mutex_Wait(try++) End If End Loop End If End If } Function Mutex_Wait(try) return number { options defined by kernel developers - CPU yield, blocking wait, sleep }


Q: New mutex types in 12c? A: Does not appear to be. SQL> l 1 select distinct mutex_type 2* from v$mutex_sleep SQL> / MUTEX_TYPE -------------------------------- Library Cache Cursor Pin


When/Where are mutexes used?   Parent cursor operations, such as building a child

cursor or updating reference count (cursor: mutex X), examining a parent cursor (cursor: mutex S), and binding a cursor (cursor: mutex X).

  Cursor statistics operations, such as building and updating cursor-related statistics (cursor: mutex X), examining cursor-related statistics (cursor: mutex S), and examining v$sqlstats (cursor: mutex S).

  Cursor (un)-pinning for execute but another session has the mutex in exclusive mode to build the cursor (cursor: pin S wait on X) or another session is also trying to ‘pin S’ to execute the cursor (cursor: pin S). The ‘cursor: pin X’ is not likely to occur because Oracle may simply build another cursor.

  There are others...


Q: New mutexes in 12c?

A: Nope.

SQL> l 1 select name, wait_class 2 from v$event_name 3 where upper(name) like '%CURSOR%' 4 or upper(name) like '%MUTEX%' 5* order by name SQL> / NAME WAIT_CLASS ------------------------- --------------- SecureFile mutex Concurrency cursor: mutex S Concurrency cursor: mutex X Concurrency cursor: pin S Concurrency cursor: pin S wait on X Concurrency cursor: pin X Concurrency library cache: mutex S Concurrency library cache: mutex X Concurrency 8 rows selected.


Q: Are there any new mutex views?

SQL> select view_name from all_views where view_name like '%MUTEX%'; VIEW_NAME -------------------------------------- V_$MUTEX_SLEEP V_$MUTEX_SLEEP_HISTORY GV_$MUTEX_SLEEP GV_$MUTEX_SLEEP_HISTORY DBA_HIST_MUTEX_SLEEP CDB_HIST_MUTEX_SLEEP 6 rows selected.

A: No new views!

No changes in v$mutex_sleep other than con_id.


Mutex parameters... cleaner. SQL> Parameter Value Description Dflt? -------------------------------- ---------- ----------------------------------- ----- _mutex_spin_count 255 Mutex spin count TRUE _mutex_wait_scheme 2 Mutex wait scheme TRUE _mutex_wait_time 1 Mutex wait time TRUE _wait_yield_hp_mode yield Wait Yield - High Priority Mode TRUE _wait_yield_mode yield Wait Yield - Mode TRUE _wait_yield_sleep_freq 100 Wait Yield - Sleep Frequency TRUE _wait_yield_sleep_time_msecs 1 Wait Yield - Sleep Time (in millise TRUE _wait_yield_yield_freq 20 Wait Yield - Yield Frequency TRUE 8 rows selected.

OSM: ipx.sql

Notice there are many parameters, they are named sensibly, there is a pretty clear

description, and the values make sense. This is a sign of product maturity.

(more details in the SP section)


Wait methods   Late in 11g (11.2.0.2) Oracle opened the door to detailed mutex control.   _mutex_wait_scheme   0 : Always yield   1 : Always sleep   2 : yield and sleep with exponential back off up to _mutex_wait_time (default)

  My short 12c tests indicate there could some changes. For example, I never saw yield regardless of the wait scheme. But there could be reasons for this... stay tuned.


Causing mutex “cursor: pin S” $ cat pincursorS2.sql

declare cursor cur_one is select count(*) from dba_objects where 1=0;begin loop open cur_one; close cur_one; end loop;end;/

$ cat causePinS.sh

for i in 1 2 3 4 5 6 7 8 do echo "Loading concurrent session number $i ..." sqlplus -s system/manager @pincursorS2.sql & sleep 10done

On a 6 core box.


#1 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- AQ Background Master: slave start 65.16 95.91 15010.000 15.010 0 CPU consumption: Oracle SP + BG procs 32.07 0.00 0.000 7.387 0 log file redo write 0.61 0.89 10.000 0.140 0 #2 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- CPU consumption: Oracle SP + BG procs 95.89 0.00 0.000 13.288 0 log file redo write 0.87 21.05 10.000 0.120 0 control file parallel write 0.65 15.79 30.000 0.090 0 target log write size 0.43 10.53 6.667 0.060 0 commit: log file sync 0.36 8.77 16.667 0.050 0 cursor: pin S 0.29 7.02 1.143 0.040 0 #3 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- CPU consumption: Oracle SP + BG procs 97.10 0.00 0.000 25.080 0 log file redo write 0.50 17.33 10.000 0.130 0 cursor: pin S 0.43 14.67 1.134 0.110 0

Linux, O12.1, default mutex parameters


#4 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- CPU consumption: Oracle SP + BG procs 97.69 0.00 0.000 35.094 0 cursor: pin S 0.58 25.30 1.111 0.210 0 log file redo write 0.31 13.25 9.167 0.110 0 #5 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- CPU consumption: Oracle SP + BG procs 97.99 0.00 0.000 45.314 0 cursor: pin S 0.63 31.18 1.098 0.290 0 log file redo write 0.35 17.20 11.429 0.160 0 #6 SQL> @rtpctx Avg Time Time Wait RT Component % TT % WT Waited (ms) (sec) Count(k) -------------------------------------- ------- ------- ----------- ----------- -------- CPU consumption: Oracle SP + BG procs 98.02 0.00 0.000 54.321 0 cursor: pin S 0.78 39.09 1.123 0.430 0 log file redo write 0.20 10.00 8.462 0.110 0


Seeing 11g mutex contention - OS [oracle@sixcore ~]$ strace -cp 10526Process 10526 attached - interrupt to quit^CProcess 10526 detached% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 99.89 0.005704 0 3218395 getrusage 0.11 0.000006 0 31 sched_yield------ ----------- ----------- --------- --------- ----------------100.00 0.005710 3218426 total

The sched_yield call simply relinquishes the processor voluntarily without blocking. The process will then be moved to the end of the queue for its priority and a new process gets to run. What is so cool is if the relinquishing process is the only process in the highest priority list at that time, this process will continue to run after a call to sched_yield. Therefore, no process will be sleeping while there are idle CPU resources! However, this strategy can result in a very long CPU run queue, which has been known to cause problems.

o11.2 Linux

60 second samples


Seeing 12c mutex contention - OS

  The next slide contains multiple Linux “strace” output during massive “cursor: pins S” contention. Total time perspective: 98% CPU, 1.5% Pin S. Wait time perspective: 82.3% Pin S...

  The traced session recorded hundreds of mutex waits over the three minute sample period.

  The _mutex_wait_scheme was set to the default of 2. All other parameters mutex related parameters were left to default: _mutex_wait_time=1, _mutex_spin_count=255

  Strace recorded no shed_yield, semop, semctl. Instead we see only the select call.



$ strace -cp 51390Process 51390 attached - interrupt to quit^CProcess 51390 detached% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 36.68 0.008619 0 3527192 getrusage 31.90 0.007495 0 2822374 clock_gettime 27.16 0.006381 0 2821617 times 4.26 0.001000 3 294 select 0.00 0.000000 0 294 gettimeofday------ ----------- ----------- --------- --------- ----------------100.00 0.023495 9171771 total

$ strace –rp 51390 0.000040 clock_gettime(CLOCK_MONOTONIC, {412187, 828423439}) = 0 0.000031 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={632, 2920}, ru_stime={73, 738790}, ...}) = 0 0.000043 gettimeofday({1382451367, 14844}, NULL) = 0 0.000050 clock_gettime(CLOCK_MONOTONIC, {412187, 828549068}) = 0 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.001089 clock_gettime(CLOCK_MONOTONIC, {412187, 829667730}) = 0 0.000052 clock_gettime(CLOCK_MONOTONIC, {412187, 829724329}) = 0 0.000029 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={632, 2920}, ru_stime={73, 738790}, ...}) = 0 0.000039 times(NULL) = 470613968 0.000027 times(NULL) = 470613968

o12.1 Linux

180 second samples

load: cursor: pin S



$ cat strace.out | grep select | more 0.000041 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000079 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000034 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000031 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000025 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000031 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000033 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000033 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000033 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000035 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000027 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000032 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000030 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000024 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000068 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000031 select(0, [], [], [], {0, 1000}) = 0 (Timeout) 0.000029 select(0, [], [], [], {0, 1000}) = 0 (Timeout)

o12.1 Linux

180 second sample

load: cursor: pin S


Detecting and resolving L|M contention.

  Understand the general L|M algorithm.   Detect significant L|M contention.   Determine which L|M.   Understand the related kernel code.   Understand why:

–  The L|M is being requested so often –  The L|M is being held so long

  Come up with multiple resolution strategies.   Take appropriate action to resolve.


Resources


cpu latch serial


cpu latch serial


Resource listing   Presentations: OraPub search: cpu, latch, serial   Craig’’s Blog – Search: cpu, latch, serial   Training from OraPub

–  Oracle Performance Firefighting (I) –  Adv Oracle Performance Analysis (II) –  Super Seminars: One day Super Saturdays

  OraPub Online Institute – Any [Device, Time, Location]   Tools at www.orapub.com

–  OSM Toolkit. OP search, “OSM”. –  BC & LC Visualizations. OP search, “bc visual”, “lc visual”

  Books –  Oracle Performance Firefighting. –  Forecasting Oracle Performance.


Craig A. Shallahamer [email protected]

SSeerriiaalliizzaattiioonn CCoonnttrrooll TThhee lloocckk,, llaattcchh,, && mmuutteexx

serialization control 78 ©2010 OraPub, Inc.

Craig A. ShallahamerOraPub, Inc.

[email protected]

OOrraaccllee

SSeerriiaalliizzaattiioonn CCoonnttrrooll((llaattcchheess && mmuutteexxeess))

The most recent version of this

presentation is on-line at www.orapub.com.Do an OraPub search

for “serial”.

serialization control the lock, latch, & mutex...10 serialization control fundamental structural...

Documents