drilldown into the logwriter inner working and communication
TRANSCRIPT
![Page 1: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/1.jpg)
©2013Enkitec
Drilldownintothelogwriterinnerworkingand
communica9ontoforegroundprocesses.FritsHoogland
1
This is the font size used for showing screen output. Be sure this is readable for you.
This is the font used to accentuate text/console output. Make sure this is readable for you too!
![Page 2: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/2.jpg)
$(whoami)
• FritsHoogland• WorkingwithOracleproductssince1996
• Blog:hFp://fritshoogland.wordpress.com• TwiFer:@fritshoogland• Email:[email protected]• OracleACEDirector• OakTableMember
2
![Page 3: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/3.jpg)
Author,togetherwithMar9nBach,KarlAraoandAndyColvin.
3
Books
Technicalreviewer:
![Page 4: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/4.jpg)
Whatisthispresenta9onabout?
SQL>commit;
4
What happens between the foreground and the LGWR on commit in polling mode?
Specifically: how do these communicate.Commitcomplete.
![Page 5: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/5.jpg)
Whatisthispresenta9onabout?
It’snotnotthedes9na9onbutthejourneythatmaFers.
Summaryofthepoem“Ithaca”byConstan9neCavafy.
5
![Page 6: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/6.jpg)
Warning
• LookingattheinnerworkingofOracle.– Alotofthisisundocumented.– Thismeanslookingat(in)theOracleexecutableando/sresources.
• Thetechniquesusedarenomethodsfordailyadministra9ontasks.– Rathertechniquestobeusedinspecialis3cedgecases.
• Usingthesetechniqueswrongcanhavesevereconsequences(instancedown,corrup6on)!
6
![Page 7: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/7.jpg)
Prerequisites
• BasicunderstandingofhowtheprocessesofanOracledatabaseworkandcommunicate.
• BasicunderstandingofCcodingandbasicflowofexecu9on.
• Understandingofthelogicofredoandredoconceptsfortheforegroundandlogwriterprocesses.
• Thisisnotahighleveloverview.Thisisamicroscopiclookintotheinnerworking.
7
Final scare away warning!
![Page 8: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/8.jpg)
Testsystem
• Thetestsandinves9ga9onisdoneinaVM:– Host:MacOSX10.11.3/VMWareFusion7.1.3.– VM:OracleLinuxx86_647u2(3.10.0-123.el7.x86_64).– Oracledatabase12.1.0.2.
• Thetestsinthispresenta9onaredonewiththedefaultsemngsfor:– COMMIT_LOGGING(immediate),COMMIT_WAIT(wait)andCOMMIT_WRITE*.
8
![Page 9: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/9.jpg)
Recap
• Followingisasummaryfrommy‘profilingthelogwriteranddatabasewriter’presenta9on.
• TherearetwomethodsforaforegroundprocesstounderstanditsredohasbeenwriFen:– Post/wait– Polling
9
![Page 10: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/10.jpg)
Logwriter,commit-post-wait
10
time
foreground
logwritersemtimedop(458755, {{15, -1, 0}}, 1, {3, 0})
semctl(458755, 15, SETVAL, 0x7fff00000001)
commit;
io_submit(139981752844288, 1, {{0x7f5008e23480, 0, 1, 0, 256}})
io_getevents(139981752844288, 1, 128, {{0x7f5008e23480, 0x7f5008e23480, 3584, 0}},
{600, 0})
log file parallel writerdbms ipc message
kcrf_commit_force()
kcscur3()
semtimedop(458755, {{33, -1, 0}}, 1, {0, 100000000})
log file sync Grayed means: optional
>=12.1.0.1: kcrf_commit_force_int()
![Page 11: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/11.jpg)
©2013Enkitec11
![Page 12: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/12.jpg)
©2013Enkitec12
![Page 13: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/13.jpg)
©2013Enkitec13
![Page 14: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/14.jpg)
©2013Enkitec14
![Page 15: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/15.jpg)
©2013Enkitec15
![Page 16: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/16.jpg)
Logwriter,commit-polling
16
time
foreground
logwritersemtimedop(458755, {{15, -1, 0}}, 1, {3, 0})
semctl(458755, 15, SETVAL, 0x7fff00000001)
commit;
io_submit(139981752844288, 1, {{0x7f5008e23480, 0, 1, 0, 256}})
io_getevents(139981752844288, 1, 128, {{0x7f5008e23480, 0x7f5008e23480, 3584, 0}},
{600, 0})
log file parallel writerdbms ipc message
kcrf_commit_force
kcscur3
log file sync
nanosleep({0, 9409000}, 0x7fff64725480)
nanosecs; varies
>=12.1.0.1: kcrf_commit_force_int()
![Page 17: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/17.jpg)
©2013Enkitec17
![Page 18: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/18.jpg)
©2013Enkitec18
![Page 19: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/19.jpg)
©2013Enkitec19
![Page 20: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/20.jpg)
©2013Enkitec20
![Page 21: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/21.jpg)
Logwriterpollingmode
• Theredoesnotseemtobeanyno9fica9onfromtheLGWRtotheforegroundprocess.– Whichmustmean‘polling’meanstheforegroundprocesspollssomethingtoseeifitslogbuffercontentsarewriFen.
21
![Page 22: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/22.jpg)
Logwriterpollingmode
• previously,Idiscoveredsomefunc9ons*beingcalledintheforegroundprocessatersemctl():– kcrf_commit_force()/kcrf_commit_force_int()asthemainrou9neinresponsibleformakingsuretheredocontentsarewriFenbytheLGWR.
– nanosleep()asawayofsuspendingexecu9onforaFIXEDperiodof9me.
– kcscur3()asafunc9onthatdoes“something”,probablyscanningthecommitSCN?– Alsousedp/w,edgecaseconsistentw/assump9on.
22
![Page 23: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/23.jpg)
Logwriterpollingmode
• BasedontheobservedpaFern:– semctl(tosignaltheLGWR)– kcrf_commit_force_int(maincheckloop)– kcscur3(supposedLGWRprogresschecking)– nanosleep(spendacalculated9meoffCPU)
• Iassumedthatkcscur3ischeckingthecommitSCN.
• JonathanLewistheorisedthatalltheFGneededtodowaskeeptrackofthewritestatusofitsblocksinthepublicredobuffer.
23
![Page 24: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/24.jpg)
• Sotheques9onis:
HowdoesaFGprocessinpollingmodedeterminethatitspubliclogbuffercontentsarewriFentodisk?
24
![Page 25: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/25.jpg)
• Whatinforma9ondowehave?
• kcscur3()• …nothingelse
25
![Page 26: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/26.jpg)
• Wedon’thavesourcecodenordebuginforma9onfromtheOracleexecutable.
• Wecanfetchthefunc9onarguments:• LinuxX86_64followstheAMD64ABI• Whichmeansfunc9onargumentsarepassedviaCPUregisters:• RDI,RSI,RDX,RCX,R8,R9
• Wedonotknowthenumberofarguments.
26
![Page 27: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/27.jpg)
kcscur3
• Let’sprofiletheforegroundsession,andprinttheargumentsofkcscur3func9on.
27
![Page 28: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/28.jpg)
©2013Enkitec28
![Page 29: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/29.jpg)
kcscur3
• Thisshowsanoddity:– Theforegroundimmediatelycon9nues.
• Nanosleepdoesn’tgetcalled.• Thismeansthe‘logfilesync’waitisomiFedtoo!
• Let’sslowdowntheLGWR!– Inordertodothat,I’lladdasleepof10mstotheIOreap(=io_geteventscall)ofthelogwriterprocess.
29
![Page 30: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/30.jpg)
©2013Enkitec30
![Page 31: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/31.jpg)
kcscur3
• Par9aloutputofbreaks,oncommit:Breakpoint 4, 0x00007ffc796a1a40 in semctl () from /lib64/libc.so.6
Breakpoint 2, 0x000000000cc5edd0 in kcrf_commit_force_int ()
kcscur3 60027c98, 7bfe4430, 1, 634
kcscur3 60027c68, 7bfe4398, 0, 0
Breakpoint 1, 0x00007ffc79b8f940 in nanosleep () from /lib64/libpthread.so.0
kcscur3 60027c98, 7bfe4430, 1, 94c7f210
kcscur3 60027c68, 7bfe4398, 0, 0
Breakpoint 1, 0x00007ffc79b8f940 in nanosleep () from /lib64/libpthread.so.0
kcscur3 60027c98, 7bfe4430, 1, 94c7f210
kcscur3 6001fbb0, 7bfe4988, 1, 7d950090
31
There are 3 addresses as the first argument to kcscur3(): 0x60027c98, 0x60027c68 and 0x6001fbb0.
![Page 32: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/32.jpg)
kcscur3
• Whataretheseaddresses?
• Let’sseeifthesearesharedmemoryaddresses:
32
![Page 33: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/33.jpg)
kcscur3
• Thesharedmemoryarea’sofanOracledatabaseareplacedinacoupleofsharedmemorysegments.
• Thesesharedmemorysegmentsaddressescanbedumpedwith:• oradebugipc
33
![Page 34: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/34.jpg)
©2013Enkitec34
![Page 35: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/35.jpg)
kcscur3Handle: 0x117fdea0 `/u01/app/oracle/product/12.1.0.2/dbhome_1fv12102'
Dump of unix-generic realm handle `/u01/app/oracle/product/12.1.0.2/dbhome_1fv12102', flags = 00000000
key 3512777704 actual_key 3512777704 num_areas 4 num_subareas 4
primary shmid: 753667 primary sanum 3 version 3
deferred alloc: FALSE (0) def_post_create: FALSE (0) exp_memlock: 1002M
Area #0 `Fixed Size' containing Subareas 2-2
Total size 00000000002cbe70 Minimum Subarea size 00000000
Area Subarea Shmid Segment Addr Stable Addr Actual Addr
0 2 655360 0x00000060000000 0x00000060000000 0x00000060000000
Subarea size Segment size Req_Protect Cur_protect
00000000002cc000 00000000002cc000 default readwrite
Area #1 `Variable Size' containing Subareas 0-0
Total size 0000000036000000 Minimum Subarea size 00400000
Area Subarea Shmid Segment Addr Stable Addr Actual Addr
1 0 688129 0x00000060400000 0x00000060400000 0x00000060400000
Subarea size Segment size Req_Protect Cur_protect
0000000036000000 0000000036000000 default readwrite
35
![Page 36: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/36.jpg)
FixedSGA
• ThefixedSGAvariablesarevisibleinx$ksmfsv– ThefixedSGAcontainsmorethanSGAvariables,likelatches*.
SQL> select ksmfsnam, ksmfsadr, ksmfssiz from x$ksmfsv
2 where to_number(‘60027c98’,’XXXXXXXX’)
3 between to_number(ksmfsadr,’XXXXXXXXXXXXXXXX’)
4 and to_number(ksmfsadr,’XXXXXXXXXXXXXXXX’)+ksmfssiz-1;
KSMFSNAM KSMFSADR KSMFSSIZ ------------------------------ ---------------- -------- kcrfsg_ 0000000060027C30 1608 -- and 60027c68: KSMFSNAM KSMFSADR KSMFSSIZ ------------------------------ ---------------- -------- kcrfsg_ 0000000060027C30 1608 -- and 6001fbb0: KSMFSNAM KSMFSADR KSMFSSIZ ------------------------------ ---------------- -------- kcsgscn_ 000000006001FBB0 48
36
![Page 37: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/37.jpg)
• Thetwoaddresses0x60027c98&0x60027c68• PointtoafixedSGAvariablecalled‘kcrfsg’
– Thisvariablestartsat0x60027c30– Thislikelyisac‘struct’,whichresemblesatable.
• Andtheaddress0x6001{b0• PointstoafixedSGAvariablecalled‘kcsgscn’
37
![Page 38: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/38.jpg)
• Okay,onestepata9me…
• Whatelsecanwesee?
• Whatvaluesdothesememoryloca9onscontain?
38
![Page 39: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/39.jpg)
©2013Enkitec39
![Page 40: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/40.jpg)
structkcrfsg&kcsgscn
• gdbxcommand:examine,/d=decimal
(gdb) x/d 0x60027c98
0x60027c98: 537122
(gdb) x/d 0x60027c68
0x60027c68: 537124
(gdb) x/d 0x6001fbb0
0x6001fbb0: 537126
• Whatarethesenumbers??SQL> select current_scn from v$database;
CURRENT_SCN
-----------
537136
40
![Page 41: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/41.jpg)
• That’stooclosetobeacoincidence!– ItlooksliketheseallcontainSCNs!– Anothersmallsteptaken.
41
![Page 42: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/42.jpg)
FixedSGAvariablekcsgscn
• KCSGSCN(aliasaddress0x6001{b0)– KCS-probablyKernelCacheService– G-global?group?– SCN-probablySCN;SystemChangeNumber
• Awayofdetec9ngusageofkcsgscnisusingawatchpoint.
42
![Page 43: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/43.jpg)
oradebugwatchpoint
43
![Page 44: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/44.jpg)
©2013Enkitec44
![Page 45: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/45.jpg)
©2013Enkitec45
select current_scn from v$database showed: 771067.
That’s in hex: 0xbc3fb
This is : 0xfbc30b —> should the ‘f’ be moved to the place of the ‘0’????
oradebug watchpoints are NOT documented and gives ‘wierd’ results…
![Page 46: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/46.jpg)
gdbwatchpoint
46
![Page 47: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/47.jpg)
©2013Enkitec47
![Page 48: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/48.jpg)
FixedSGAvariablekcsgscn
• Thewatchpointshowscurrent_scnusingthisvariable
• Inthepreviousgdbwatchpointwesawthefunc9onkcsgbsn()accessingit.– kcsgbsn=kernelcacheservicegetbatchedSCN
• kcsgscncontainsthecurrentSCNfortheinstance.
48
![Page 49: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/49.jpg)
structkcrfsg
• Let’sseeoftherearex$viewsthatresemblethestructname:
SQL> select name from v$fixed_table where upper(name) like upper(‘%kcrf%’);
NAME
------------------------------
X$KCRFWS
X$KCRFSTRAND
X$KCRFDEBUG
X$KCRFX
49
![Page 50: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/50.jpg)
structkcrfsgSQL> select addr from x$kcrfx;
no rows selected
SQL> select addr from x$kcrfdebug;
ADDR
----------------
0000000060028828
SQL> select addr from x$kcrfstrand;
ADDR
----------------
00007F68F1F195C0
00007F68F1F195C0
SQL> select addr from x$kcrfws;
ADDR
----------------
0000000060027C38
50
Close, but past kcrfsg (0x60027c30) and 0x60027c98 and 0x60027c68.
These are PGA memory addresses (the high ones structurally are). And this makes sense with X$KCRFSTRAND, which probably has to do with private redo strands.
Bingo! 8 bytes past kcrfsg (0x60027c30) and before 0x60027c98 and 0x60027c68.
![Page 51: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/51.jpg)
X$KCRFWS
• SowegotafixedSGAvariablecalledkcrfsg_• Whichis(quiteprobably)astructcalledkcrfsg• WhichisexternalisedbyX$KCRFWS
• IsX$KCRFWSusedina‘dynamicperformanceview’,aliasaV$view?
51
![Page 52: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/52.jpg)
X$KCRFWSSQL> select view_name from v$fixed_view_definition
2 where lower(view_definition) like '%kcrfws%';
VIEW_NAME
------------------------------
GV$XSTREAM_CAPTURE
• Streams??– Xstreamisthenextgenera9onstreamsusedbyOGG.– Actually,itmakessensethatstreams/OGGhaveastrongdependencyonredowritedetails!
52
![Page 53: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/53.jpg)
X$KCRFWS
• Streams??– SearchingtheinternetIfoundsitesmen9oningX$KCRFWSisrelatedtostreams.– Yes,aviewrelatedtostreamsusesit.
– Don’ttrusttheinternetun6lyouhaveverified!
• X$KCRFWSisallaboutredo.– MyguessisX$KCRFWSactuallymeans:
– “KernelCacheRedoFileWriteStatus”.
53
![Page 54: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/54.jpg)
X$KCRFWS Name Null? Type ----------------------------------------- -------- ---------------------------- ADDR RAW(8) INDX NUMBER INST_ID NUMBER CON_ID NUMBER NEXT_BLK NUMBER LAST_BLK NUMBER ON_DISK_SCN_BAS NUMBER ON_DISK_SCN_WRP NUMBER ON_DISK_PING_SCN_BAS NUMBER ON_DISK_PING_SCN_WRP NUMBER LAST_WRITE_SCN_BAS NUMBER LAST_WRITE_SCN_WRP NUMBER LWN_SCN_BAS NUMBER LWN_SCN_WRP NUMBER LAST_WRITE_SCN NUMBER LAST_WRITE_SCN_TIME DATE REAL_REDO_SCN_BAS NUMBER REAL_REDO_SCN_WRP NUMBER REAL_WRITE_TIME DATE
54
0x60027c38
Okay, what do we know? - We got a view called X$KCRFWS that describes redo writing. - It contains one record (in my case). - The starting address is in the ADDR field. - Which field(s) are 0x60027c68 and 0x60027c98?
} 0x60027c68?
0x60027c98?
![Page 55: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/55.jpg)
X$KCRFWS
• Ifirstthoughtusingthe“magicoffsettable”wouldbeaneasyway:
SQL> select c.kqfconam, c.kqfcooff
2 from x$kqfco c, x$kqfta t
3 where t.indx = c.kqfcotab
4 and t.kqftanam='X$KCRFWS'
5 order by c.kqfcooff;
55
KQFCONAM KQFCOOFF ------------------------------ ---------- ADDR 0 INDX 0 REAL_REDO_SCN_WRP 0 REAL_REDO_SCN_BAS 0 LWN_SCN_WRP 0 LWN_SCN_BAS 0 ON_DISK_PING_SCN_WRP 0 INST_ID 0 CON_ID 0 NEXT_BLK 0 ON_DISK_SCN_BAS 0 ON_DISK_SCN_WRP 0 ON_DISK_PING_SCN_BAS 0 LAST_BLK 4 LAST_WRITE_SCN_TIME 12 LAST_WRITE_SCN_BAS 144 LAST_WRITE_SCN 144 LAST_WRITE_SCN_WRP 148 REAL_WRITE_TIME 328
0x60027c68-0x60027c38= 48 0x60027c98-0x60027c38= 96
Both offsets are not in the offset table…
![Page 56: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/56.jpg)
X$KCRFWS
• Thenitneedsamore“hardcore”approach…
• Forthisawatchpointcanbeusedtoo.• Awatchpointbreaksexecu9onifthespecifiedaddressisread,wriFenorboth.
56
![Page 57: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/57.jpg)
X$KCRFWS
• Nowforthetricktofindthefieldthatbelongto0x60027c68and0x60027c98:
– Putareadwatchpointontheaddresses.– QueryX$KCRFWSfieldbyfieldun9lithitsthewatchpoint.
57
![Page 58: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/58.jpg)
©2013Enkitec58
![Page 59: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/59.jpg)
X$KCRFWS
• Address0x60027c68:– LWN_SCN_BASandLWN_SCN_WRP
• LWN:LogWriteNumber;agroupofredoblockstobewriFenbytheLGWRisappointedanumbercalledLWN.
• LWNSCN:Thepoten9almaximumSCNinthecurrentLWN.
• Address0x60027c98:– ON_DISK_SCN_BASandON_DISK_SCN_WRP
• OndiskSCN:thehighestSCNthatthedatabasecanberecoveredtowithwriFenredo.
59
![Page 60: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/60.jpg)
kcscur3
• Backtotheoriginalinves9ga9on.– Thefirstargumentofkcscur3isactuallyavarietyofSCNnumbers:• 0x60027c68:LWNSCN• 0x60027c98:OndiskSCN• 0x6001{b0:global(current)SCN
– So:thefunc9onkcrf_commit_force(_int)checksdifferentSCNvaluesusingkcscur3duringcommit.
• Let’slookatwhatishappeningduringcommitagain:
60
![Page 61: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/61.jpg)
©2013Enkitec61
![Page 62: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/62.jpg)
©2013Enkitec62
kcrf_commit_force_int -> 1st function: writing into the public log buffer and semctl’ing the LGWR
kcrf_commit_force_int -> 2nd function: check log writer progress, and go to sleep if not progressed far enough
check the log writer progress, and sleep if not progressed far enough
check the log writer progress, and sleep if not progressed far enough
here is detected that the LGWR progressed writing far enough. mind the kcscur3() call to 0x6001fbb0
![Page 63: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/63.jpg)
Yetanotherstep…
• So,wenowknowthatthecommimngprocesscheckstheon-diskandLWNSCN.– Ithinktheon-diskSCNisusedbytheforegroundprocesstocheckforredowriteprogressinpollingmode(andpost/waitincertaincases).
• Obviously,anotherprocessmustchangetheon-diskSCN.– Thatprocessisquitelikelythelogwriter.– OrtheLGWRslaves,whichIdisabledforthesakeofclarity.
63
![Page 64: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/64.jpg)
Yetanotherstep…
• So,Isuspectthelogwriter:– Writesthelogbuffer.– Thenupdatestheon-diskSCNtoindicatewriteprogress.
• TounderstandwhattheLGWRdoeswecan:– Putaread/writewatchpointon
– 0x60027c68(LWNSCN)– ox60027c98(ondiskSCN)
– ToseewhattheLGWRisdoing.
64
![Page 65: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/65.jpg)
©2013Enkitec65
![Page 66: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/66.jpg)
watch *0x60027c68
watch *0x60027c98
commands 1-2
c
end
break kcsnew3
break kcscur3
break kcsadj3
break io_submit
break io_getevents_0_4
commands 3-7
silent
output $rip
printf “\t%x, %x, %x, %x\n”, $rdi, $rsi, $rdx, $rcx
c
end
break semtimedop
silent
printf “semtimedop\n”
c
end 66
![Page 67: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/67.jpg)
• Logwriterwake-upfunc9onswithoutwrite:(void (*)()) 0xcc61670 <kcsnew3> 6001fbb0, a9060900, 60027c68, 3b37e000
Hardware watchpoint 2: *0x60027c68
Old value = 780329
New value = 780330
0x000000000cc61754 in kcsnew3 ()
(void (*)()) 0xcc61600 <kcscur3> 60027c98, a906073c, 1, 0 (void (*)()) 0xcc61600 <kcscur3> 60027c68, a9060708, 1, 0
(void (*)()) 0x2d5aed0 <kcsadj3> 60027c98, a9060708, 0, 0
Hardware watchpoint 1: *0x60027c98
Old value = 780329
New value = 780330 0x0000000002d5af53 in kcsadj3 ()
(void (*)()) 0xcc61600 <kcscur3> 6001fbb0, a9060270, 1, 79291280
67
Every wake-up of the log writer increases the LWN SCN using the kcsnew3 function.
Also, the on-disk SCN is increased.
Please mind there is nothing written in the online redo log!
![Page 68: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/68.jpg)
• Logwriterwake-upfunc9onswithwrite:(void (*)()) 0xcc61670 <kcsnew3> 6001fbb0, a9060aa0, 60027c68, 3b37e000
Hardware watchpoint 2: *0x60027c68
Old value = 780330
New value = 780350
0x000000000cc61754 in kcsnew3 ()
(void (*)()) 0x2d5aed0 <kcsadj3> 60027d50, a9060860, 0, 0 (void (*)()) 0x7f4a75e0d690 <io_submit> 7929f000, 1, a9059360, 8f24bbe8
(void (*)()) 0x7f4a75e0d650 <io_getevents> 7929f000, 1, 80, a905f1e8
(void (*)()) 0xcc61600 <kcscur3> 60027c98, a906083c, 1, 0 (void (*)()) 0x2d5aed0 <kcsadj3> 60027c98, 92fac724, a9060834, a906082c
Hardware watchpoint 1: *0x60027c98
Old value = 780330
New value = 780350
0x0000000002d5af53 in kcsadj3 () (void (*)()) 0xcc61600 <kcscur3> 6001fbb0, a9060270, 1, 79291280
68
If the log writer writes something, the on-disk SCN is increased too.
Probably not all SCNs need writing.
Look! There is another kcsadj3 call. This call has 0x60027d50 as first argument.
0x60027d50 is REAL_REDO_SCN_(BAS|WRP)
The LWN and on-disk SCNs are increased when the log writer writes too, as expected.
Probably not all SCNs need writing.
![Page 69: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/69.jpg)
• So,whatIthinkishappeningis:• AFGsessioncommitsandnotescommitSCN.• FGsemctl’sLGWRtowrite*.• Thencheckson-diskSCNifSCNincreasedbeyonditscommitSCN,thennanosleep().
• LGWRdeterminesLWNSCN.• WritesblocksintheLWNbatch.• Updateson-diskSCN.• FGreadsupdatedon-diskSCNandcon9nues.
69
![Page 70: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/70.jpg)
• HowtovalidateaFGjustcheckstheon-diskSCN?(void (*)()) 0xcc61670 <kcsnew3> 6001fbb0, a9060aa0, 60027c68, 3b37e000
Hardware watchpoint 2: *0x60027c68
Old value = 780330
New value = 780350
0x000000000cc61754 in kcsnew3 ()
(void (*)()) 0x2d5aed0 <kcsadj3> 60027d50, a9060860, 0, 0 (void (*)()) 0x7f4a75e0d690 <io_submit> 7929f000, 1, a9059360, 8f24bbe8
(void (*)()) 0x7f4a75e0d650 <io_getevents> 7929f000, 1, 80, a905f1e8
(void (*)()) 0xcc61600 <kcscur3> 60027c98, a906083c, 1, 0 (void (*)()) 0x2d5aed0 <kcsadj3> 60027c98, 92fac724, a9060834, a906082c
Hardware watchpoint 1: *0x60027c98
Old value = 780330
New value = 780350
0x0000000002d5af53 in kcsadj3 () (void (*)()) 0xcc61600 <kcscur3> 6001fbb0, a9060270, 1, 79291280
70
At this point the on-disk SCN is updated. This supposedly makes the FG to continue if the SCN is high enough.
So what if we make the LGWR stop here?
The FG should keep on waiting.
While the LGWR is suspended at this point, the FG waits.
We can prove the FG only waits for the on-disk SCN, if the FG continues when we: - Keep the LGWR suspended, but - MANUALLY update the on-disk SCN
![Page 71: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/71.jpg)
• Thefollowingtechniquesareforexperimen9ngandinves9ga9onONLY.
• Doingthisonareal,livedatabasecouldcausecorrup3onorlossoftheen3redatabase!
WARNING
71
![Page 72: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/72.jpg)
©2013Enkitec72
![Page 73: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/73.jpg)
©2013Enkitec73
![Page 74: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/74.jpg)
©2013Enkitec74
![Page 75: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/75.jpg)
©2013Enkitec75
![Page 76: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/76.jpg)
• Aforegroundsessionhastwocommitmodes:• Post/wait,thetradi9onalway.• Polling,thenewmethod.
• Aforegroundsessionno9fiestheLGWRbyexecu9nga‘semctl’call.• Nono9fica9onnecessaryifLGWRalreadyprogressedbeyondFGcommitSCN.
Conclusion
76
![Page 77: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/77.jpg)
• ThereareacoupleofSCNvaluesthedatabasekeepsinthefixedSGA:• kcbgscn,global/currentSCN,0x6001{b0• kcrfsg,LWNSCN,0x60027c68• kcrfsg,on-diskSCN,0x60027c98• kcrfsg,realredoSCN,0x60027d50
• Thekcscur3func9onseemstobethefunc9ontoreadthesevariables.
Conclusion
77
![Page 78: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/78.jpg)
• Theforegroundprocessusesthekcrf_commit_force(_int)func9onto:• Flushitsredodataintothepubliclogbuffer.• Checklogwriterprogressviatheon-diskSCN.
• Thisofcourseisinpollingmode.– Withpost/wait,theon-diskSCNischeckedtoo!
Conclusion
78
![Page 79: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/79.jpg)
• Thelogwriterhasacertaincycleevery3sec:• ReadcurrentSCNandLWNSCN.• UpdateLWNSCN.• Ifneeded:updaterealredoSCNandwriteoutpubliclogbuffer.
• Updateon-diskSCN.
• ItseemstheSCNsetasLWNSCNatthebeginningofthecycle,isequaltotheon-diskSCN.
Conclusion
79
![Page 80: drilldown into the logwriter inner working and communication](https://reader034.vdocument.in/reader034/viewer/2022042707/58a2cadb1a28abbe5a8bbf7b/html5/thumbnails/80.jpg)
• LWNandon-diskSCNsprogressevenifthereisnoredowriFenfromlogbuffertodisk.
• TheSCNofthelatestredotrulywriFentodiskisintherealredoSCN.
Conclusion
80