tips for performance bottlenecks

8/2/2019 Tips for Performance Bottlenecks

1/25

Aboutthisdocument

Thisdocumentdescribeshowtocheckforresourcebottlenecksandidentifytheprocessesthattaxthem.Resourcesonasystemincludememory,CPU,andInput/Output(I/O).Thisdocumentcoversbottlenecksacrossanentiresystem.Thisdocumentdoesnotaddressthebottlenecksofaparticularapplicationorgeneralnetworkproblems.Thefollowingcommandsaredescribed:

*vmstat*svmon*ps*tprof*iostat*netpmon*filemon

NOTE:PAIDE/6000mustbeinstalledinordertousetprof,svmon,netpmon,andfilemon.Tocheckifthisisinstalled,runthefollowingcommand:

lslpp-lperfagent.tools.

IfyouareatAIXVersion4.3.0orhigher,PAIDE/6000canbefoundontheAIXBaseOperatingSystemmedia.Otherwise,toorderPAIDE/6000,contact

yourlocalIBMrepresentative.

Thisfaxalsomakesreferencetothevmtuneandschedtunecommands.Thesecommandsandtheirsourcearefoundinthe/usr/samples/kerneldirectory.Theyareinstalledwiththebos.adt.samplesfileset.

Relatedinformation

ConsultLinePerformanceAnalysis-TheAIXSupportFamilyoffersasystemanalysiswithtuningrecommendations.FormoreinformationcalltheIBMAIXSupportCenter.

PerformanceTuningGuide(SC23-2365)-ThisIBMpublicationcovers

performancemonitoringandtuningofAIXsystems.ContactyourlocalIBMrepresentativetoorder.

Fordetailedsystemusageonaperprocessbasis,afreeutlitycalledUTLDcanbeobtainedbyanonymousftpfromftp.software.ibm.cominthe/aix/tools/perftools/utlddirectory.FormoreinformationseetheREADMEfile/usr/lpp/utldafterinstallationoftheutld.objfileset.

------------------------------------------------------------------------

Memorybottlenecks

Thefollowingsectiondescribesmemorybottlenecksolutionswiththe

followingcommands:vmstat,svmon,ps.

1.vmstat

Runthefollowingcommand:

vmstat1

NOTE:Systemmayslowdownwhenpiandpoareconsistentlynon-zero.


2/25

pinumberofpagespersecondpagedinfrompagingspaceponumberofpagespersecondpagedouttopagingspace

WhenprocessesonthesystemrequiremorepagesofmemorythanareavailableinRAM,workingpagesmaybepagedouttopagingspaceandthenpagedinwhentheyareneededagain.AccessingapagefrompagingspaceisconsiderablyslowerthanaccessingapagedirectlyfromRAM.Forthisreason,constantpagingactivitycancausesystemperformancedegradation.

NOTE:Memoryisover-committedwhenthefr:srratioishigh.

frnumberofpagesthatmustbefreedtoreplenishthefreelistortoaccommodateanactiveprocesssrnumberofpagesthatmustbeexaminedinordertofreefrnumberofpages

Anfr:srratioof1:4meansforeveryonepagefreed,fourpagesmustbeexamined.Itisdifficulttodetermineamemoryconstraintbasedon

thisratioaloneandwhatconstitutesahighratioisworkload/applicationdependent.

Thesystemconsidersitselftobethrashingwhenpo*SYS>frwhereSYSisasystemparameterviewedwiththeschedtunecommand.Thedefaultvalueis0ifasystemhas128MBormorewhichmeansthatmemoryloadcontrolisdisabled.Otherwise,thedefaultis6.Thrashingoccurswhenthesystemspendsmoretimepagingthanperformingwork.Whenthisoccurs,selectedprocessesmaybesuspendedtemporarily,andthesystemmaybenoticeablyslower.

2.svmon

Asroot,runthefollowingcommand:

#svmon-Pau10|more

Sampleoutput:

PidCommandInusePinPgspace13794dtwm16031449Pid:13794Command:dtwmSegidTypeDescriptionInusePinPgspaceAddressRangeb23pers/dev/hd2:248492000..114a5pers/dev/hd2:248420000..2

6179worklibdata1310980..891280aworksharedlibrarytext11010100..65535181workprivate28713410..310:65277..6553557d5perscode,/dev/hd2:6172282000..135

Thiscommandliststhetoptenmemoryusingprocessesandgivesareportabouteachone.Ineachprocessreport,lookwhereType=workandDescription=private.Checkhowmany4K(4096byte)pagesareusedunderthePgspacecolumn.Thisistheminimumnumberofworkingpagesthissegmentisusinginallofvirtualmemory.APgspacenumber


3/25

thatgrows,butneverdecreases,mayindicateamemoryleak.Memoryleaksoccurwhenanapplicationfailstodeallocatememory.

341*4096=1,396,736or1.4MBofvirtualmemory

3.ps


psgv|head-n1;psgv|egrep-v"RSS"|sort+6b-7-n-r

sizeamountofmemoryinKBallocatedfrompagespaceforthememorysegmentofType=workandDescription=privatefortheprocessaswouldbeindicatedbysvmonRSSamountofmemoryinKBcurrentlyinuse(inRAM)forthememorysegmentofType=workandDescription=privateplusthememorysegment(s)ofType=persandDescription=codefortheprocessaswouldbeindicatedbysvmontrsamountofmemory,inKB,currentlyinuse(inRAM)forthememorysegment(s)ofType=persandDescription=codefortheprocessaswouldbeindicatedbysvmon

%memRSSvaluedividedbythetotalamountofsystemRAMinKBmultipliedby100

------------------------------------------------------------------------

CPUbottlenecks

ThefollowingsectiondescribesCPUbottlenecksolutionsusingthefollowingcommands:vmstat,tprof,ps.

1.vmstat


vmstat1

NOTE:Systemmayslowdownwhenprocesseswaitontherunqueue.

idpercentageoftimetheCPUisidlernumberofthreadsontherunqueue

Iftheidvalueisconsistently0%,theCPUisbeingused100%ofthetime.

Looknextatthercolumntoseehowmanythreadsareplacedontherunqueuepersecond.Thehigherthenumberofthreadsforcedtowaitontherunqueue,themoresystemperformancewillsuffer.

2.tprof

TofindouthowmuchCPUtimeaprocessisusing,runthefollowingcommandasroot:


4/25

#tprof-xsleep30

Thisreturnsin30secondsandcreatesafileinthecurrentdirectorycalled__prof.all.

In30seconds,theCPUischeckedapproximately3000times.TheTotalcolumnisthenumberoftimesaprocesswasfoundintheCPU.Ifoneprocesshas1500intheTotalcolumn,thisprocesshastaken1500/3000orhalfoftheCPUtime.ThetprofoutputexplainsexactlywhatprocessestheCPUhasbeenrunning.ThewaitprocessrunswhennootherprocessesrequiretheCPUandaccountsfortheamountofidletimeonthesystem.

3.netpmon

TofindouthowmuchCPUtimeaprocessisusing,andhowmuchofthattimeisspentexecutingnetwork-relatedcode,runthefollowingcommandasroot:

#netpmon-o/tmp/netpmon.out-Ocpu-v;sleep30;trcstop

Thisreturnsin30secondsandcreatesafileinthe/tmpdirectorycallednetpmon.out.

TheCPUTimeindicatesthetotalamountofCPUtimefortheprocess.%CPUisthepercentageofCPUusagefortheprocess,andNetworkCPU%isthepercentageoftotaltimethattheprocessspentexecutingnetwork-relatedcode.

4.ps

Runthefollowingcommands:

ps-ef|head-n1ps-ef|egrep-v"UID|0:00|\0\"|sort+3b-4-n-r

ChecktheCcolumnforaprocesspenaltyforrecentCPUusage.The

maximumvalueforthiscolumnis120.

ps-e|head-n1ps-e|egrep-v"TIME|0:"|sort+2b-3-n-r

ChecktheTimecolumnforprocessaccumulatedCPUtime.

psgupsgu|egrep-v"CPU|kproc"|sort+2b-3-n-r

Checkthe%CPUcolumnforprocessCPUdependency.ThepercentCPUisthetotalCPUtimedividedbythethetotalelapsedtimesincetheprocesswasstarted.

------------------------------------------------------------------------

I/0bottlenecks

Thissectiondescribesbottlenecksolutionsusingthefollowingcommands:iostat,filemon.

1.iostat


5/25

NOTE:Highiowaitwillcauseslowerperformance.


iostat5

%iowaitpercentageoftimetheCPUisidlewhilewaitingonlocalI/O%idlepercentageoftimetheCPUisidlewhilenotwaitingonlocalI/O

ThetimeisattributedtoiowaitwhennoprocessesarereadyfortheCPUbutatleastoneprocessiswaitingonI/O.AhighpercentageofiowaittimeindicatesthatdiskI/Oisamajorcontributortothedelayinexecutionofprocesses.Ingeneral,ifsystemslownessoccursand%iowaitis20%to25%orhigher,investigationofadiskbottleneckisinorder.

%tm_actpercentageoftimethediskisbusy

NOTE:Hightm_actpercentagecanindicateadiskbottleneck.

When%tm_actortimeactiveforadiskishigh,noticeableperformance

degradationcanoccur.Onsomesystems,a%tm_actof35%orhigherforonediskcancausenoticeablyslowperformance.

oLookforbusyvs.idledrives.Movingdatafrommorebusytolessbusydrivesmayhelpalleviateadiskbottleneck.oCheckforpagingactivitybyfollowingtheinstructionsinthe"Memorybottlenecks"section.PagingtoandfromdiskwillcontributetotheI/Oload.

2.filemon

Tofindoutwhatfiles,logicalvolumes,anddisksaremostactive,runthefollowingcommandasroot:

#filemon-u-Oall-o/tmp/fmon.out;sleep30;trcstop

In30seconds,areportiscreatedin/tmp/fmon.out.

oCheckformostactivesegments,logicalvolumes,andphysicalvolumesinthisreport.oCheckforreadsandwritestopagingspacetodetermineifthediskactivityistrueapplicationI/Oorisduetopagingactivity.oCheckforfilesandlogicalvolumesthatareparticularlyactive.Iftheseareonabusyphysicalvolume,movingsomedatatoalessbusydiskmayimproveperformance.

TheMostActiveSegmentsreportliststhemostactivefilesbyfilesystemandinode.Themountpointofthefilesystemandinodeofthefilecanbeusedwiththencheckcommandtoidentifyunknownfiles:

#ncheck-i

Thisreportisusefulindeterminingiftheactivityistoafilesystem(segtype=persistent),theJFSlog(segtype=log),orto


6/25

pagingspace(segtype=working).

Byexaminingthereadsandreadsequencescounts,youcandetermineiftheaccessissequentialorrandom.Asthereadsequencescountapproachesthereadscount,fileaccessismorerandom.Thesameappliestothewritesandwritesequences.

------------------------------------------------------------------------

SMPperformancetuning

Performancetools

1.SMPonly(someSMPsdonotsupportthiscommand)

cpu_state-lThisdisplaysthecurrentstateofeachprocessor(enabled,disabled,orunavailable).

2.AIXtoolsthathavebeenadaptedinordertodisplaymoremeaninfulinformationonSMPsystemsps-m-oTHREADTheBNDcolumnwillindicatetheprocessornumbertowhichaprocess/threadisbound,ifitisbound.

pstat-ATheCPUIDcolumnwillindicatetheprocessornumbertowhichaprocess/threadisbound.

sar-PALLLoadonalltheprocessors.

vmstatDisplayskthr(kernelthreads).

netpmon-tPrintsCPUreportsonaper-threadbasis.

3.OtherAIXtoolsthatdidnotchange

filemoniostatsvmontprof

------------------------------------------------------------------------

Tuningmethodology

1.Checkavailablityofprocessors.

cpu_state-l

2.Checkbalancebetweenprocessors.

sar-PALL

3.Identifyboundprocesses/threads.

ps-m-oTHREAD


7/25

pstat-A

4.Unbindanyboundprocesses/threadsthatcanandshouldbeunbound.

5.Continueaswithuniprocessorsystem.

------------------------------------------------------------------------

Additionalinformation

1.KBUFFERSvs.VMM

TheBlockI/OBufferCache(KBUFFERS)isonlyusedwhendirectlyaccessingablockdevice,suchas/dev/hdisk0.NormalaccessthroughtheJournaledFileSystem(JFS)ismanagedbytheVirtualMemoryManager(VMM)anddoesnotusethetraditionalmethodforcachingthedatablocks.AnyI/OoperationstorawlogicalvolumesorphysicalvolumesdoesnotusetheBlockI/OBufferCache.

2.I/OPacing

UsersofAIXoccasionallyencounterlonginteractive-applicationresponsetimeswhenotherapplicationsinthesystemarerunninglargewritestodisk.Becausemostwritesareasynchronous,FIFOI/Oqueues

ofseveralmegabytesmaybuildupandtakeseveralsecondstocomplete.Theperformanceofaninteractiveprocessisseverelyimpactedifeverydiskreadspendsseveralsecondsworkingthroughthequeue.I/OpacinglimitsthenumberofoutstandingI/Orequestsagainstafile.Whenaprocesstriestowritetoafilewhosequeueisatthehigh-watermark(shouldbeamultipleof4plus1),theprocessissuspendeduntilenoughI/Oshavecompletedtobringthequeueforthatfiletothelow-watermark.Thedeltabetweenthehighandlowwatermarksshouldbekeptsmall.

ToconfigureI/OpacingonthesystemviaSMIT,enterthefollowingatthecommandlineasroot:

#smittychgsys

3.AsyncI/O

AsyncI/Oisperformedinthebackgroundanddoesnotblocktheuserprocess.ThisimprovesperformancebecauseI/Ooperationsandapplicationprocessingcanrunconcurrently.However,applicationsmustbespecificallywrittentotakeadvantageofAysncI/Othatismanagedbytheaiodaemonsrunningonthesystem.

ToconfigureAsyncI/OforthesystemviaSMIT,enterthefollowingatthecommandlineasroot:

#smittyaio

[TechDocsRef:90605198014824PublishDate:Dec.15,19994FAXRef:2445]PerformanceTuning--ThevmstatTool

------------------------------------------------------------------------

Contents


8/25

AboutthisdocumentRelateddocumentationAboutvmstatSummarystatistics------------------------------------------------------------------------

Aboutthisdocument

Thisdocumentprovidesanoverviewoftheoutputofthevmstat.ThisinformationappliestoAIXVersions4.x.

Relateddocumentation

Thefieldsproducedbythes,f,and[Drives]optionsofvmstatarefullydocumentedintheAIXPerformanceTuningGuide,publicationnumberSC23-2365,andintheonlineproductdocumentation.

TheAIXandRS/6000productdocumentationlibraryisalsoavailable:

www.rs6000.ibm.com/library/------------------------------------------------------------------------

Aboutvmstat

Althoughasystemmayhavesufficientrealresources,itmayperformbelowexpectationsiflogicalresourcesarenotallocatedproperly.

Usevmstattodeterminerealandlogicalresourceutilization.Itsampleskerneltablesandcountersandthennormalizestheresultsandpresentstheminanappropriateformat.

Bydefault,vmstatsendsitsreporttostandardout,butitcanberunwiththeoutputredirected.

vmstatisnormallyinvokedwithanintervalandacountspecified.Theintervalisthelengthoftimeinsecondsoverwhichvmstatistogatherandreportdata.Thecountisthenumberofintervalstorun.Ifno

parametersarespecified,vmstatwillreportasinglerecordofstatisticsforthetimesincethesystemwasbooted.Theremayhavebeeninactivityorfluctuationsintheworkload,sotheresultsmaynotrepresentcurrentactivity.Beawarethatthefirstrecordintheoutputpresentsstatisticssincethelastboot(exceptwheninvokedwiththe-for-soption).Inmanyinstances,thisdatacanbeignored.

vmstatreportsstatisticsaboutprocesses,virtualmemory,pagingactivity,faults,CPUactivity,anddisktransfers.Optionsandparametersrecognizedbythistoolareindicatedbytheusageprompt:

vmstat[-fs][Drives][Interval][Count]

Thefollowingfigurelistsoutputwherethesmallestworkunitiscalledakernelthread(kthr).Therandbunderthiscolumnrepresentthenumberof"threads",notprocesses,placedonthesequeues.

--------------------------------------------------------------------|||kthrmemorypagefaultscpu||-------------------------------------------------------||rbavmfrerepipofrsrcyinsycsussyidwa||0067471253000000114102201260|


9/25

|106747125300000011311843174790||0067471253000000118993383890|||--------------------------------------------------------------------Figure:Sampleoutputfromvmstat13

kthr

Thecolumnsunderthekthrheadingintheoutputprovideinformationabouttheaveragenumberofthreadsonvariousqueues.

r

Thercolumnindicatestheaveragenumberofkernelthreadsontherunqueueatone-secondintervals.

Thisfieldindicatedthenumberof"run-able"threads.Thesystemcountsthenumberofready-to-runthreadsoncepersecondandaddsthatnumbertoaninternalcounter.vmstatthensubtractstheinitialvalueofthiscounterfromtheendvalueanddividestheresultbythenumberofsecondsinthemeasurementinterval.Thisvalueistypicallylessthanfivewithastableworkload.Ifthisvalueincreasesrapidly,lookforanapplicationproblem.Iftherearemanythreads(especiallyCPU-intensiveones)competingfortheCPUresource,itis

quitepossibletheywillbescheduledinround-robinfashion.Ifeachoneexecutesforacompleteorpartialtimeslice,thenumberof"run-able"threadscouldeasilyexceed100.

b

Thebcolumnshowstheaveragenumberofkernelthreadsonthewaitqueueatone-secondintervals(awaitingresource,awaitinginput/output).

Kernelthreadsareplacedonthewaitqueuewhenscheduledforexecutionandarewaitingforoneoftheirprocesspagestobepagedin.Onceasecond,thesystemcountsthethreadswaitingandaddsthat

numbertoaninternalcounter.vmstatthensubtractstheinitialvaluefromtheendvalueanddividestheresultbythenumberofsecondsinthemeasurementinterval.Thisvalueisusuallynearzero.Donotconfusethiswithwa--waitingoninput/output(I/O).

NOTE:OnanSMPsystem,therewillbeanadditionalblockedprocessshowninthebcolumn.ThisisforthelrudkprocthatispartoftheVirtualMemoryManager's(VMM)page-replacementalgorithm.

Also,onasystemwithacompressedjournaledfilesystem(JFS)thatismounted,therewillbeanadditionalblockedprocess:thejfsckproc.

memory

Theinformationunderthememoryheadingprovidesinformationaboutrealandvirtualmemory.

avm

Theavmcolumngivestheaveragenumberofpagesallocatedtopagingspace.(InAIX,apagecontains4096bytesofdata.)


10/25

Whenaprocessexecutes,spaceforworkingstorageisallocatedonthepagingdevices(backingstore).Thiscanbeusedtocalculatetheamountofpagingspaceassignedtoexecutingprocesses.Thenumberintheavmfielddividedby256willyieldthenumberofmegabytes(MB),systemwide,allocatedtopagespace.Thelsps-acommandalsoprovidesinformationonindividualpagingspace.Itisrecommendedthatenoughpagingspacebeconfiguredonthesystemsothatthepagingspaceuseddoesnotapproach100percent.Whenfewerthan128unallocatedpagesremainonthepagingdevices,thesystemwillbegintokillprocessestofreesomepagingspace.

VersionsofAIXbefore4.3.2allocatedpagingspaceblocksforpagesofmemoryasthepageswereaccessed.Onalargememorymachine,wheretheapplicationsetissuchthatpagingisneverorrarelyrequired,thesepagingspaceblockswereallocatedbutneverneeded.AIXVersion4.3.2implementsdeferredpagingspaceallocation,inwhichthepagingspaceblocksarenotallocateduntilpagingisnecessary,thus,helpingreducethepagingspacerequirementsofthesystem.Theavmvalueinvmstatindicatesthenumberofvirtualmemory(workingstorage)pagesthathavebeenaccessedbutnotnecessarilypagedout.Withthepreviouspolicyof"latepagespaceallocation",avmhadthesamedefinition.However,sincetheVMMwouldallocatepagingspacediskblocksforeachworkingpagethatwasaccessed,thepagingspaceblockswasequaltotheavm.Thereasonforthepagingspaceblocksto

beallocatedatthetimetheworkingpagesareaccessedissothatifthepageshadtobepagedoutofmemory,therewouldbediskblocksonthepagespacelv'savailableforthein-memorypagestogo.Onsystemsthatneverpage-outtopage-space,it'sawasteofdiskspacetohaveasmanypagespacediskblocksasthereismemory.Withdeferredpolicy,thepagespacediskblocksareonlyallocatedforthepagesthatdoneedtobepagedout.Theavmnumberwillgrowasmoreprocessesgetstartedand/orexistingprocessesusemoreworkingstorage.Likewise,thenumberwillshrinkasprocessesexitand/orfreeworkingstorage.

fre

Thefrecolumnshowstheaveragenumberoffreememoryframes.Aframeisa4096-byteareaofrealmemory.

Thesystemmaintainsabufferofmemoryframes,calledthefreelist,thatwillbereadilyaccessiblewhentheVMMneedsspace.Thenominalsizeofthefreelistvariesdependingontheamountofrealmemoryinstalled.Onsystemswith64MBofmemoryormore,theminimumvalue(MINFREE)is120frames.Forsystemswithlessthan64MB,thevalueistwotimesthenumberofMBofrealmemory,minus8.Forexample,asystemwith32MBwouldhaveaMINFREEvalueof56freeframes.

IfthefrevalueissubstantiallyabovetheMAXFREEvalue(whichisdefinedasMINFREEplus8),thenitisunlikelythatthesystemis

thrashing(continuouslypaginginandout).However,ifthesystemisthrashing,beassuredthatthefrevalueissmall.MostUNIXandAIXoperatingsystemswillusenearlyallavailablememoryfordiskcaching,soyouneednotbealarmedifthefrevalueoscillatesbetweenMINFREEandMAXFREE.

page

Theinformationunderthepageheadingincludesinformationaboutpagefaultsandpagingactivity.


11/25

re

Therecolumnshowsthenumber(rate)ofpagesreclaimed.

ReclaimedpagescansatisfyanaddresstranslationfaultwithoutinitiatinganewI/Orequest(thepageisstillinmemory).Thisincludespagesthathavebeenputonthefreelistbutareaccessedagainbeforetheyarereassigned.ItincludespagespreviouslyrequestedbyVMMforwhichI/Ohasnotyetbeencompletedorthosepre-fetchedbyVMM'sread-aheadmechanismbuthiddenfromthefaultingsegment.

pi

Thepicolumndetailsthenumber(rate)ofpagespagedinfrompagingspace.

Pagingspaceisthepartofvirtualmemorythatresidesondisk.Itisusedasanoverflowwhenmemoryisovercommitted.Pagingconsistsofpaginglogicalvolumesdedicatedtothestorageofworkingsetpagesthathavebeenstolenfromrealmemory.Whenastolenpageisreferencedbytheprocess,apagefaultoccursandthepagemustbereadintomemoryfrompagingspace.Thereisno"good"numberforthis

duetothevarietyofconfigurationsofhardware,software,andapplications.

Onetheoryisthatfivepage-inspersecondshouldbetheupperlimit.Usethistheoreticalmaximumasareferencebutdonotadheretoitrigidly.Thisfieldisimportantasakeyindicatorofpagingspaceactivity.Ifapage-inoccurs,thentheremusthavebeenapreviouspage-outforthatpage.Itisalsolikelyinamemory-constrainedenvironmentthateachpage-inwillforceadifferentpagetobestolenand,therefore,pagedout.

po

Thepocolumnshowsthenumber(rate)ofpagespagedouttopagingspace.

Wheneverapageofworkingstorageisstolen,itiswrittentopagingspace.Ifnotreferencedagain,itremainsonthepagingdeviceuntiltheprocessterminatesordisclaimsthespace.Subsequentreferencestoaddressescontainedwithinthefaulted-outpagesresultinpagefaults,andthepagesarepagedinindividuallybythesystem.Whenaprocessterminatesnormally,anypagingspaceallocatedtothatprocessisfreed.Ifthesystemisreadinginasignificantnumberofpersistentpages,youmayseeanincreaseinpowithoutcorrespondingincreasesinpi.Thissituationdoesnotnecessarilyindicatethrashing,butmaywarrantinvestigationintodataaccesspatternsof

theapplications.

fr

Thefrcolumndetailsthenumber(rate)ofpagesfreed.

AstheVMMpage-replacementcoderoutinescansthePageFrameTable(PFT),itusescriteriatoselectwhichpagesaretobestolentoreplenishthefreelistofavailablememoryframes.ThetotalpagesstolenbytheVMM--bothworking(computational)andfile


12/25

(persistent)pages--arereportedasaratepersecond.JustbecauseapagehasbeenfreeddoesnotmeanthatanyI/Ohastakenplace.Forexample,ifapersistentstorage(file)pagehasnotbeenmodified,itwillnotbewrittenbacktothedisk.IfI/Oisnotnecessary,minimalsystemresourcesarerequiredtofreeapage.

sr

Thesrcolumndetailsthenumber(rate)ofpagesscannedbythepage-placementalgorithm.

TheVMMpage-replacementcodescansthePFTandstealspagesuntilthenumberofframesonthefreelistisatleasttheMAXFREEvalue.Thepage-replacementcodemayhavetoscanmanyentriesinthePageFrameTablebeforeitcanstealenoughtosatisfythefreelistrequirements.Withstable,unfragmentedmemory,thescanrateandfreeratemaybenearlyequal.Onsystemswithmultipleprocessesusingmanydifferentpages,thepagesaremorevolatileanddisjointed.Inthisscenario,thescanratemaygreatlyexceedthefreerate.

cy

ThecycolumnprovidestherateofcompletescansofthePageFrameTable.

cyshowshowmanytimes(persecond)thepage-replacementcodehasscannedthePageFrameTable.SincethefreelistcanbereplenishedwithoutacompletescanofthePFTandbecauseallofthevmstatfieldsarereportedasintegers,thisfieldisusuallyzero.

faults

Theinformationunderthefaultsheadinginthevmstatoutputprovidesinformationaboutprocesscontrol.

in

Theincolumnshowsthenumber(rate)ofdeviceinterrupts.

Thiscolumnshowsthenumberofhardwareordeviceinterrupts(persecond)observedoverthemeasurementinterval.Examplesofinterruptsarediskrequestcompletionsandthe10millisecondclockinterrupt.Sincethelatteroccurs100timespersecond,theinfieldisalwaysgreaterthan100.

sy

Thesycolumndetailsthenumber(rate)ofsystemcalls.

Resourcesareavailabletouserprocessesthroughwell-definedsystem

calls.Thesecallsinstructthekerneltoperformoperationsforthecallingprocessandexchangedatabetweenthekernelandtheprocess.Sinceworkloadsandapplicationsvaryanddifferentcallsperformdifferentfunctions,itisimpossibletosayhowmanysystemcallspersecondaretoomany.

cs

Thecscolumnshowsthenumber(rate)ofcontextswitches.


13/25

ThephysicalCPUresourceissubdividedintologicaltimeslicesof10millisecondseach.Assumingaprocessisscheduledforexecution,itwillrununtilitstimesliceexpires,itispreempted,oritvoluntarilygivesupcontroloftheCPU.WhenanotherprocessisgivencontroloftheCPU,thecontext,orworkingenvironment,ofthepreviousprocessmustbesavedandthecontextofthecurrentprocessmustbeloaded.AIXhasaveryefficientcontextswitchingprocedure,soeachswitchisinexpensiveintermsofresources.Anysignificantincreaseincontextswitchesiscauseforfurtherinvestigation.

cpu

TheinformationunderthecpuheadinginthevmstatoutputprovidesabreakdownofCPUusage.

us

TheuscolumnshowsthepercentofCPUtimespentinusermode.

Processesexecuteinusermodeorsystem(kernel)mode.Wheninusermode,aprocessexecuteswithinitscodeanddoesnotrequirekernelresourcestoperformcomputations,managememory,setvariables,andsoon.

sy

ThesycolumndetailsthepercentofCPUtimespentinsystemmode.

Ifaprocessneedskernelresources,itmustexecuteacallandgointosystemmodetomakethatresourceavailable.I/Otoadrive,forexample,requiresacalltoopenthedevice,seek,andreadandwritedata.ThisfieldshowsthepercentoftimetheCPUwasinsystemmode.OptimumusewouldhavetheCPUworking100percentofthetime.Thisholdstrueinthecaseofasingle-usersystemwithnoneedtosharetheCPU.Generally,ifus+sytimeisbelow90percent,asingle-usersystemisnotconsideredCPUconstrained.However,ifus+sytimeonamulti-usersystemexceeds80percent,theprocessesmayspendtime

waitingintherunqueue.Responsetimeandthroughputmightsuffer.

id

Iftherearenoprocessesavailableforexecution(therunqueueisempty),thesystemdispatchesaprocesscalledwait.Thepsreport(withthe-korgoption)identifiesthisaskprocwithaprocessID(PID)of516.Donotworryifyourpsreportshowsahighaggregatetimeforthisprocess.Itmeansyouhavehadsignificantperiodsoftimewhennootherprocessescouldrun.IftherearenoI/Ospendingtoalocaldisk,alltimechargedtothewaitprocessisclassifiedasidletime.

wa

ThewacolumndetailsCPUidletime(percent)withpendinglocaldiskI/O.

IfthereisatleastoneoutstandingI/Otoalocaldiskwhenthewaitprocessisrunning,thetimeisclassifiedas"waitingonI/O".Awavalueover40percentcouldindicatethatthedisksubsystemmaynotbebalancedproperly,oritmaybetheresultofadisk-intensiveworkload.Ifthereisonlyoneprocessavailableforexecution--


14/25

oftenthecaseonatechnicalworkstation--theremaybenowaytoavoidwaitingonI/O.

NOTE:ThewacolumnonSMPmachinesrunningAIXVersion4.3.2orearlierissomewhatexaggerated.Thisisduetothemethodusedincalculatingwio.

MethodusedinAIX4.3.2andearlierAIXVersions

Ateachclockinterruptoneachprocessor(100timesasecondinAIX),adeterminationismadeastowhichoffourcategories(usr/sys/wio/idle)toplacethelast10millisecondsoftime.IftheCPUwasbusyinusrmodeatthetimeoftheclockinterrupt,thenusrgetstheclocktickaddedintoitscategory.IftheCPUwasbusyinkernelmodeatthetimeoftheclockinterrupt,thenthesyscategorygetsthetick.IftheCPUwasNOTbusy,thenacheckismadetoseeifanyI/Otodiskisinprogress.IfanydiskI/Oisinprogress,thenthewiocategoryisincremented.IfNOdiskI/OisinprogressandtheCPUisnotbusy,thentheidlcategorygetsthetick.

------------------------------------------------------------------------

Summarystatistics

vmstatwiththe-soptionreportsabsolutecountsofvariouseventssincethesystemwasbooted.Thereare23separateeventsreportedinthevmstat-soutput;thefollowing4haveprovenmosthelpful.The19remainingfieldscontainavarietyofactivitiesfromaddresstranslationfaultstolockmissestosystemcalls.Theinformationinthose19fieldsisalsovaluablebutislessfrequentlyused.

pageins

Thepageinsfieldshowsthenumbersystemwidepage-ins.

Whenapageisreadfromdisktomemory,thiscountisincremented.ItisacountofVMM-initiatedreadoperationsand,withthepageoutsfield,

representstherealI/O(diskreadsandwrites)initiatedbytheVMM.

pageouts

Thepageoutsfieldshowsthenumberofsystemwidepage-outs.

Theprocessofwritingpagestothediskiscountincremented.ThepageoutsfieldvalueisatotalcountofVMM-initiatedwriteoperationsand,withthepageinsfield,representsthetotalamountofrealI/OinitiatedbytheVMM.

pagingspacepageins

ThepagingspacepageinsfieldisthecountofONLYpagesreadfrompagingspace.

pagingspacepageouts

ThepagingspacepageoutsfieldisthecountofONLYpageswrittentopagingspace.

Usingthesummarystatistics


15/25

Thefourprecedingfieldscanbeusedtoindicatehowmuchofthesystem'sI/Oisforpersistentstorage.Ifthevalueforpagingspacepageinsissubtractedfromthe(systemwide)valueforpageins,theresultisthenumberofpagesthatwerereadfrompersistentstorage(files).Likewise,ifthevalueforpagingspacepageoutsissubtractedfromthe(systemwide)valueforpageouts,theresultisthenumberofpersistentpages(files)thatwerewrittentodisk.

Rememberthatthesecountsapplytothetimesincesysteminitialization.Ifyouneedcountsforagiventimeinterval,executevmstat-satthetimeyouwanttostartmonitoringandagainattheendoftheinterval.Thedeltasbetweenlikefieldsofsuccessivereportswillbethecountfortheinterval.Itiseasiertoredirecttheoutputofthereportstoafileandthenperformthemath.

[TechDocsRef:90605226914708PublishDate:Nov.04,19994FAXRef:6220]

PerformanceTuning--TheiostatTool

Contents

AboutthisdocumentSyntax

TTYstatisticsintheiostatoutputCPUstatisticsintheiostatoutputDiskstatisticsintheiostatoutputAnalyzingthedataConclusions

Aboutthisdocument

Thisdocumentisbasedon"PerformanceTuning:AContinuingSeries--TheiostatTool",byBarrySaad,fromtheJanuary/February1994issueofAIXTRA:IBM'SMAGAZINEFORAIXPROFESSIONALS.

ThisarticlediscussesiostatandhowitcanidentifyI/O-subsystemandCPU

bottlenecks.iostatworksbysamplingthekernel'saddressspaceandextractingdatafromvariouscountersthatareupdatedeveryclocktick(1clocktick=10milliseconds[ms]).Theresults--coveringTTY,CPU,andI/Osubsystemactivity--arereportedasper-secondratesorasabsolutevaluesforthespecifiedinterval.ThisdocumentappliestoAIXVersions4.x.

------------------------------------------------------------------------

Syntax

Normally,iostatisissuedwithbothanintervalandacountspecified,withthereportsenttostandardoutputorredirectedtoafile.The

commandsyntaxappearsbelow:

iostat[-t][-d][Drives][Interval[Count]]

The-dflagcausesiostattoprovideonlydiskstatisticsforalldrives.The-tflagcausesiostattoprovideonlysystem-wideTTYandCPUstatistics.

NOTE:The-tand-doptionsaremutuallyexclusive.


16/25

Ifyouspecifyoneormoredrives,theoutputislimitedtothosedrives.Multipledrivescanbespecified;separatetheminthesyntaxwithspaces.

Youcanspecifyatimeinsecondsfortheintervalbetweenrecordstobeincludedinthereports.Theinitialrecordcontainsstatisticsforthetimesincesystemboot.Succeedingrecordscontaindatafortheprecedinginterval.Ifnointervalisspecified,asinglerecordisgenerated.

Ifyouspecifyaninterval,thecountofthenumberofrecordstobeincludedinthereportcanalsobespecified.Ifyouspecifyanintervalwithoutacount,iostatwillcontinuerunninguntilitiskilled.

--------------------------------------------------------------------|||tty:tintoutavg-cpu:%user%sys%idle%iowait||2.23.30.41.397.70.6||||Disks:%tm_actKbpstpsmspsKb_readKb_wrtn||hdisk00.41.10.31176751087266||hdisk10.31.00.2592301017734||hdisk20.00.20.018018946832||cd00.00.00.000||||tty:tintoutavg-cpu:%user%sys%idle%iowait|

|2.23.30.41.397.70.6||||Disks:%tm_actKbpstpsmspsKb_readKb_wrtn||hdisk00.41.10.31176751087266||hdisk10.31.00.2592301017734||hdisk20.00.20.018018946832||cd00.00.00.000|||--------------------------------------------------------------------Figure1.SampleOutputfromiostat22

Thefollowingsectionsexplaintheoutput.------------------------------------------------------------------------

TTYstatisticsintheiostatoutput

ThetwocolumnsofTTYinformation(tinandtout)intheiostatoutputshowthenumberofcharactersreadandwrittenbyallTTYdevices,includingbothrealandpseudoTTYdevices.RealTTYdevicesarethoseconnectedtoanasynchronousport.Some"pseudoTTYdevices"areshells,telnetsessions,andaixterms.

*tinshowsthetotalcharacterspersecondreadbyallTTYdevices.*toutindicatesthetotalcharacterspersecondwrittentoallTTYdevices.

Generallytherearefewerinputcharactersthanoutputcharacters.Forexample,assumeyourunthefollowing:

iostat-t130cd/usr/sbinls-l

Youwillseefewinputcharactersandmanyoutputcharacters.Ontheotherhand,applicationssuchasviresultinasmallerdifferencebetweenthenumberofinputandoutputcharacters.Analystsusingmodemsfor


17/25

asynchronousfiletransfermaynoticethenumberofinputcharactersexceedingthenumberofoutputcharacters.Naturally,thisdependsonwhetherthefilesarebeingsentorreceivedrelativetothemeasuredsystem.

SincetheprocessingofinputandoutputcharactersconsumesCPUresource,lookforacorrelationbetweenincreasedTTYactivityandCPUutilization.Ifsucharelationshipexists,evaluatewaystoimprovetheperformanceoftheTTYsubsystem.Stepsthatcouldbetakenincludechangingtheapplicationprogram,modifyingTTYportparametersduringfiletransfer,orperhapsupgradingtoafasterormoreefficientasynchronouscommunicationsadapter.

------------------------------------------------------------------------

CPUstatisticsintheiostatoutput

TheCPUstatisticscolumns(%user,%sys,%idle,and%iowait)provideabreakdownofCPUusage.Thisinformationisalsoreportedintheoutputofthevmstatcommandinthecolumnslabeledus,sy,id,andwa.

%user

The%usercolumnshowsthepercentageofCPUresourcespentinusermode.

AUNIXprocesscanexecuteinuserorsystemmode.Wheninusermode,aprocessexecuteswithinitsowncodeanddoesnotrequirekernelresources.

%sys

The%syscolumnshowsthepercentageofCPUresourcespentinsystemmode.ThisincludesCPUresourceconsumedbykernelprocesses(kprocs)andothersthatneedaccesstokernelresources.Forexample,thereadingorwritingofafilerequireskernelresourcestoopenthefile,seekaspecificlocation,andreadorwritedata.AUNIXprocessaccesseskernelresourcesbyissuingsystemcalls.

Typically,theCPUispacing(thesystemisCPUbound)ifthesumofuser

andsystemtimeexceeds90percentofCPUresourceonasingle-usersystemor80percentonamulti-usersystem.ThisconditionmeansthattheCPUisthelimitingfactorinsystemperformance.

Theratioofusertosystemmodeisdeterminedbyworkloadandismoreimportantwhentuninganapplicationthanwhenevaluatingperformance.

AkeyfactorwhenevaluatingCPUperformanceisthesizeoftherunqueue(providedbythevmstatcommand).Ingeneral,astherunqueueincreases,userswillnoticedegradation(anincrease)inresponsetime.

%idle

The%idlecolumnshowsthepercentageofCPUtimespentidle,orwaiting,withoutpendinglocaldiskI/O.Iftherearenoprocessesontherunqueue,thesystemdispatchesaspecialkernelprocesscalledwait.OnmostAIXsystems,thewaitprocessID(PID)is516.

%iowait

The%iowaitcolumnshowsthepercentageoftimetheCPUwasidlewithpendinglocaldiskI/O.


18/25

TheiowaitstateisdifferentfromtheidlestateinthatatleastoneprocessiswaitingforlocaldiskI/Orequeststocomplete.UnlesstheprocessisusingasynchronousI/O,anI/Orequesttodiskcausesthecallingprocesstoblock(orsleep)untiltherequestiscompleted.Onceaprocess'sI/Orequestcompletes,itisplacedontherunqueue.

Ingeneral,ahighiowaitpercentageindicatesthesystemhasamemoryshortageoraninefficientI/Osubsystemconfiguration.UnderstandingtheI/ObottleneckandimprovingtheefficiencyoftheI/Osubsystemrequiremoredatathaniostatcanprovide.However,typicalsolutionsmightinclude:

*limitingthenumberofactivelogicalvolumesandfilesystemsplacedonaparticularphysicaldisk(TheideaistobalancefileI/Oevenlyacrossallphysicaldiskdrives.)*spreadingalogicalvolumeacrossmultiplephysicaldisks(Thisisusefulwhenanumberofdifferentfilesarebeingaccessed.)*creatingmultipleJFSlogsforavolumegroupandassigningthemtospecificfilesystems(Thisisbeneficialforapplicationsthatcreate,delete,ormodifyalargenumberoffiles,particularlytemporaryfiles.)*backingupandrestoringfilesystemstoreducefragmentation(Fragmentationcausesthedrivetoseekexcessivelyandcanbealargeportionofoverallresponsetime.)

*addingadditionaldrivesandrebalancingtheexistingI/Osubsystem

Onsystemsrunningaprimaryapplication,highI/Owaitpercentagemayberelatedtoworkload.Inthiscase,theremaybenowaytoovercometheproblem.Onsystemswithmanyprocesses,somewillberunningwhileotherswaitforI/O.Inthiscase,theiowaitcanbesmallorzerobecauserunningprocesses"hide"waittime.Althoughiowaitislow,abottleneckmaystilllimitapplicationperformance.TounderstandtheI/Osubsystemthoroughly,examinethestatisticsinthenextsection.

NOTE:The%iowaitcolumnonSMPmachinesrunningAIXVersions4.3.2orearlierissomewhatexaggerated.Thisisduetothemethodusedincalculatingwio.

MethodusedinAIX4.3.2andearlierAIXVersions

Ateachclockinterruptoneachprocessor(100timesasecondinAIX),adeterminationismadeastowhichoffourcategories(usr/sys/wio/idle)toplacethelast10msoftime.IftheCPUwasbusyinusrmodeatthetimeoftheclockinterrupt,thenusrgetstheclocktickaddedintoitscategory.IftheCPUwasbusyinkernelmodeatthetimeoftheclockinterrupt,thenthesyscategorygetsthetick.IftheCPUwasNOTbusy,thenacheckismadetoseeifanyI/Otodiskisinprogress.IfanydiskI/Oisinprogress,thenthewiocategoryisincremented.IfNOdiskI/OisinprogressandtheCPUisnotbusy,thentheidlcategorygetsthetick.------------------------------------------------------------------------

Diskstatisticsintheiostatoutput

ThediskstatisticsportionoftheiostatoutputprovidesabreakdownofI/Ousage.Thisinformationisusefulindeterminingwhetheraphysicaldiskislimitingperformance.

DiskI/Ohistory

Thesystemmaintainsahistoryofdiskactivitybydefault.Notethat


19/25

historyisdisabledifyouseethemessage:

Diskhistorysincebootnotavailable.

Thismessagedisplaysonlyinthefirstoutputrecordfromiostat.

DiskI/OhistoryshouldbeenabledsincetheCPUresourceusedinmaintainingitisinsignificant.History-keepingcanbedisabledorenabledinSMITunderthefollowingpath:

->SystemEnvironments->Change/ShowCharacteristicsofOperatingSystem->ContinuouslymaintainDISKI/Ohistory->true|false

Choosetruetoenablehistory-keepingorfalsetodisableit.

Disks

TheDisks:columnshowsthenamesofthephysicalvolumes.Theyareeitherhdiskorcdfollowedbyanumber.(hdisk0andcd0refertothefirstphysicaldiskdriveandthefirstCDdiskdrive,respectively.)

%tm_act

The%tm_actcolumnshowsthepercentageoftimethevolumewasactive.Thisistheprimaryindicatorofabottleneck.

Adriveisactiveduringdatatransferandcommandprocessing,suchasseekingtoanewlocation.Thedisk-usepercentageisdirectlyproportionaltoresourcecontentionandinverselyproportionaltoperformance.Asdiskuseincreases,performancedecreasesandresponsetimeincreases.Ingeneral,whenadisk'suseexceeds70percent,processesarewaitinglongerthannecessaryforI/OtocompletebecausemostUNIXprocessesblock(orsleep)whilewaitingfortheirI/Orequeststocomplete.

Kbps

KbpsshowstheamountofdatareadfromandwrittentothedriveinKBspersecond.ThisisthesumofKb_readplusKb_wrtn,dividedbythenumberofsecondsinthereportinginterval.

tps

tpsreportsthenumberoftransferspersecond.AtransferisanI/Orequestatthedevicedriverlevel.

Kb_read

Kb_readreportsthetotaldata(inKBs)readfromthephysicalvolume

duringthemeasuredinterval.

Kb_wrtn

Kb_wrtnshowstheamountofdata(inKBs)writtentothephysicalvolumeduringthemeasuredinterval.

------------------------------------------------------------------------

Analyzingthedata


20/25

Takenalone,thereisnounacceptablevalueforanyoftheprecedingfieldsbecausestatisticsaretoocloselyrelatedtoapplicationcharacteristics,systemconfiguration,andtypesofphysicaldiskdrivesandadapters.Therefore,whenevaluatingdata,youmustlookforpatternsandrelationships.Themostcommonrelationshipisbetweendiskutilizationanddatatransferrate.

Todrawanyvalidconclusionsfromthisdata,youmustunderstandtheapplication'sdiskdataaccesspatterns--sequential,random,oracombination--andthetypeofphysicaldiskdrivesandadaptersonthesystem.

Forexample,ifanapplicationreadsandwritessequentially,youshouldexpectahighdisk-transferratewhenyouhaveahighdisk-busyrate.(NOTE:Kb_readandKb_wrtncanconfirmanunderstandingofanapplication'sreadandwritebehaviorbuttheyprovidenoinformationonthedataaccesspatterns.)

Generallyyoudonotneedtobeconcernedaboutahighdisk-busyrateaslongasthedisk-transferrateisalsohigh.However,ifyougetahighdisk-busyrateandalowdata-transferrate,youmayhaveafragmentedlogicalvolume,filesystem,orindividualfile.

Whatisahighdata-transferrate?Thatdependsonthediskdriveandtheeffectivedata-transferrateforthatdrive.Youshouldexpectnumbersbetweentheeffectivesequentialandeffectiverandomdisk-transferrates.BelowisachartofeffectivetransferratesforseveralcommonSCSI-1andSCSI-2diskdrives.

Table1.EffectiveTransferRate(KB/sec),Part1of2TYPEOFACCESS400MBDRIVE670MBDRIVE857MBDRIVERead-Sequential158915252142Read-Random241172262Write-Sequential118511081588Write-Random327275367Table2.EffectiveTransferRate(KB/sec),Part2of2

TYPEOFACCESS1.2GBDRIVE1.37GBDRIVE1.2GBDRIVE1.37GBS-2DRIVERead-Sequential2169266721803123Read-Random292299385288Write-Sequential1464218921562357Write-Random362491405549

Thetransferratesweredeterminedduringperformancetestingandgivemoreaccurateexpectationsofdiskperformancethanthemedia-transferrate,whichreflectsthehardwarecapabilityanddoesnotaccountforoperatingsystemandapplicationoverhead.

Anotheruseofthedataistoanswerthequestion:"DoIneedanotherSCSIadapter?"Ifyou'veeverbeenaskedthisquestion,youprobablyprovideda

genericanswerorjustplainguessed.

Youcanusedatacapturedbyiostattoanswerthequestionaccuratelybytrackingtransferrates,findingthemaximumdata-transferrateforeachdisk.Assumethatthemaximumrateoccurssimultaneouslyforalldrives(theworstcase).Formaximumaggregateperformance,themeasuredtransferratesfordrivesattachedtoagivenadaptermustbebelowtheeffectiveSCSIadapterthroughputrating.

Forplanningpurposes,youshoulduse70percentoftheadapter'srated


21/25

throughput(forexample,2.8MBpersecondforaSCSI-1adapter).Thispercentageshouldprovideasufficientbufferforoccasionalpeakratesthatmayoccur.Whenaddingadrive,youmustassumethedata-transferrate.Atleastyouwillhavethecollecteddataandtheeffectivetransferratestouseasabasis.

KeepinmindthattheSCSIadaptermaybesaturatedifthedata-transferratesovermultipleintervalsapproachtheeffectiveSCSIadapterthroughputrating.Inthatcase,theprecedinganalysisisinvalid.

------------------------------------------------------------------------

Conclusions

TheprimarypurposeoftheiostattoolistodetectI/Obottlenecksbymonitoringthediskutilization(%tm_actfield).iostatcanalsobeusedtoidentifyCPUproblems,assistincapacityplanning,andprovideinsightintosolvingI/Oproblems.Armedwithbothvmstatandiostat,youcancapturethedatarequiredtoidentifyperformanceproblemsrelatedtoCPU,memory,andI/Osubsystems.

[TechDocsRef:90605205314704PublishDate:Oct.15,19994FAXRef:9779]

TheAIXVirtualMemoryManager(VMM)

------------------------------------------------------------------------

Contents

AboutthisdocumentVMMoverviewReal-MemorymanagementFreelistPersistentvs.memorysegmentsPagingspaceandvirtualmemoryVMMmemoryloadcontrolfacility

VMSTAT'savmfieldVMSTAT'sfrefieldHowthesystemisusingmemoryExplanationofsvmonoutput------------------------------------------------------------------------

Aboutthisdocument

ThisdocumentaddresseshowRAMandpagingspaceareused.ThisinformationappliestoAIXVersion4.x.------------------------------------------------------------------------

VMMoverview

TheVirtualMemoryManager(VMM)servicesmemoryrequestsfromthesystemanditsapplications.Virtual-memorysegmentsarepartitionedinunitscalledpages;eachpageiseitherlocatedinphysicalmemory(RAM)orstoredondiskuntilitisneeded.AIXusesvirtualmemoryinordertoaddressmorememorythanisphysicallyavailableinthesystem.ThemanagementofmemorypagesinRAMorondiskishandledbytheVMM.------------------------------------------------------------------------

Real-Memorymanagement


22/25

InAIX,virtual-memorysegmentsarepartitionedinto4096-byteunitscalledpages.Realmemoryisdividedinto4096-bytepageframes.TheVMMhastwomajorfunctions:1)managetheallocationofpageframes,and2)resolvereferencestovirtual-memorypagesthatarenotcurrentlyinRAM(storedinpagingspace)ordonotyetexist.

Inordertoaccomplishitstask,theVMMmaintainsafreelistofavailablepageframes.TheVMMalsousesapage-replacementalgorithmtodeterminewhichvirtual-memorypagescurrentlyinRAMwillhavetheirpageframesreassignedtothefreelist.Thepage-replacementalgorithmtakesintoaccounttheexistenceofpersistentvs.workingsegments,repaging,andVMMthresholds.------------------------------------------------------------------------

Freelist

TheVMMmaintainsalistoffreepageframesthatitusestosatisfypagefaults.Thefreelistismadeupofunallocatedpageframes.AIXtriestouseallofRAMallofthetime,exceptforasmallamountwhichitmaintainsonthefreelist.TomaintainthissmallamountofunallocatedpagestheVMMwillusePageOutsandPageStealstofreeupspaceandreassignthosepageframestothefreelist.Thevirtual-memorypageswhosepageframesaretobereassignedareselectedviatheVMM's

page-replacementalgorithm.------------------------------------------------------------------------

Persistentvs.workingmemorysegments

AIXdistinguishesbetweendifferenttypesofmemorysegments,andtounderstandtheVirtualMemoryManager,itisimportanttounderstandthedifferencebetweenworkingandpersistentsegments.Apersistentsegmenthasapermanentstoragelocationondisk.Filescontainingdataorexecutableprogramsaremappedtopersistentsegments.WhenaJournaledFileSystem(JFS)fileisopenedandaccessedthefiledataiscopiedintoRAM.VMMparameterscontrolwhenphysicalmemoryframesallocatedtopersistentpagesshouldbeoverwrittenandusedtostoreotherdata.

Workingsegmentsaretransitoryandexistonlyduringtheirusebyaprocess,andhavenopermanentdiskstoragelocation.Processstackanddataregionsaremappedtoworkingsegmentsandsharedlibrarytextsegments.Pagesofworkingsegmentsmustalsohavediskstoragelocationstooccupywhentheycannotbekeptinrealmemory.Thediskpagingspaceisusedforthispurpose.Whenaprogramexits,allofitsworkingpagesareplacedbackonthefreelistimmediately.------------------------------------------------------------------------

Pagingspaceandvirtualmemory

WorkingpagesinRAMthatcanbemodifiedandpagedoutareassigneda

correspondingslotinpagingspace.Theallocatedpagingspacewillonlybeusedifthepageneedstobepagedout.However,anallocatedpageinpagingspacecannotbeusedbyanotherpage.Itremainsreservedforaparticularpageforaslongasitexistsinvirtual-memory.Sincepersistentpagesarepagedouttotheirlocationondiskfromwhichtheycame,pagingspacedoesnotneedtobeallocatedforpersistentpagesresidinginRAM.

TheVMMhasthreemodesforallocatingpagingspace:earlypagingspaceallocation,latepagingspaceallocationanddeferredpagingspace


23/25

allocation.Earlyallocationpolicyreservespagingspacewheneveramemoryrequestforaworkingpageismade.Lateallocationpolicyonlyassignspagingspacewhentheworkingpageisreferenced.AIXVersions4.3.2andlaterimplementadeferredpagingspaceallocationpolicy.Withdeferredpagingspacepolicy,pagingspaceblocksarenotallocateduntiltheworkingpagesareactuallypagedoutofmemory.Thissignificantlyreducesthepagingspacerequirementsofthesystem.------------------------------------------------------------------------

VMMmemoryloadcontrolfacility

Whenaprocessreferencesavirtual-memorypagethatisondisk,becauseiteitherhasbeenpagedoutorhasneverbeenread,thereferencedpagemustbepagedin,andthismaycauseoneormorepagestobepagedoutifthenumberofavailablefreepageframesislow.TheVMMattemptstostealpageframesthathavenotbeenrecentlyreferenced,andthusunlikelytobereferencedinthenearfuture,viathepage-replacementalgorithm.

Asuccessfulpage-replacementkeepsthememorypagesofallcurrentlyactiveprocessesinRAM,whilethememorypagesofinactiveprocessesarepagedout.However,whenRAMisover-committed,itbecomesdifficulttochoosepagesforpageoutbecausetheywillbereferencedinthenearfuturebycurrentlyrunningprocesses.Theresultisthatpagesthatwillsoonbereferencedstillgetpagedoutandthenpagedinagainlater.When

thishappens,continuouspaginginandpagingoutmayoccurifRAMisover-committed.Thisconditioniscalledthrashing.Thesystemspendsmostofitstimepaginginandpagingoutinsteadofexecutingusefulinstructions,andnoneoftheactiveprocessesmakeanysignificantprogress.TheVMMhasamemoryloadcontrolalgorithmthatdetectswhenthesystemisthrashingandthenattemptstocorrectthecondition.------------------------------------------------------------------------

VMSTAT'Savmfield

avmstandsfor"ActiveVirtualMemory"andnot"AvailableMemory".TheavmvalueinVMSTATindicatesthenumberofvirtual-memorypagesthathavebeenaccessedbutnotnecessarilypagedout.Withthepreviouspolicyoflate

pagespaceallocation,avmhadthesamedefinitionbutsincetheVMMwouldallocatepagingspacediskblocksforeachworkingpagethatwasaccessed,thenthepagingspaceblockswereequaltotheavm.Withdeferredpolicy,thepagespacediskblocksareonlyallocatedforthepagesthatneedtobepagedout.Theavmnumberwillgrowasmoreprocessesgetstartedand/orexistingprocessesusemoreworkingstorage.Likewise,thenumberwillshrinkasprocessesexitand/orfreeworkingstorage.------------------------------------------------------------------------

VMSTAT'Sfrefield

freisthenumberof4Kpagesthatarecurrentlyonthefreelist.Whenanapplicationterminates,allofitsworkingpagesareimmediatelyreturned

tothefreelist.Itspersistentpages,however,remaininRAMandarenotaddedbacktothefreelistuntiltheyarestolenbytheVMMforotherprograms.Persistentpagesarealsofreedifthecorrespondingfileisdeleted.

Forthisreason,thefrevaluemaynotindicatealltherealmemorythatcanbereadilyavailableforusebyprocesses.Ifapageframeisneeded,thenpersistentpagesrelatedtoterminatedapplicationsareamongthefirsttobehandedovertoanotherprogram.


24/25

TheminimumnumberofpagesthattheVirtualMemoryManagerkeepsonthefreelistisdeterminedbytheminfreeparameterofvmtune.Ifthenumberofpagesonthefreelistdropsbelowminfree,theVirtualMemoryManagerwillstealpagesuntilthefreelisthasbeenrestoredtothemaxfreevalue.------------------------------------------------------------------------

Howthesystemisusingmemory

Thesvmoncommandcanbeusedtodetermineroughlyhowmuchmemorythesystemisusing.

NOTE:PAIDE/6000mustbeinstalledinordertousesvmon.Checktoseeifthisisinstalled,byexecutingthefollowingcommand:

$lslpp-lperfagent.tools.

IfyouareatAIXVersion4.3.0orhigher,thenthisfilecanbefoundontheAIXBaseOperatingSystemmedia.Otherwise,toorderPAIDE/6000,callIBMDIRECT1-800-426-2255orcontactyourlocalIBMrepresentative.

Asroot,typesvmon.Underthepgspaceheading,theinusefieldisthenumberofworkingpagesthatareinuseinallofvirtualmemory.------------------------------------------------------------------------

Explanationofsvmonoutput

memory:

SIZEtotalsizeofmemoryin4Kpages

INUSEnumberofpagesinRAMthatareinusebyaprocessplusthenumberofpersistentpagesthatbelongedtoaterminatedprocessandarestillresidentinRAM.Thisvalueisthetotalsizeofmemoryminusthenumberofpagesonthefreelist.

FREEnumberofpagesonfreelist.

PINnumberofpagespinnedinRAM(apinnedpageisapagethatisalwaysresidentinRAMandcannotbepagedout)

inuse:

WORKnumberofworkingpagesinRAM

PERSnumberofpersistentpagesinRAM

CLNTnumberofclientpagesinRAM(clientpageisaremotefilepage)

pin:

WORKnumberofworkingpagespinnedinRAM

PERSnumberofpersistentpagespinnedinRAM

CLNTnumberofclientpagespinnedinRAM

pgspace:

SIZEtotalsizeofpagingspacein4Kpages


25/25

INUSEtotalnumberofallocatedslots.(Seeexplanationaboveonallocationofpagingspace).

Tofindouthowmuchmemoryaprocessisusing,type

$svmon-PPID(foroneprocess)or$svmon-Pau|more(forallprocesses)

Toseethenumberofworkingpagesuniquetothisprocess'privatestackanddatauseinallofvirtualmemory,lookattheworktypeanddescriptionprivate.Thesvmonoutputmayalsolistseveralsharedsegments.Foracompletepicture,determinewhichsegmentsareuniquetoanindividualprocessandwhicharesharedwithotherprograms.Multiplythevaluesby4096togetthenumberofbytesinmemorytheprocessisusing.Thenumber4096comesfromthefactthateachpageis4KBinsize.Youcanalsodividethenumberofpagesby256inordertogetmegabytes.

tips for performance bottlenecks

Documents