011000358700006399522006 e
TRANSCRIPT
8/13/2019 011000358700006399522006 e
http://slidepdf.com/reader/full/011000358700006399522006-e 1/5Subscribe today. Visit www.SAPinsider.com.
Implementation teams often conductcustom load tests in large and mission-critical SAP projects to ensure that thenewly installed system landscape willfulfill performance needs. A customerplanning to install a new SAP system for10,000 users, for example, could simulatethe user load and determine if the systemcan meet its capacity requirementsunder various operating conditions.
Any custom load test presentsimplementation teams with severalchallenges. Simulating the right user
load is a precise science, since teamsneed to test exactly the functionalitythat those 10,000 users will use, gatherbusiness requirements to derive anaccurate system load profile, and prop-erly simulate user behaviors with aload test tool.
But the most critical part of a loadtest is how to analyze system behaviorwhen certain functional and performanceissues occur during the actual load test,
and to understand why they occurred.This is especially challenging for SAP
project teams experienced with ABAPapplications, but not as familiar withJava-based applications.
Project teams need an approach thatcan reduce the complexity of load tests.And once load tests are complete, teamsneed to understand their results andexplain how Java-specific key perform-ance indicators (KPIs) can help evaluateand optimize performance (Hint: lookingonly at response time will leave you
shortchanged) . Both the load testapproach and the Java KPIs introducedin this article have been used extensivelyin SAP performance-tuning projects,in benchmark tests for several SAPNetWeaver releases, and on a varietyof operating system platforms.
How to Simplify Load Testsfor Complex Java ApplicationsA complex Java-based application usuallyconsists of many interacting system com-ponents.To identify performance issues,
you need to drill down and pinpoint theparticular component(s) causing the
Demystifying Java-Based
Load Tests and Their ResultsLessons Learned fromSAP NetWeaver Portal Benchmarks
bottleneck.This is an extensive process,
especially for Java applications.Toidentify the bottleneck components,you need an accurate architecturaloverview of the Java application, andspecialized monitoring tools to analyzethe complex interaction patternsbetween system components.
An alternative to testing (andstressing!) the entire application at onceis to perform load tests on individualsystem components and then combinethese results to obtain a performanceoverview of the complete application.Thisapproach can significantly reduce thecomplexity of load tests and improve theefficiency of analysis efforts. This best-practice method involves three steps:
■ Derive component-specific load profilesfrom the overall load profile
■ Test one component at a time, whilesimulating the other components
Regular Feature
Performance &Data ManagemeCorner
Xiaoqing Cheng, SAP AG
Why Conduct a Load Test?A successful custom load test can ensure that the performance requirementsof a complex software application, including the hardware landscape and thecommunication network infrastructure, have been met. Load tests are a bestpractice for large, highly customized, and mission-critical SAP systemimplementations.
This article appeared in the Oct ■ Nov ■ Dec 2006 issue of SAP Insider and appears here withpermission from the publisher, Wellesley Information Services (WIS), www.WISpubs.com .
8/13/2019 011000358700006399522006 e
http://slidepdf.com/reader/full/011000358700006399522006-e 2/5
8/13/2019 011000358700006399522006 e
http://slidepdf.com/reader/full/011000358700006399522006-e 3/5
Step 2: Simulate the InteractingComponentsOne of the most important design goalsof SAP NetWeaver Portal benchmarksis to focus the load only on the portalcomponent.The backend system shouldnot have any influence on performance.This design goal has been achieved in theEP-ESS benchmark by simply omittingthe HTTP requests sent to the backendsystem from the benchmark script, asshown in Figure 1.This simulation con-cept applies generally to URL iViews thatretrieve iView content directly from thebackend system.
To simulate the CRM back end inEP-PCC scenarios (see Figure 2 ), weincluded a switch parameter in CRMJava iViews used in the EP-PCC bench-mark.Through this switch parameter, asimulation function module in CRM iscalled instead of the original one in per-formance tests. A simulation functionmodule returns precompiled businesscontent without database access and anyexpensive processing — with a responsetime of less than 30 ms. A package ofsimulation modules is available as ofmySAP CRM 4.0 SP8. Detailed configu-ration instructions are described athttp://service.sap.com/benchmarkDocumentation .
Step 3: Combine the ResultsAfter successfully completing the loadtests for each component, you cancombine their individual performanceresults (depending on their specificinteraction patterns) to determine theoverall KPIs. Individual resourceconsumptions, such as CPU andmemory, likewise must be summarizedto obtain the total system resourceconsumption.
When we consider the averageresponse time of user interaction steps,the response times of the individualcomponents generally have to be addedto the total response time — if theinteractions between the componentsoccurred sequentially, that is. In cases
of parallel component interactions, themaximum component response timerepresents the total response time.
In the case of the EP-ESS scenarios,only one iView is displayed in each portalpage.Therefore, the total system response
time is the sum of the response times ofthe portal and the ESS component. InEP-PCC scenarios, the portal pagesdisplay between two and four CRM JavaiViews, which are processed in parallel.
This parallel processing is already takeninto consideration in the average responsetime of the portal component when theCRM back end is simulated.
Each CRM Java iView calls severalCRM function modules. The maximum
sequential response time of the functionmodules called by one Java iView mustbe added to the response time of theportal component to build the totalaverage response time.
Figure 1 A high-level look at the two different types of interaction patternsin EP-ESS scenarios
Figure 2 The two different interaction patterns of EP-PCC scenarios
Subscribe today. Visit www.SAPinsider.com.
8/13/2019 011000358700006399522006 e
http://slidepdf.com/reader/full/011000358700006399522006-e 4/5
Separating the testing process intothe individual system components greatlyreduces load-test complexity. But nowhow do implementation teams interpretthe test results to better understandtheir application’s performance?
Understand the RelevantPerformance KPIs for JavaAn essential part of performanceanalysis and optimization is using a setof well-defined KPIs. Good KPIs shouldcorrectly reflect performance require-ments.They must also be accuratelymeasurable, and in an ideal case, shouldindicate possible optimizations.
The response time of a user interaction
step is the most important KPI in termsof the end-user performance experience.Response time in this context is definedas the elapsed clock time between theuser input action and the display of thenext screen. But comprehensive KPIanalysis is about more than just responsetime.The unique characteristics ofJava make a more comprehensive KPIanalysis necessary. Next we’ll discusssome additional KPIs that are particu-
larly important for Java applications.
CPU Time
Related to response time is CPU time —a critical system resource shared by manyconcurrent users on a server. The CPUtime is measured per user interactionstep. For Java applications, this KPI hasthe same advantages as it does for non-Java applications, such as high accuracyand reproducibility. In load tests, it is
also possible to measure the server’s CPUutilization as an operating system counter.However, it is not suitable for accurateperformance measurements and compar-isons because CPU utilization depends onthe current system throughput.
Memory Consumption
Memory consumption is another importantKPI. It is quite a challenge to measureand analyze the memory consumption of
Java applications. Both Java’s flexibleheap configuration and its automaticgarbage collection (GC) have significantinfluence on memory behavior and resultin varied performance. 2 For a completeobservation of Java memory consumptionbehavior, teams should consider threememory KPIs:
■ The framework space representsthe memory requirement on a JavaVirtual Machine (VM) after the SAPNetWeaver Application Server Javahas been started and warmed up. ThisKPI depends on the deployment ofapplications on the Java VM. Usingthis KPI, you can calculate how muchmemory in the Java heap is still avail-able to run applications on this JavaVM. For measurement, log on to theSAP NetWeaver Application ServerJava via the telnet interface, jumpto the Java VM, and trigger full GCseveral times. 3 Then you can view the“bytes after GC” counter in the GC log.
■ The user session space is the amountof memory occupied by a logged on,but inactive, user. After start-up andwarm-up, measure the initial heap
usage (M1) after several triggered fullGCs as described above.You log N(for example, 100) users on to thesystem and let them execute sometypical transactions, either manuallyor by using a load test tool. Keepingthe users logged on but inactive, youagain measure the heap usage (M2)applying the same procedure asbefore.The user session space equals(M2-M1)/N.
■ The processing space is defined asthe average garbage-collected bytesper user interaction step. This KPIrepresents the dynamic memoryconsumption of an active user whenprocessing specific user interactionsteps. You can measure this KPI bytotaling the garbage-collected bytes(B) during the execution of a largenumber (K) of user interactionssteps, and then calculating the KPIas B/K. Using a load test tool to puta moderate load on a system for acertain time period (long enough toensure that B is greater than themaximum heap size), the measure-ment results of this KPI are veryaccurately reproducible. According to
current results, this dynamic part isdominant compared to the static usersession space.
The memory consumption of a givenJava application depends not only onthe number of concurrent users, but alsoon how active the users are.This is whywe use three memory KPIs, instead ofthe traditional one, to describe the com-plete Java memory behavior.
Comprehensive performance KPI analysis is about more than just response time. The unique characteristicsof Java make a more comprehensiveKPI analysis necessary.
So far, the response time, CPU, andmemory KPIs we have introducedhave been application-specific, and inde-pendent of actual heap configurations.Therefore, they are used either for opti-mizing Java applications in developmentor for the hardware sizing of Java appli-cations in customer projects. From anadministrative and operational point ofview, we also need to tune the perform-ance of the application by finding theoptimal configuration of the SAPNetWeaver Application Server Javaand the Java VMs.
Subscribe today. Visit www.SAPinsider.com.
2 Heap is the memory space a Java Virtual Machine usesto create and store objects.There are a variety of JavaVM parameters to control the size, internal structure,and behavior of the heap. Automatic garbage collectionenables the Java runtime to take over the responsibilityof memory management.The Java programmer onlydecides when to create objects.These objects can thenbe used through references.Once an object has nomore references, it will be automatically detected andremoved by the garbage collector.
3 I recommend looking at the heap usage after severalfull GCs so you see only the live objects, not the garbage.This also ensures a high reproducibility of results.
8/13/2019 011000358700006399522006 e
http://slidepdf.com/reader/full/011000358700006399522006-e 5/5
Configuration TuningBecause general discussions of configura-tion topics aren’t within the scope of thisarticle, here we’ll focus on GC tuning —specifically, minimizing the performanceimpact of GC through optimal heap con-figurations. A lack in memory resourceswill result in very high GC activities.Therefore, when we consider GC tuning,we assume that the application is alreadyoptimized with respect to memoryconsumption and that the hardwareinfrastructure has been sized correctly. 4
To describe GC, two KPIs are key:
■ Interval — The average time betweentwo successive GC occurrences
■ Duration — The average elapsed timefor completion of a GC cycle
Both KPIs can be calculated froma GC log for a certain time period.Most Java VMs supported by the SAPNetWeaver Application Server Java,such as Sun Hotspot VM and IBM J9VM, provide generational GC.This meansthat by separating the heap into a youngspace and an old space, the most tempo-rary objects can be recycled through
faster, more frequent minor GCs inthe young space, while long-life objectsstay in the old space where they areonly checked by less frequent but long-running full GCs. Our GC tuning recom-mendations, using our interval and dura-tion KPIs, are as follows:
1. The interval of full garbage collectionshould be kept above several minutes,and the duration below 10 seconds.
2. The interval of minor garbage collec-tion should be above 1 second, and theduration below 200 ms.
Some Java performance-tuning guidesrecommend a single GC KPI, called the
relative garbage collection time , definedas duration divided by execution time.These guides recommend a relative GCtime of between 5% and 20% for a well-behaved Java application. We prefer toconsider our two GC KPIs using absolutetime values. In our experience, long GCdurations could cause a so-called Javaresonance phenomenon because waitingrequests are accumulated during the“stop-the-world pause” of GC cycles,especially when the application is underhigh load. In such a situation, perform-ance is destroyed and system stability isendangered.
A Look at the Numbers
Figure 3 shows concrete values for allof the KPIs discussed above. These num-bers give you a sense of what real loadtest results could look like, so you canstart to benchmark your system per-formance against that of your counter-parts in other companies.
These values are the result of theabove-named certified EP-ESS bench-mark with SAP NetWeaver Portal 6.0,SP13, running on a central server with1230 SAPS per CPU core.
SummaryCustomers often use custom load testsin large and mission-critical customerprojects to ensure that the installedsystem landscape is scalable and willperform optimally. These tests can bequite complex, especially in a Javaenvironment, due to garbage collectionand other Java-related factors — butthey don’t have to be.
By testing individual system compo-nents and employing suitable CPU andmemory KPIs — not just responsetime — you can significantly reducethe complexity of your load tests, betterunderstand Java-specific performanceissues, and optimize overall system
performance. And with SAP StandardApplication Benchmarks now availablefor Java-based applications, SAP devel-opers, partners, and customers havea comparative reference for theirperformance projects.
For more information on performanceand sizing, visit http://service.sap.com/ performance . And for more on theJava-based SAP NetWeaver Portalbenchmarks, visit www.sap.com/ benchmark .
Subscribe today. Visit www.SAPinsider.com.
4 To understand the impact of garbage collection on theaverage response time of load tests and related Javaprogramming guidelines, see “Taking Out the Trash:Avoid Performance Bottlenecks from Java GarbageCollection” by Susanne Janssen and Rudolf Meier inthe July-September 2005 issue of SAP Insider (www.SAPinsider.com) .
Xiaoqing Cheng received his Ph.D. in
computer science at the University of
Karlsruhe, Germany. He joined SAP in
1999, after conducting robotics and
artificial intelligence research and gathering
extensive experience in OOP software devel-
opment in the telecommunications industry.
As a member of the Performance, Data
Management, and Scalability team at SAP,
his areas of responsibility are performance,
benchmarking, and sizing for the presenta-
tion layer of SAP applications (SAP
NetWeaver Portal, BSP server, ITS) and
of SAP Web front ends (WebDynpro, BSP,
SAP GUI for HTML). He can be reached
Number of concurrentusers with 10 secondsthink time
1440users
CPU utilization 98%
Average response time 1.940 s
CPU time per userinteraction step
0.033 s
Framework space 208 MB
User session space 0.735 MB
Processing space 3.250 MB
GC interval 1.876 s
GC duration 0.200 s
Figure 3 Sample KPI values fromthe EP-ESS benchmark