22 - performance tuning and stress testing - slide

© 2003 BEA Systems, Inc. 819

Performance Tuning and Stress Testing

At the end of this module you will be able to:

� Understand the need for systematic stress testing

� Use the console to recognize performance bottlenecks

� Tune WebLogic Server server parameters

� Tune WebLogic EJB parameters

Performance Tuning and Stress Testing-1

Module 22

Performance Tuning and Stress Testing-1


Road Map

1. Performance Tuning and Stress Testing

– The Need for Stress Testing

– When the CPU Is Bound

– When the IO Is Bound

– Tuning Various WebLogic Server Parameters

2. WLS EJB Tuning Parameters

3. Benchmarking Concepts

4. Web Benchmark and Grinder

Performance Tuning and Stress Testing-2Performance Tuning and Stress Testing-2


Introduction

� The performance of a WebLogic Server system is rated by its:– response time

– latency

– throughput

� The scalability of a WebLogic Server system is rated by its ability to add more:– concurrent users

– resources



Stress Testing

� Stress testing a WLS application requires:– plans that describes what will be tested

– ways to simulate live systems

– ways to identify performance problems

� Stress testing should include:– micro-benchmarks

– a test of the entire system



Benchmarks

� A benchmark is a measurement of the performance of your system.

� Benchmarks:– must be realistic

– should use the database access

– have clients connecting over a WAN

– should have real people testing the application

� Since all applications run differently, your application will be your best benchmark.



Changing WLS Properties

� Potential bottlenecks must be isolated to determine if they are indeed problems.

� Steps for changing properties:– create an appropriate and repeatable test

– measure baseline performance

– modify a single WebLogic Server property

– re-measure performance



CPU Bound

� The CPU bound situation is easy to spot since the CPU is running at or near 100%.

� Possible causes include:– garbage collection

– the Java application

� Solutions include:– monitoring the garbage collection

– profiling the Java application



Java Virtual Machine and the Heap

� JDK 1.3.x and above use code optimization based on usage statistics (Hot Spot).

� When stress testing, run more than one test to allow the optimizer to gather statistics.

� The Java Virtual Machine heap contains:– objects actively being used

– non-referenced objects

– free, available memory



Memory and Heap Size

� How much memory is enough?– use 80-85% of available RAM

� Upper and lower limits can be specified.– the minimum represents an amount always dedicated to

this JVM instance

– the maximum represents the most that can be taken by this JVM to instantiate new objects

– for best performance set the minimum and maximum values to be the same

Specify minimum and maximum heap with –ms and –mx:1001011110

java –ms384m –mx384m … weblogic.Server



Monitoring JVM Memory…



� With JDK 1.3.x, you can monitor garbage collection from the command line.

…Monitoring JVM Memory

java –ms128m –mx128m –verbosegc … weblogic.Server

Syntax of using –verbosegc:

1001011110



Determining Heap Size

� Turn on verbose garbage collection output and redirect stderr & stdout to log file.

� Analyze:– how often is GC taking place?

– how long is GC taking? (major GC should be shorter than 3~5 seconds)

– what is average memory footprint?

� May want to set generation sizes.

� Make sure the heap size is not larger than the available free RAM on system.



Detecting Low Memory Conditions

� WebLogic Server detects low memory by sampling the available free memory a set number of times during a time interval.



I/O Bound

� The I/O bound case has the following properties:– the CPU is not saturated

– performance is the same regardless of client load (no matter how many clients there you always have 50 TPS to the database)

� Types of I/O bound:– database bound

– network bound



Database Bound

� The bottleneck for your application is found to be your database.

� Configuration solutions:– use indexing

– allow more database connections

– use a fast, dedicated machine

– fine-tune your database (see vendor documentation)



Optimizing the Connection Pool

� More connections in a connection pool may boost performance.

� Don’t use more connections than the number of execute threads.

� Ideally, new connections never have to be created while the application is running.



Configuring a Prepared StatementCache for a JDBC Connection Pool

� WLS stores prepared statements used in applications and EJBs until it reaches the number of prepared statements that you specify as the prepared statement cache size in the JDBC Pool.

� When an application or EJB calls any of the prepared statements stored in the cache, WebLogic Server reuses the statement stored in the cache.



Network Bound

� The network can be your bottleneck if it gets saturated.

� Monitor the network to determine how much bandwidth is being used.

� Networks can saturate with as low as 50% bandwidth use.

� Possible solutions:– buy more bandwidth

– larger packets make better use of the network bandwidth

– OS-specific configuration



� The Performance Monitor is a Windows utility to monitor system characteristics

� The Performance Monitor displays statistics for:– processors

– memory

– file handles

– threads

– processes

– network activity

Using the Windows PerformanceMonitor



Monitoring Network Activities Under Windows

11

22

33 44

55



Tuning Windows TCP Parameters

� The Windows TcpTimedWaitDelay parameter can impact network performance.

• Registry location: HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

• Value Name: TcpTimedWaitDelay

• Value Type: REG_DWORD –Time in seconds

• Valid Range: 30-300(decimal)

• Default: 240 (decimal)

• Recommended value: 60(decimal)

• Registry location: HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

• Value Name: TcpTimedWaitDelay

• Value Type: REG_DWORD –Time in seconds

• Valid Range: 30-300(decimal)

• Default: 240 (decimal)

• Recommended value: 60(decimal)



Some UNIX Monitoring Tools

� UNIX provides many tools for monitoring system performance, such as:– sar: system activity (Solaris)

– mpstat: per-processor statistics (Solaris)

– vmstat: virtual memory statistics

– netstat: network statistics

– iostat: I/O statistics



Tuning Solaris TCP Parameters

� The following parameters can impact network performance:– tcp_time_wait_interval

– tcp_conn_req_max_q

– tcp_conn_req_max_q0

Default values and commonly used values:tcp_time_wait_interval : default 240000 ms (240 sec)

recommended 60000 ms (60 sec)tcp_conn_req_max_q : default 128

recommended >= 128tcp_conn_req_max_q0 : default 1024

1001011110



Threads & Performance

� WLS does all processing in thread pools.

� Threads are allocated as:– reader threads for handing network traffic

– execute threads for processing user requests

Execute Thread Pool

Reader Thread Pool

Incoming RequestsExecute Queue

9 9 9 999

1

1

1



Managing Threads

� The thread count should be increased when WLS is rejecting requests and the CPU is not running at 100%.

� This happens when calls to a database or network resource blocks.

Blocked Execute Threads

Full Execute Queue

Server Waits and So Do Clients

9 9 9 999 9 9 9

1

1

1



Socket Reader Threads

� Cluster performance is highly dependent on socket reader threads.

� Two factors determine socket communication efficiency:– type of socket reader implementation

– if Java socket readers are used, server must be configured with enough socket reader threads

� Two types of socket readers are:– native—provide best performance

– pure Java—forced to use inefficient socket polling



Configuring Reader Threads

55

22

44

33

11



Configuring Execute Threads…

33

22

11



…Configuring Execute Threads

55

44

� Typical values for the Thread Count are between 15-100 threads.



Built-in Execute Queues

� By default, WLS is configured with – a default execute queue (“default”) that contains 15

threads,

– 2 reserved queues for communicating with the WLS Admin Console (“__weblogic_admin_html_queue” that contains 2 threads and “__weblogic_admin_rmi_queue” that contains 10 threads)



Thread Usage Control Through Multiple Execute Queues

� An execute queue represents a named collection of execute threads that are available to one or more designated servlets, JSPs, EJBs, or RMI objects.

� Possible reasons to configure additional queues include:– optimize the performance of critical applications

– throttle the performance of nonessential applications

– remedy deadlocks in thread usage



Assigning Servlets, JSPs, EJB Objects to Execute Queues

� For servlets and JSPs, in the web.xml file, specify as an initialization parameter the queue name as the value of the wl-dispatch-policyparameter

� For RMI objects, use the -dispatchPolicy option in the weblogic.rmic compiler.

� For EJB objects, use the -dispatchPolicy option with weblogic.ejbc. weblogic.ejbc passes this option and its argument to weblogic.rmic when compiling the EJB.



Tuning Execute Queues for Overflow Conditions

� To detect and address overflow conditions in a queue, configure:– the threshold at which the server indicates an overflow

condition. This value is set as a percentage of the configured size of the execute queue (the QueueLengthvalue).

– the number of threads to be added to the queue when overflow is detected.

– the min and max number of threads available to the queue.



Detecting Stuck Threads

� WLS diagnoses threads as stuck if it is continually working (not idle) for a set period of time.

� If all threads in default queue become stuck, server changes its health state to "critical".

� A server changes its health state to "warning" if the following execute queues become stuck:– __weblogic_admin_html_queue

– __weblogic_admin_rmi_queue

– user-defined execute queue



Tuning Connection Backlog Buffering

� The AcceptBacklog attribute specifies how many TCP connections can be buffered in a wait queue.



Transactions and Performance

� A typical transaction commits or rolls back within 1-2 seconds.

� The default transaction timeout is 30 seconds.

� Suggestions on using transactions include:– keep the transactions short to help performance

– only use transactions when they are necessary



Configuring Transactions



Section Review

� The need for stress testing

� Various bottleneck conditions

� Tuning WebLogic Server parameters

In this section we discussed:



Road Map







EJB Pooling & Caching in WLS

� A free-pool stores identical EJB instances for EJB types stateless, entity and message-driven beans.

� A cache stores unique and identifiable EJB instances in memory to improve performance for EJB types stateful and entity beans.

� Pooling and caching are (mostly) set up by editing the file META-INF\weblogic-ejb-jar.xml.



Configuring Stateless Session EJB Pool

� You can manage the stateless session EJB pool by:– capping the number of instances

– setting an initial pool size

Snippet from META-INF\weblogic-ejb-jar.xml:<!–- Other Tags As Appropriate Here… -->

<weblogic-enterprise-bean><ejb-name>HelloEJB</ejb-name><stateless-session-descriptor>

<pool><max-beans-in-free-pool>15</max-beans-in-free-pool><initial-beans-in-free-pool>5</initial-beans-in-free-pool>

</pool> </stateless-session-descriptor><jndi-name>HelloWorld</jndi-name>

</weblogic-enterprise-bean>1001011110



Configuring Stateful Session EJB Cache

� You can manage the stateful session EJB cache by:– capping the number of instances

– setting a idle-timeout period

– setting the passivation strategy for the cached EJBs

Snippet from META-INF\weblogic-ejb-jar.xml:<!–- Other Tags As Appropriate Here… -->

<weblogic-enterprise-bean><ejb-name>HelloEJB</ejb-name><stateful-session-descriptor>

<stateful-session-cache><max-beans-in-cache>1000</max-beans-in-cache><idle-timeout-seconds>60</idle-timeout-seconds><cache-type>NRU</cache-type>

</stateful-session-cache> </stateful-session-descriptor><jndi-name>HelloEJB</jndi-name>

</weblogic-enterprise-bean> 1001011110



Configuring Entity EJB Pool & Cache

� You can manage the entity EJB:– pool by capping the number of instances and setting an

initial pool size

– cache by capping the number of instances and setting a timeout period

META-INF\weblogic-ejb-jar.xml Snippet:<weblogic-enterprise-bean>

<ejb-name>DogEBean</ejb-name><entity-descriptor><pool><max-beans-in-free-pool>15</max-beans-in-free-pool><initial-beans-in-free-pool>5</initial-beans-in-free-pool>

</pool><entity-cache> <max-beans-in-cache>100</max-beans-in-cache><idle-timeout-seconds>30</idle-timeout-seconds>

</entity-cache> </entity-descriptor> 10

01011110



Determining Pool Size

� Factors to consider to determine how many EJBs should be in the pool include:– number of threads set in WLS

– number of concurrent clients

– number of dependent backend resources (for example, database connections)

� Pool size will equally affect all servers of a cluster.



Determining Cache Size

� Some criteria that affect the size of your cache include:– the total amount of Java heap space available in the VM

– the number of different stateful session and entity beans deployed

– the amount of memory a single EJB consumes while active

– the number of concurrent clients

– clustering requires more memory because of replication of EJBs



ejbLoad() and ejbStore()Behavior for Entity EJBs

� By default, WLS calls ejbLoad() andejbStore():– a transaction is initiated for the entity EJB. The client

may explicitly initiate a new transaction and invoke the bean, or WLS may initiate a new transaction in accordance with the bean's method transaction attributes

– WLS calls ejbLoad() to read most current version of bean's persistent data from datastore

– when the transaction commits, WLS calls ejbStore()to write persistent fields back to datastore



Entity EJB Concurrency Strategies...

� Exclusive concurrency strategy:– places an exclusive lock on cached entity EJB instances

when the bean is associated with a transaction

– blocks other requests for the EJB instance until the transaction completes

– was the default for WLS 3.1 through 5.1



…Entity EJB Concurrency Strategies...

� Database concurrency strategy:– defers locking requests for an entity EJB to the

underlying datastore

– allocates a separate entity bean instance and allows locking and caching to be handled by the database

– is the default option



…Entity EJB Concurrency Strategies…

� In optimistic concurrency:– locks are not held in the EJB container or database

during a transaction

– the EJB container verifies that none of the data updated by the transaction has changed before committing the transaction

– the EJB container rolls back the transaction if any updated data changed



…Entity EJB Concurrency Strategies

� ReadOnly concurrency strategy:– is used only for read-only entity beans

– activates a new instance for each transaction so that requests proceed in parallel

– configure WLS to check read-only entity beans with read-timeout configured whether cached data is older

– invalidates read-only beans when invalidate() is called and sends multicast message through cluster to invalidate cached copies



Combined Caching with Entity EJBs

� Combined caching allows multiple entity beans that are part of the same J2EE application to share a single run-time cache.

� Specify an application-level cache in weblogic-application.xml.

� Specify an entity-descriptor in weblogic-ejb-jar.xml to use the application-level cache.



Caching Entity Beans Between Transactions

� Caching Entity Beans between transactions is:– allowed only if the concurrency strategy is Exclusive, ReadOnly or

Optimistic

– disabled with Exclusive Concurrency

� In Optimistic concurrency strategy EJB container:– reuses the cached values from previous transactions.

– ensures that the updates are transactionally consistent by checking for optimistic conflicts at the end of the transaction

– broadcast notifications for updates of optimistic data to other cluster members to help avoid optimistic conflicts

� EJB Container always performs long-term caching of read-only data.



Section Review

� WebLlogic Server specific EJB tuning parameters




Road Map




– Understand Why to Benchmark Applications

– Describe What and When to Benchmark

– Describe Benchmark Results




Why to Benchmark

� Benchmarks:– allow you to determine if performance requirements are

being met

– can be used to determine that a use case is performing poorly

– can determine if tuning the application, operating system, or WLS increased or decreased performance



What to Benchmark

� Benchmarks should be performed on:– functional areas of the application

– the full application

� Benchmarks can be used to record:– timing measurements

– hardware measurements

– resource consumption

– timing measurements under varying loads



When to Benchmark

� Benchmarks should be performed on:– functional areas once they are written

– the whole application once it is finished

� Benchmark the application after any change to the application, an operating system parameter, or a WLS parameter.



How to Benchmark

� Perform a benchmark with a commercial or freely available tool, such as the:– Web Benchmark (BEA)

– Grinder

– SPECjbb2000

– ECPerf

– WebBench

– many others

� Alternatively, you can write your own measurement code to perform benchmarks.



Section Review

� Why to perform benchmarks within a development project

� What to perform benchmarks on

� When to benchmark

� Reporting benchmark results




Road Map





– Describe the Web Benchmark

– Configure the Web Benchmark

– Describe a Benchmark Example That Uses the Web Benchmark

– Grinder



Introduction to the Web Benchmark

� The Web Benchmark:– is an Alpha BEA product, suitable for simple testing

– is a Java program that is executed from the command-line

– can be used to stress test HTTP servers

– allows administrators to hit pages on a particular site with multiple clients/threads

� The time each client/thread takes to complete is reported to the output.



Configuring the Web Benchmark

Follow these steps to configure the Web Benchmark:1. Download Web Benchmark.

The Web Benchmark, distributed in a zip file bench.zip, can be retrieved:

– From BEA’s developer site:

http://developer.bea.com/ftp_bin/bench.zip– In the student’s folder:

%STUDENT%\lab28\benchmarkKit\bench.zip

2. Unzip bench.zip to a directory.Create a new directory and unzip the contents of bench.zip to

this directory.

3. Update the system CLASSPATH.

Add the newly created directory to your system's CLASSPATH.



Executing the Web Benchmark

java BenchClient <#_of_threads> <host> <port> "<URI>" <iter>

General Syntax:

#_of_threads := The number of clientshost := The server’s DNS name or IP addressport := The port the server is listening onURI := The virtual mapping, extra path information, and query stringiter := The number of requests each client will send

Examples:

java BenchClient 15 192.168.1.100 7001 "/index.jsp" 45

java BenchClient 20 localhost 4001 "/getItem.jsp?id=4" 5

1001011110



Example: The Application…

Promote an Employee

Raise an Employee’s Salary

Fire an Employee

Hire an Employee

Manager

List all Current Employees



…Example: The Application…

employeeinfo

getConnection()getAllEmployees()updatePosition()updateSalary()removeEmployee()addEmployee()getNewID()getAllEmployeeInfo()

HRdatabase

allemployees.jsp

payroll.war

hire.jsp

fire.jsp

raise.jsp

promotion.jsp

welcome.html



…Example: The Environment…

� The WebLogic Server parameters are:– version = 7.0– service pack = 1– heap size = 128 MB– number of execute threads = 15– percentage of socket readers = 33– accept backlog = 50

� The hardware parameters are:– processor speed = 600 MHz Pentium III– amount of RAM = 512 MB– operating system = Windows 2000 version 5



…Example: Starting a Test…

20 clients each making 15 requests to the URL http://localhost:7001/payroll/welcome.html

http://localhost:7001/payroll/welcome.html

URL to test:

Web Benchmark command:

1001011110

java BenchClient 20 localhost 7001 "/payroll/welcome.html" 15



…Example: The Results



…Example: Starting A Test…

Note: The URL and the Web Benchmark command are single-

line commands.

URL to test:http://localhost:7001/payroll/promotion.jsp?employee=9+Ryan

+Lewis&newposition=Senior+Programmer&btnSubmit=Update

+Employee+Information

Web Benchmark command:

1001011110

java BenchClient 10 localhost 7001 "/payroll/promotion.jsp

?employee=9+Ryan+Lewis&newposition=Senior+Programmer

&btnSubmit=Update+Employee+Information" 2



…Example: The Results



Grinder

� Grinder:– is load testing software released under the GNU public

license

– support a graphical view of benchmark results

– uses java plug-ins as clients to test code

– provides sample plug-ins for testing HTTP and EJB Services

– Can be downloaded from http://grinder.sourceforge.net



Grinder Installation

Follow these steps to configure the Grinder:Step 1: Download GrinderGrinder is distributed in a JAR file and can be downloaded from http://grinder.sourceforge.net.

Step 2: Unzip the file to a directory.Extract the files to a directory.Step 3:Update the system CLASSPATH.Add the grinder.jar file to your CLASSPATH. The grinder.jar file is located in the lib directory.Step 4: Setup a test directory.Step 5: Execute Grinder.Step 6: Execute Grinder’s console.



Setting Up a Test Directory for Grinder

� The test directory:– contains the grinder.properties file

– can be located anywhere on your computer

� The grinder.properties file contains:– Configuration information about tests

– Configuration property name/value pairs



Grinder.properties file

Example of a grinder.properties files:grinder.processes=5grinder.threads=20grinder.cycles=15

grinder.receiveConsoleSignals=truegrinder.multicaseAddress=228.1.1.1grinder.multicastPortel=1234

grinder.reportToConsole=truegrinder.console.multicastPort=1235

grinder.logDirectory=loggrinder.appendLog=False

grinder.plugin=net.grinder.plugin.http.HttpPlugin

grinder.test0.sleepTime=19177grinder.test0.parameter.url=http://localhost:7001/securitygrinder.test1.description=Imagegrinder.test1.sleepTime=40grinder.test1.parameter.url=http://localhost:7001/security/logo.gif

1001011110<grinder.properties>

This Directory must existbefore starting Grinder



Using the Grinder Console

11

22 33



Summary Statistics



Benchmark Results…

� Benchmark results should be reported with details about:– the application's design

– the type of operating system

– the version and service pack of WLS

– the hardware

– performance characteristics of external systems



… Benchmark Results

� A metric should be developed which tabulates the benching results:

PendReq# of

ServReq# of CPU

100

100

100

100

100

# of Iterations

Total Time

for Execution

HeapUsage(kb)

# of IdleQue

Min

http:\\localhost:7001\estore\control category

?productList_Dogs_startIndex=5

&productList_Dogs_next_true

&category_id=DOGS&x=58&y=14

http:\\localhost:7001\estore\control\category?category_id=BIRDS

http:\\localhost:7001\estore\control\cart?action=purchaseItem&itemId=EST-3

http:\\localhost:7001\estore\category?category_id=DOGS

http:\\localhost:7001\estore\control\main

# of MEM

# of Clients

URL

� It may be beneficial to store the benchmark results in a database also. It would allow administrators to observe long-term trends and establish reliable baseline data.



Using JMX Programming to Log Data During Load Testing

� Using JMX programming for logging purposes avoids any UI overhead during heavy load testing.

� Some useful runtime Mbeans for performance monitoring during load testing:– JVMRuntimeMBean, ExecuteQueueRuntimeMBean, JTARuntimeMBean, EJBCacheRuntimeMBean, EJBPoolRuntimeMBean, EJBTransactionRuntimeMBean, JDBCConnectionPoolRuntimeMBean, JMSDestinationRuntimeMBean, ServletRuntimeMBean, WebAppComponentRuntimeMBean



Section Review

� An introduction to the Web Benchmark

� How to configure the Web Benchmark

� A benchmark example that used the Web Benchmark

� An introduction to Grinder




Module Review

� The need for stress testing

� Various bottleneck conditions

� Tuning WebLogic Server parameters

� Tuning WebLogic EJB parameters

In this module we discussed:


22 - performance tuning and stress testing - slide

Documents