joerg nalik - sap · pdf filesap labs 2003, stress testing, joerg nalik / 27 sap j2ee 6.20...

2003 Business Information, Technology & Infrastructure Forum

Stress Testing Component Landscapes for Performance Optimization and Sizing

Joerg NalikSAP LABsSession Code: 1612

SAP LABs 2003, Stress Testing, Joerg Nalik / 2

High level Goals of Stress Testing

• Achieve required transactional throughput

• Achieve good system response times for end users

• Minimize hardware costs

Stress testing helps to achieve and verify these goals!


Overview

• Introduction

• Stress Testing Project Management

• Stress Testing Strategy

• Monitoring and Tuning– Abap components– Java/J2EE components– Web-servers– Network– Response times– Business processes


Introduction


Background of this presentation

• Experiences from SAP internal performance testing of the SRM solution and other SAP products.

• SAP standard procedures and recommendations for monitoring, tuning and sizing.

• Managing complex stress testing projects.

• Contacts with customers who do stress testing.

• Helping SAP support with customer system performance issues.


Component landscapes

SAP’s business solutions follow a component approach. In general, multiple application components of a solution are installed on multiple hardware servers and bound together via the NetWeaver Infrastructure components.

EBP, Buyer

NetWeaver-XI

SUS, Seller


Stress Testing Project Management


Technical Goals of Stress Testing

• Complement functional testing: Verify system stability under load

• Verify hardware capacity planning

• Test for scalability and performance bottlenecks

• Check end user response time stability under high load

• Optimize performance through monitoring and tuning


Challenges of Stress Testing

• Find a suitable test landscape:– Co-use of QA or Production landscapes needs

good coordination. Temporary exclusive access to a test landscape is recommended.

• Tools for producing test load– line up a group of manual testers: only for one-

time tests.– use of QA or hard-beat monitoring tools: might

not be able to generate high load, lack of integration.

– use of load testing tools: license expense.

• Defining test strategy, cases and goals


Build your Team

• Environment Specialist(s):– Experienced in hardware and network

infrastructure

• Application Specialist(s):– Trained in application configuration and

administration

• Business Scenario Specialist(s):– Experienced in business processes and their

implementation across application components

• Test Engineer(s):– Trained in using load testing tools


Project Milestones

• End of project definition phase for resources, test cases and test goals.

• All application components are installed and interconnected (application integration).

• Test cases are implemented and run functionally correct.

• Test-Tuning cycle is finished.

• Final measurements, documentation of results.


Stress Testing Strategy


Component versus Scenario Stress Tests

• Application integration to Business Solutions leads to solution landscapes consisting out of many individual infrastructure (example NetWeaver) and application components (example SRM).

• Solution business scenarios are executed by users with different roles and might include overall >100 screen changes.

! Reducing complexity of stress testing:" testing of important sub-scenarios only" one user role only in a sub-scenario.


Choosing Test cases

• What causes most CPU utilization in a productive environment?

• Which sub-scenarios are response time critical (end-user satisfaction)?

• Example: SAP’s User Management Engine– 90% case: Login and Logout - performance is critical – 10% case: Administer User Accounts, important for

functional correctness, but not critical for performance testing

Tip: Review capacity planning for CPU intensive scenarios.


Choose Metrics and Goals

• Metrics: What should be measured? What impacts CPU and other resource utilization?

• Goals: When is it good enough? Examples:– 100 Purchase Orders per hour with– average 5 line items per Purchase Order– average response times below 3 seconds


Create and customize test scripts

• The load testing tool should allow:– recording of scenarios, – parameterization and run-time substitution, for

instance for user names and business objects IDs.

• To avoid artificial resource leaks, bracket every scenario with login and logout.


Important Features for Stress Test Execution

• programmable load variation over time

• for web-applications: correct simulation of static content download, browser caching, cookie handling and connection handling (keep-alive).

• integration of internal load test tool and external monitors of the operating system and other components (application servers, databases, …)

• automated test result recording and reporting.


Test Methodology

Considerations for Concurrent Users versus Transaction throughput:

• How many Concurrent Users can be served by an application system?

– The number of concurrent users is meaningless without defining “think time”. Think time is the time needed by users to send a subsequent request after receiving a response. Think time helps to simulate real user behavior.

• How many Transactions per hour can be handled?– Example: 100 conc. users, 60 sec. think time, 2 sec. resp.

time, 40 pages/transact.:– Transact./h = 100 * 3600 /[( 60 + 2 ) * 40] = 145

TIP: Backend session management load is impacted by number of concurrent users and think time. Consider session time out settings, connection time outs ….


Test Strategy I

• Determine maximum sustainable load– “Ramp-up” users slowly over time. Monitor CPU

consumption and transactions/second.


Test Strategy II

• Use constant load Test for stability verification, and UI response time.

– Start with a fast ramp-up, long period of constant load (hours), then quick ramp-down (model of SAP benchmarks)

– Each simulated user performs many iterations during the test. Use about 70% of max. load known from ramp-up test.

– Monitor in detail: CPU, memory, network traffic, Java Garbage Collection ….


Test Strategy III

Strategy for all load tests:• Start with small and short test runs for detecting and fixing

easy problems and tuning. When small tests are successful continue with next higher load level tests.

• Example:

• Goal: processing of 10,000 Transactions/12 hours, Test time 5 days.

• Big-Bang approach: about 5 test/improvement runs possible.

• Start small approach:– 1st day: 12 runs of 100 Transactions– 2nd day: 6 runs of 500 Transactions each– 3rd day: 3 runs of 2000 Transactions each– 4th+5th day: 1 run with 10,000 Transactions each day– Total: 23 test/improvement cycles


Monitoring and Tuning


What to monitor?

Load Test inherent monitors:• Transactions/time(second,hour) • number of simulated users• Sum of Server+Network request response times• Hits/second• Data volume download and upload• For web-appl.: Breakdown of page content download volumes

and times (#, size and download time of mime files, breakdown of network roundtrips for connect, https verification, data package downloads ….)

Other monitors, preferable integrated in load tool:• OS monitors for: CPU, Memory, Disk I/O, Network, …• Application and infrastructure component monitors (application

servers, databases, ….)


What to tune?

• Abap components

• Java/J2EE components

• Web-servers

• Network

• Response times

• Business processes


Abap component tuning

Useful Abap monitor transactions:

• ST02: check buffer sizing

• ST04: database monitor, in particular for locking

• ST06: OS monitoring

• ST03: Application monitoring

• ST05: SQL-, RFC-, Enqueue- tracing


Java Virtual Machine tuning

Start memory configuration:• for Sun/HP JDK 1.3.x: tune “young” space to be about 1/3rd of

total Java heap. Example:– -Xms256M, -Xmx512M– -XX:NewSize = -XX:MaxNewSize = 192M– see details in OSS note 552522

• Disable programmed forced Garbage collection: -XX:+DisableExplicitGC

• Enable Garbage Collection monitoring: -verbose:gc

Check that time spent in Garbage Collection is <5% of CPU time. Otherwise try to tune Java runtime parameters.

Useful GC trace viewers: HPJtune, SUN GC Portal


SAP J2EE 6.20 tuning

Monitor and tune:

• Thread pools

• DB-connection pools

• concurrent session/network socket usage

• …

Reference:

Session 1603: Performance Tuning your Java Engine by Marc Chan

Monday, October 27, 2003; 2:40 PM - 3:50 PM


Web-server tuning

• The raw (un-buffered) size of a complete Web-site is in the range of 100KB and it takes 10 to 20 request/response cycles to download all web-objects of one page. Reduce volume to about 10KB/page through 2 to 5 roundtrips:

– If access is via WAN enable compression of certain mime files (.js, .css) and dynamic html content.

– Tag mime files with an expiration date in order to avoid repetitive downloads.

– configure “keep-alive” for connections.

• References: OSS notes: 553311, 350646


Networks inner workings

Network tuning can have a major impact on response times through WAN, Internet and modem connections. The following are key parameters:

• Bandwidth: impacts transmission times for a certain data volume.

• “ping time” between sending and receiving end. Good measure for latency time for one TCP/IP network roundtrip

• number of TCP/IP roundtrips for one application roundtrip. TCP/IP roundtrips occur for:

– opening a connection. Tune keep-alive to keep connections open longer. However, watch number of concurrently open sessions in application servers, which might increase.

– https certificate verification– first package download– subsequent package downloads. Might be reduced through use of

compression


Estimating network timesExample: • 10KB dynamic website content• 3 application roundtrips, • 1 connect, 1 https, 1 first package TCP/IP roundtrip• 0.1 second ping time• 56 kbps modem connection bandwidth

! Bandwidth time = 10 KB * 8 / 56kbps = 1.4 seconds Accumulative latency time = 3 appl. r. * 3 TCP/IP r. * 0.1 sec. ping time = 0.9 sec

Total maximum time = 2.3 second

Note: application roundtrips might be executed in parallel and actual network times therefore might be shorter.


Response time contributions

Browser

Web Server

Application Server

staticcontent

dynamiccontent

• Browser rendering time

• Network bandwidth and latency times

• Web server response times

• Application server response times

Due to use of parallel connections and multi threading of browsers response times might be shorter than sum of individual times.


Response time optimization summary

Bad response times are not necessarily caused by the application (server). Response times can be optimized in all tier components and the network. Consider:

– for network times: compression, expiration tagging of mime files, local proxy server.

– for browser rendering times: faster desktop hardware.

– for web and application servers: tuning, keep-alive configuration, faster hardware.


Business process tuning

Individual business scenarios might be tuned in their own particular ways. Example:– Reverse auctioning in SRM Live Auctioning

Cockpit (LAC):# Each bidder is polling latest bid price updates every few

seconds.# Bids are done with fixed size bid decrements.

– Server load can be reduced by increasing bid update interval times and by reducing #of bids/auction through setting larger bid decrements.


Summary

Key success factors for component landscape stress testing:

• manage stress testing as a project

• define test goals and deliverables at the beginning

• define your test methodology

• consider use and features of load testing tools

• start small and work your way up

• plan for monitoring/tuning improvement cycles


Acknowledgement

We like to thank our partners for their support of the effort required to develop input for this presentation:

– HP for hardware and engineering support.– SUN Microsystems for hardware and engineering

support.– Mercury Interactive for load testing software and

engineering support.


2003 Business Information, Technology & Infrastructure Forum

Thank You For Attending!

Please remember to complete and return your evaluation form following this session.

Session Code: 1612


• No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

• Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

• Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation.

• IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix and Informix® Dynamic ServerTM are trademarks of IBM Corporation in USA and/or other countries.

• ORACLE® is a registered trademark of ORACLE Corporation.• UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group.• Citrix®, the Citrix logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®,

VideoFrame®, MultiWin® and other Citrix product names referenced herein are trademarks of Citrix Systems, Inc.

• HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

• JAVA® is a registered trademark of Sun Microsystems, Inc. • JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under

license for technology invented and implemented by Netscape. • MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and

Copyright 2003 SAP AG. All Rights Reserved


• Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die aus-drückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankün-digung geändert werden.

• Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten.

• Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® und SQL Server®sind eingetragene Marken der Microsoft Corporation.

• IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix und Informix® Dynamic ServerTM sind Marken der IBM Corporation in den USA und/oder anderen Ländern.

• ORACLE® ist eine eingetragene Marke der ORACLE Corporation.• UNIX®, X/Open®, OSF/1® und Motif® sind eingetragene Marken der Open Group.• Citrix®, das Citrix-Logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®,

VideoFrame®, MultiWin® und andere hier erwähnte Namen von Citrix-Produkten sind Marken von Citrix Systems, Inc.

• HTML, DHTML, XML, XHTML sind Marken oder eingetragene Marken des W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

• JAVA® ist eine eingetragene Marke der Sun Microsystems, Inc. • JAVASCRIPT® ist eine eingetragene Marke der Sun Microsystems, Inc., verwendet

unter der Lizenz der von Netscape entwickelten und implementierten Technologie.

Copyright 2003 SAP AG. Alle Rechte vorbehalten

joerg nalik - sap · pdf filesap labs 2003, stress testing, joerg nalik / 27 sap j2ee 6.20...

Documents