performance engineeringsigist.org.il/_uploads/dbsattachedfiles/performance_eng.pdf · goals of...
TRANSCRIPT
Performance
Engineering
Salpeter Ariel
May 26 ,2009
Performance Engineering - PCoE
2
Introduction
Definitions
Performance Engineering
Project Flow - SDLC
PCOE
3
Introduction
Definitions
Performance Engineering
Project Flow - SDLC
PCOE
4
Copyright ©
What Is Performance?
Computational performance
How many instructions are required to execute a statement?
How much overhead does a particular virtual method call incur?
Should I use a quick-sort or a bubble-sort here?
5
Copyright ©
What Is Performance?
Computational performance
RAM foot-print
Amount of memory needed to run the application
Virtual memory
Good in dev, poor in deploy
Consider in design !
Co-hosting?
6
Copyright ©
What Is Performance?
Computational performance
RAM foot-print
Startup time
GUI application (BRRRR…..).
Web application.
7
Copyright ©
What Is Performance?
Computational performance
RAM foot-print
Startup time
Application scalability
A system that doesn't scale well
0
50
100
150
200
0 500 1000 1500 2000
Number of Users
Avera
ge R
esp
on
se
Tim
e
Series1
A system that scale well
0
5
10
15
20
25
0 500 1000 1500
Number of Users
Avera
ge R
esp
on
se
Tim
e
Series1
8
Copyright ©
What Is Performance?
Computational performance
RAM foot-print
Startup time
Application scalability
Perceived performance
Mouse cursor -> wait cursor
Background tasks
Responsive UI
9
Copyright ©
10
Copyright ©
Some Saying About Load & Performance
“There is no time now to create the optimize system, just develop a prototype and then…”
“The machines that we have are strong enough to hold the performance required for the system, no need to think about design aspects for load…”
“Why we need to invest on load, we’re delivering functionality?”
“There is no need to study exhaustively…”
11
Introduction
Definitions
Performance Engineering
Project Flow - SDLC
PCOE
12
Load & Performance – Wikipedia
• Is the process of creating demand on a system or device and measuring its response.
Load testing
• Is testing that is performed, from one perspective, to determine how fast some aspect of a system performs under a particular workload.
Performance testing
• Performance engineering within software engineering, encompasses set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the system.
Performance Engineering
Goals of Performance Engineering
Improve product quality
By producing reliable and tuned products that enhance our industry leadership position and are deliverable and serviceable
promote Company competitive advantage
By reduce COGS through benchmarks processes
high Performance products, that successfully Address Our customers’ business needs .
14
Software Performance Engineering
Build Performance into the Product rather then fixing it later
Lower the cost of handling performance issues
Requirements Design Implement Integration Test Maintain
$1 $2.50 $6.50 $16 $40 $110
“Clearly, it pays to invest effort in finding requirements errors early and correcting them in, say, 1
man-hour rather than waiting to find the error during operations and having to spend 100 man-
hours correcting it.” [Boehm]
Relative values
Software Performance EngineeringLower Costs
15
Performance EngineerBring Focus on Performance in Product
• Performance Testing
• Performance Tuning
• Analyzing Performance impact during Design
• Performance Benchmarking
• Capacity Planning and Sizing
• Defining Performance Requirements
• Enable Performance focus during Implementation
• Problem Diagnostics
• Collaborating with CS for Customer Performance Issues
16
Performance Engineer Skills
Networking knowledge
Load tools/simulators
Protocols knowledge
Operation System knowledge
monitoring
Analysis
Scripting knowledge
programming
Hardware knowledge
Database knowledge
17
Performance Engineer interfaces
18
SYS EngRequirements1. Definition of product performance 2. Database population pattern 3. external processes to simulate 4. Usage Model5. Density of events 6. Transaction response time criteria7. Band width calculation and
limitation
Performance Engineer interfaces
19
Introduction
Definitions
Performance Engineering
Project Flow - SDLC
PCOE
Performance Project Flow Diagram
P.20
Development Flow
PCoE Activity
PCoE Deliverable
Push Performance Upstream in SDLC
Component sizing, resource budget metrics, configuration tuning Build models, prototypes, Validate/POC
BenchmarkingTesting identified scenariosIdentify key performance
scenarios
ProductionQADesign ImplementationRequirements
Define performance
requirements
Software Performance Engineering
Software Development LifeCycle
22
SDLC Phase: Requirements
Pre-requisites: Product requirements exists
Goals Identify high level risks during requirements/design phases, mitigate or
manage them
Practices Analyze product requirements
Assess the performance risk of each requirement
Come up with recommended mitigation
After assessing all requirements, performance requirements are formed based
on the risky areas
Software Performance Engineering
Performance Risks Analysis
Overlay on Software Life Cycle
23
SDLC Phase: RequirementsPre-requisites Performance Risks Analysis Performance Requirements
Goals Define measurable performance objectives that are aligned with customer
needs and agreed upon all stakeholders
Practices Analyze customer environments (Workload Characterization) Determine the right workload for the testing environment.
Workload consists of: number of users, concurrent active users, data volumes, key scenarios and transaction volumes
Define objectives for: Response Time, Throughput, Resource Utilization Communicate Performance Objectives Get commitment for Performance objectives
Software Performance Engineering
Performance Objectives
Overlay on Software Life Cycle
24
SDLC Phase: Design
Pre-requisites:Performance Objectives
Goals
Ensure application key design choices takes performance principles under
consideration
Practices Learn design tradeoffs for performance and scalability Apply a principle-based approach on design Identify and use a performance and scalability framework Learn design considerations for scaling up and scaling out Minimize communication and data transformation overhead Improve application concurrency Manage resources efficiently Cache application data effectively Manage application state efficiently Design an efficient presentation layer, business layer and data access layer
Software Performance Engineering
Performance Design Guidelines
Overlay on Software Life Cycle
25
SDLC Phase: Design, Develop, TestPre-requisites: Performance Objectives High Level Design
Goals Characterize and understand the system’s behavior so that effective decisions may be
made.
Practices Identify key scenarios Identify workload Identify performance objectives Identify budget Identify processing steps Allocate budget Evaluate: run models, prototype Validate: measure in load tests
Software Performance Engineering
Performance Modeling
Overlay on Software Life Cycle
26
SDLC Phase: Design, Develop, Test, Deploy, Maintain
Goals Identify how the application performance stands in relation to defined performance goals Helps to identify the bottlenecks that affect application performance
Practices Measure System and Platform metrics Measure Network Profiling Analyze log files Instrument the application
Software Performance Engineering
Performance Measuring
Overlay on Software Life Cycle
Software Performance Engineering
SDLC Phase: Test
Goals Identify how well application performs in relation to performance objectives
Practices Identify key scenarios. Identify application scenarios that are critical for performance. Identify workload. Distribute the total application load among the key scenarios
identified in step 1. Identify metrics. Identify the metrics that you want to collect about the application
when running the test. Create test cases. Create the test cases where you define steps for executing a
single test along with the expected results. Simulate load. Use test tools to simulate load according to the test cases and to
capture the result metrics. Analyze the results. Analyze the metric data captured during the test.
27
Overlay on Software Life Cycle
Performance Testing
28
SDLC Phase: Develop, Test, Deploy, Maintain
Goals identification and systematic elimination of bottlenecks to improve performance
Categories for Tuning
Application. Includes the application configuration parameters.
Platform. Includes the host operating system, together with Web and database servers
System. Includes the hardware resources on a server. These resources include CPU,
memory, disk I/O, and network I/O.
Practices Assess the problem and establish numeric values that categorize acceptable behavior. Measure the performance of the system before modification. Identify the part of the system that is critical for improving the performance. This is called
the bottleneck. Modify that part of the system to remove the bottleneck. Measure the performance of the system after modification
Software Performance Engineering
Performance Tuning
Overlay on Software Life Cycle
29
SDLC Phase: Test, Deploy
Goals Assess the relative performance of an object, normally by running a number of standard
tests and trials against it Equip field personnel with valuable information to facilitate product sales
Details
Benchmarks provide a method of comparing the performance of various subsystems
across different chip/system architectures. Benchmarking is helpful in understanding how
the database manager responds under varying conditions. You can create scenarios that
test deadlock handling, utility performance, different methods of loading data, transaction
rate characteristics as more users are added, and even the effect on the application of
using a new release of the product.
Software Performance Engineering
Performance Benchmarking
Overlay on Software Life Cycle
30
SDLC Phase: Test, Deploy, Maintain
Goals Planning for growth and forecasting peak usage periods in order to meet system and
application capacity requirements
Practices Transaction cost analysis
Compile a user profile Execute discrete tests Measure the cost of each operation Calculate the cost of an average user profile Calculate site capacity Verify site capacity
Predictive analysis Collect performance data Query the existing historical data Analyze the historical performance data Predict the future requirements
Software Performance Engineering
Capacity Planning
Overlay on Software Life Cycle
31
Introduction
Performance Eng
Project Flow - SDLC
Interfaces
PCOE
Executive summary
P.32
Solution :
performance Center of excellence-
central scalable performance group that produces
significant quality improvement and synergy according to
Comverse business strategy
Lack of synergy between Performance teams ,and
Performance engineers abandonment leads to
unbalanced Performance level in the various products
33
What is a CoE- Center of Excellence?
TECHNOLOGY
METHODOLOGY
EXPERTS
Focus & Expertise
DriveOptimization
ProveValue
ConsolidateKnowledge
DisseminateExpertise
Maximize Business Value
Goals of the Performance Group
Improve product quality
By producing reliable and tuned products that enhance our industry leadership position and are deliverable and serviceable
promote Comverse competitive advantage
By reduce COGS through benchmarks processes
Increase efficiency
Enable synergetic leverage of the company performance teams and experts
Reduce performance engineers abandonment
By creating technical career path for performance engineers34
high quality products, that successfully Address Our customers’ business
needs and enhance our competitive edge
Platform Group Gathering – Q1 2008
Comverse PCoE – Products
• MMS
• SMS
• Unified Messaging
• Mobile IM
• Network address book
• Mobile Email
• Mobile Gateway
Advantages
Expertise in Performance Area Solid performance testing methodology
Monitoring best practices
Expertise in load Testing, diagnostics and profiling tools
Scalable Performance Team Quick ability to recover in case of leave
Shares human resources upon need to handle peak times
Experience in recruiting engineers with appropriate talent/potential
Technical Career Path for Performance Experts Performance engineering oriented management
Long term vision
Performance experts home
36
Advantages
Bring Focus on Performance in Product Development
Analyzing performance impact during design
Defining performance requirements
Capacity planning and sizing
Drive performance agenda across organization
Centralized Performance specific knowledge base
Performance is our profession
Collaboration and Visibility PCoE representative attends product status meeting and provide performance
status
work on daily basis with developers
Team manager works closely with QA manager on priorities and realigns efforts
per demand
Delivering Scalable & Reliable Product Development awareness & attention to performance aspects
Ability to handle customers’ escalations
37
Requirements
phase
Design
phase
Implement
phase
Execution
phase
38
performance Eng in Project life CycleKnowledge Base/
Group Experts
TL review PCEO reviewTL review
Performance
Eng
SUT Testing
PU&PCOE review
PCOE life cycle
The End
40
Scalability Testing
• Complementary to load testing
• Workload Simulated: Normal & Peak workloads expected for a given system configuration
• Test Duration: typically few hours
• Examined Aspect
− Resource Utilization Levels
− Others: only if the load testing didn’t reveal relevant issues (e.g. response times)
• Can be used as a basis for Capacity Planning models
• Scale up or Scale Out tests might be considered in case system configuration is not sufficient for examined workload
Quantify Performance Under Increased Levels of Workloads
Scalability Testing
Platform Group Gathering – Q1 200841
• Load Testing
• Stress Testing
• Scalability Testing
• Other activities outside testing zone
− Capacity planning
− Performance Optimization
− Performance Tuning
Performance Tests Types
Platform Group Gathering – Q1 200842
Load Testing
• Workload Simulated: 120% of Normal & Peak Workload
• Test Duration: varies from few hours to few days
• Examined Aspects
− Response times
− Throughput rates
− Resource utilization levels
− Breaking Points (in case it happens below the peak workload conditions)
Test Behavior Under Normal & Peak Workload
Load Testing
Platform Group Gathering – Q1 200843
Stress Testing
• Workload Simulated: Extreme workload. Might reach 500% of Normal & Peak workload
• Duration: Typically less then an hour
• Goal: unearth application bugs that surface only under high load conditions
• Examined Aspects
− Synchronization issues
− Race conditions
− Memory leaks
Evaluate Behavior Beyond Normal & Peak Workloads
Stress Testing