cloud performance testing with loadrunner
TRANSCRIPT
Cloud Performance Testing
with LoadRunner
B6424
Richard Bishop & Gordon Appleby
3rd December 2014
© Copyright 2014 Vivit Worldwide
Cloud Performance Testing with LoadRunner
Richard Bishop & Gordon Appleby
• Richard Bishop
• Vivit Leader in UK, Board Member
• Performance Tester since 2000
• Background:
IT Infrastructure > MCSE >
Automated Software Testing
• Gordon Appleby
• Vivit Co-Leader in UK
• Performance Tester since 2000
• Background:
Ops > Dev > Infrastructure >
Messaging / Collaborative
Technology > Consultancy >
Automated Software Testing
Trust IV
Agenda
Introductions
Cloud timeline and hype
LoadRunner history and development
Choosing the right test tool
LoadRunner in the Cloud
Testing from the Cloud
• Use Cases
• Case Study
Cloud testing tips
Demo
Q&A / Discussion
What is the Cloud?
“Cloud is a style of computing where scalable and elastic IT-related capabilities are provided
as a service to external customers using Internet technologies”
Gartner 2010
1962 1989 1992 1999 2006 2014
1968 1990 1998
Last 10 years
Cloud Hype
LoadRunner and related products
LoadRunner6.0
PerformanceCenter
9.0
1990sO
n-P
rem
ise
In-c
loud
2006
PerformanceCenter
2008 2010 2014
StormRunner
LoadRunner in the Cloud
V2
LoadRunner8.0
LoadRunner9.0
2012
LoadRunner11.0
LoadRunner12.0
PerformanceCenter
11.0
PerformanceCenter
12.0
LoadRunner in the Cloud
V1
What does LoadRunner offer ?
Features such as TruClient make scripting easier.
Simple record / replay
Multi-protocol support, unlike most competition.
Single tool for Mobile, Ajax, Flex, HTML / HTML5, .Net, Java,
Silverlight, SOAP, Citrix, ERP and legacy
Browser and mobile device emulation built in.
Options to extend testing with network and service virtualisation.
Various cloud options:
Integration with Amazon & HP Cloud or other third parties.
SaaS options with StormRunner and Performance Centre.
Integration with multiple IDEs as well as CI and automated build systems.
Real time analytics built in, options to extend with SiteScope.
Diagnostic tool support, HP Diagnostics & third party tools.
Excellent reporting.
StormRunner
LoadRunner in the Cloud
V2
LoadRunner12.0
PerformanceCenter
12.0
LoadRunner and related products
Best of breed performance test tool
Integrates with other ALM products
Multi-protocol support, excellent reporting capabilities
Alternative to LoadRunner (on-premise)
On-demand licensing and hardware
Detailed reporting
Multi-protocol support
Cloud only
On demand usage
Uses standard LoadRunner scripts (web only)
Less detailed reporting but great for “burst testing”
Enterprise testing
Better integration with HP ALM suite
Dashboards, shared assets, ideal for global “follow the sun” operations
Which tool should I choose?
• Application Under Test
• Comms protocols
• Virtual user numbers
• Virtual user locations
• Tester locations
• Tester skillset
• Existing test assets
• Script complexity
• Test data requirements
• Security
• Budget
• Timescales
• Cloud?
Decision factors
Choosing a cloud-based test tool
StormRunnerLoadRunner in the Cloud
LoadRunner12.0
PerformanceCenterSaaS
PerformanceCenter
On-premise
Load Generators
Load Generators
LoadRunner on-premise
Load Generators
Load Generators
LoadRunner12.0
LoadRunner in the Cloud
LoadRunner in the Cloud
Load Generators
Load Generators
Performance Center on-premise
PerformanceCenter
On-premise
Load Generators
Load Generators
Performance Center SaaS
PerformanceCenterSaaS
Load Generators
Load Generators
StormRunner
StormRunner
Load Generators
So which product should I choose?
StormRunner
LoadRunner in the Cloud
LoadRunner
PerformanceCenter
Quick, high volume tests
Web only
Remote load generation
Short-term engagement (s)
Any protocol
Remote and local load generation
Advanced use e.g. NV, diagnostics etc.
More frequent testing
C of E approach
Remote and local load generation
Detailed reporting and analysis
Very frequent testing
Larger C of E / “follow the sun”
Dashboard for stakeholders
Integration with ALM etc.
LoadRunner in the Cloud
Web-based VM admin console
• Pre-configured servers, flexible license options.
• 100, 200 & 500+ vUsers on-demand
• Support for any other number
• All LR protocols supported
• Controllers and Load Generators
• CPU / RAM / HDD configurable
• With or without firewall
• Remote or local LGs
• All HP ALM integrations supported
• ALM / QC / UFT
• NV software pre-installed
• UFT and other software available
• On-demand availability and pricing
• PayPal or Account
• Pay for what you use
• Efficient license use
Is “The Cloud” any good for testing?
“Cloud computing is a model for enabling convenient, on-demand network access to a shared
pool of configurable computing resources that can be rapidly provisioned and released with
minimal management effort or service provider interaction.”
Source: The National Institute of Standards and Technology
BUT….
• Unproven technology for testing:
reliability/repeatability
• Steep learning curve
• Costs difficult to quantify
• Security concerns
• Performance
But does it work?
Use Case 1
Variable / unanticipated demandAd-hoc testing at short notice
Use Case 2
Small organisation, limited budgetPossibly considering open-source
PerformanceCenter
LoadRunner
LoadRunner in the Cloud
StormRunner
Use Case 3
Users outside firewallBetter end to end tests
Use Case 4
Agile Dev/OpsDevelopers do own testing, limited budget, pay-per-use desirable
Multi-protocol support.
SOA testing
Component-level testing
Service Virtualisation etc.
Integration with multiple IDEs as well
Integration with CI and automated build systems.
Use Case 5
High Virtual User NumbersLots of load generators needed
Virtual User Type
vUsers per Load Generator
Web (HTTP / HTML) 2000
TruClient 50
RDP 200
Citrix ICA 20
SAP GUI 20
Anon
Case Study
UK Financial organisation
250 vUser tests
Citrix and Web
Needed “outside firewall testing”
Needed remote testing
Tips when testing from cloud
Avoid pitfalls
Know your platform’s capabilities
Over provision
Smoke test AUT and test platform
Re-validate tests, test and re-test (best practice)
Use diagnostics software to simplify performance engineering
Monitor all infrastructure closely
Optimise test tool for virtualisation platform
Valid concerns
1 - “You don’t know what you’re getting with virtual machines”
2 - “Performance of load generators is too variable”
3 - Understand your application
4 - Time synchronisation
Research the subject
Use benefits of the cloud/virtualisation
Low cost
On-demand
Flex up & down
Over provision
Demonstration
LoadRunner in the Cloud
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change
without notice.
Thank youPlease ask / get in touch if you have any more questions:
Richard Bishop
@richardbishop
http://uk.linkedin.com/in/richardbishop
www.vivit-worldwide.org
blog.trustiv.co.uk
Gordon Appleby
@trustiv
http://uk.linkedin.com/in/gordonappleby