1 Private and Confidential 1 Private and Confidential

Continuous Performance TestingMikhail Klokov, QA Technical Director08/10/2016

2 Private and Confidential

• Section 1: Introduction to CPT

• Section 2: CPT solution

• Section 2: Jagger CPT Solution

• Section 4: Customer Success Story

2

Agenda

3 Private and Confidential

Why performance testing?2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load.

2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems.

2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower.

2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.

4 Private and Confidential

Section 1: Introduction to CPT

5 Private and Confidential

Performance only as sign off

Commit

Build

Unit

Smoke

Regression

Integration

Release deployment

Performance tests and sign off

Security sign off

UAT sign off

Functional E2E

Release Candidate

6 Private and Confidential

Performance impact of each change

Add new feature

Functionality check

Performance impact

7 Private and Confidential

Implementation issues

Cost Lack of skillsAvailability of adequate performance testing environments Immature continuous integration and continuous delivery practicesNo focus on performance tests, because other tests are “more important

8 Private and Confidential

Commit

Build

Unit

Smoke

Perf Smoke

Regression

Perf Regressio

n

Integration

Perf E2E

Include performance tests to CI loop

Release deployment

Performance sign off

Security sign off

UAT sign off

Functional E2E

Release Candidate

9 Private and Confidential

Principles#1: Divide and conquer: stage your test execution#2: One throat to choke: target APIs for all testing goals#3: Keep your eyes on the prize: get your risks and KPIs right#4: Automate everything: one-click automation must be end-to-end#5: Analyze this: discover, visualize and deliver performance insights#6: Retain all results: value of data grows over time

10 Private and Confidential

CPT goals

Certify that the application meets performance targets.Plan capacity and manage growth.Track useful performance metrics. Identify load-related weaknesses and track SLA break points.

11 Private and Confidential

Section 2: CPT Solution

12 Private and Confidential

Solution partsCPT PLATFORM - test framework, results storage, metrics collection, and visualization

TEST DATA MANAGEMENT  - data manipulation to establish the CPT process

ENVIRONMENT MANAGEMENT -  controlling the performance test environment life cycle

13 Private and Confidential

CPT platformCreate test loads for our SUT (System Under Test) Collect metrics and store the resultsProvide an analytics and visualization engine Integrate with the CI/CD pipeline

14 Private and Confidential

Test data managementGenerate data based on existing subsetsAlways use fresh test dataCompare test results on test data and productionMask any sensitive test data

15 Private and Confidential

Environment managementCloudProvisioning on demandAutomated configuration Automated deployment

16 Private and Confidential

Section 3: Jagger, Grid Dynamics’ CPT platform

17 Private and Confidential

Bird’s Eye View on Jagger

18 Private and Confidential

Summary view

19 Private and Confidential

Trends between different test sessions

20 Private and Confidential

Detailed report:Parameters during test execution

21 Private and Confidential

Test environment configuration

22 Private and Confidential

Jagger specifications• General

• Performance test in Jagger – Java project• To develop test, some basic knowledge in programming is required • Simple tests can be implemented just by XML editing

• How to start • Download and start test project example:

https://github.com/griddynamics/jagger/wiki/Getting-Started • Download test project with customization examples:

https://github.com/griddynamics/jagger/wiki/Samples

23 Private and Confidential

Section 4: Customer Success Story

24 Private and Confidential

PreconditionsBig eCommerce site – Top 3 on US RetailerLoad during Holiday season – 5.4k TPS for clusterOnly sign-off performance testing processRegular issues on release stage which cause release postponed and expensive fixes

New project for critical service – new search engine and catalog

25 Private and Confidential

What was doneEvaluating specific SLAs for search serviceCreating test data based on user request from production Implementing tests and configure automation execution on build server Introduction Performance testing to CI loop

26 Private and Confidential

ResultsCatch performance issues of search engine on stage of developmentSuccessfully pass performance sign-off as part of full siteNo failures during high season

27 Private and Confidential

Thank you!

Mikhail Klokov: mklokov@griddynamics.comGrid Dynamics blog: blog.griddynamics.comFollow up on twitter: @griddynamicsJagger: jagger.griddynamics.net

We are hiring!griddynamics.com/careers