performance testing automation for ci/cd...
TRANSCRIPT
![Page 1: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/1.jpg)
PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.
Henrik Rexed
Partner Solution Evangelist
Neotys
![Page 2: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/2.jpg)
▪ Henrik Rexed (khnightloader)*
▪ 14 years of Performance engineering
▪ Technical Evangelist @Neotys
▪ Lego Fan
▪ Focus on technologies for Performance engineers
Who is this speaker?his speaker?
![Page 3: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/3.jpg)
Agenda 01
02
03
04
The current challenge with Performance Testing
Make sure your tests are relevant
Let’s test earlier
What are the SLI I should focus on?
Lorem ipsum dolor sit amet
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.05 Demo
![Page 4: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/4.jpg)
![Page 5: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/5.jpg)
Requirements Design Implementation Testing Acceptance Deployment
Inception Elaboration Construction Transition
Development Operations
DevOps
Big BangWaterfall
IncrementalRational Unified
Process
AgileScrum, Kanban
ContinuousDevOps
Digital Singularity
Agile Event Horizon
6+
4
2
1
Change of MethodologyChange of Methodology
![Page 6: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/6.jpg)
Performance testing is like going to the Gym
▪ Over 1 in 5 Americans belong to at least one U.S. health club or studio
▪ 12% gym members sign up in January
▪ Most of the people quit or stop going after 24 weeks
▪ 50% is going to find the soulmate
![Page 7: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/7.jpg)
▪ From Monolitic
Presentation Layer
Business Layer
Data Layer
One application per hostHaving the entire business logic
Ok but let’s review our architecture first
![Page 8: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/8.jpg)
Ok but let’s review our architecture first
▪ To New Microservices Front
Service A Service B Service C Sevice D Service E
Service F Service G
Every service is managed by a dedicated team
![Page 9: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/9.jpg)
Let’s move to the cloud
▪ Reduce the cost
▪ Increase the availability of the application
▪ Simplify, modernize
![Page 10: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/10.jpg)
How Can We Resolve All Those Challenges?How can we resolve all those challenges?
![Page 11: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/11.jpg)
Agenda 01
02
03
04
The current challenge with Performance Testing
Make sure your tests are relevant
Let’s test earlier
What are the SLI I should focus on?
Lorem ipsum dolor sit amet
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.05 Q/A
![Page 12: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/12.jpg)
Performance testing used to be ….Performance testing is like
![Page 13: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/13.jpg)
Efficient and realistic workload model
Service calls / User action Request Mix
Geographic sessions
Device type & Mix
Transaction volume (TPS)
0
20
40
60
80
100
120
140
160
180
08am 09am 10am 11am 12pm 1pm 2pm 3pm 4pm 5pm 6pm
Load Distribution
VA01 VA02 POP1 ME21N
![Page 14: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/14.jpg)
Let’s reuse monitoring data
Traditionalmonitoring
Tracking system APM
![Page 15: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/15.jpg)
Let’s extract data from our APM to learn how our application is used
Advantage
• URL
• API
• Request /min
• Distribution during time
• User sessions
• User actions
• ….Etc
Disadvantage
• Aggregration
• You need APM solution
![Page 16: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/16.jpg)
Agenda 01
02
03
04
The current challenge with Performance Testing
Make sure your tests are relevant
Let’s test earlier
What are the SLI I should focus on?
Lorem ipsum dolor sit amet
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.05 Q/A
![Page 17: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/17.jpg)
Architecture is Based on ComponentsArchitecture is Based on Components
![Page 18: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/18.jpg)
Continuous Performance Testing API Testing Integration Testing Application Testing
Test a Component Test a System Test Real World
Continuous Testing Embedded in CI/CD Pipelines
![Page 19: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/19.jpg)
The complexity of the test depends of project life cycle
Your Footer Here• Component testing
Dev environment
• Soak Test
• Limit
QA Environment
• Load testing using devices and browser to validate the UX
• Fail Over testing
• Cloud Testing
Pre-Production Environment
• RUM
• Synthetic
• Load in production
Production
![Page 20: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/20.jpg)
Agenda 01
02
03
04
The current challenge with Performance Testing
Make sure your tests are relevant
Let’s test earlier
What are the SLI I should focus on?
Lorem ipsum dolor sit amet
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.05 Demo
![Page 21: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/21.jpg)
Process to analyse
Error rate
Throughput + Response Time versus objectives
Resources check
Test Failure
![Page 22: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/22.jpg)
Quality Gate
File defining the indicatorsand the objectives
Quality Gate
Collecting the specification
Retrieving the dataFrom the datasource
Calculate the score
Datasource
![Page 23: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/23.jpg)
Not limit the evaluation only on response timeProduction
• Rendering time
• Number of clicks to achiev an actions
• Average visit time
• IO
• Number of Pods utilizd
User Experience
• On a representative environement:
• Response Time Rendering time
• Number of container Pods/utilized
• The behavior of the network
• ..etc
Regression
Detecting regression by looking at:
• Response time
• Hit/s
• Memory used
• Cpu
Scalability
If it’s a cloud native architecture :
• Number of containers/Pods utilized
• The memory used
• CPU
• Response time
01
02
03
04
![Page 24: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/24.jpg)
Agenda 01
02
03
04
The current challenge with Performance Testing
Make sure your tests are relevant
Let’s test earlier
What are the SLI I should focus on?
Lorem ipsum dolor sit amet
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.05 Demo
![Page 25: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/25.jpg)
NeoLoad’s platform
On-Prem Docker Stack
RESTful APIs
NeoLoadCore
NeoLoadWeb
NeoLoadSaaS
Managed / Hosted
Load Generators
![Page 26: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/26.jpg)
What we prepared for you todayKubernetes cluster
Docker Registry with pre-built app images
GitHub Org
Staging Production
Test Orchestration & Test History
Keptn Repo for project files
Load generators
SaaS ClusterOneAgent
Operator
https://github.com/keptn-orders/
![Page 27: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/27.jpg)
Demo time
![Page 28: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/28.jpg)
Take aways
• Build realistic test based on real production situation.
• Test earlier by runningcontinuous testing
• Automate the analysisBy calculating a score of your release
• Select indicators that will validate :• Response time• The health of the infrastructure• The cost of the environment
![Page 29: PERFORMANCE TESTING AUTOMATION FOR CI/CD PIPELINES.ltb2020.eecs.yorku.ca/slides/HenrikRexed-LTB2020.pdf · Efficient and realistic workload model Service calls / User action Request](https://reader034.vdocument.in/reader034/viewer/2022043013/5fad9f69ef593f3fe654cb7e/html5/thumbnails/29.jpg)
QA