continuous performance testing
TRANSCRIPT
Continuous Performance Testing
Mairbek Khadikov
Story #1
Performance, we are
losing it
Story #1
DeliveryManager
New version goes live soon
Let's check performance
Story #1
Performance Tester
Performance degraded
=
Story #1
Developer
Let's profile our app
Story #1
Performance Tester
Performance increased on 10%
But it's still worse than the previous release
Story #1
DeliveryManager
We can't wait much longer
CPU utilization is 50%
Story #1
Alert is rased
too late
Story #2
It's all tested, but...
Story #2
Operations
We deployed new release
It passed all smoke tests
Works incorrect with load balancer
Story #2
QA
We tested all this features
Story #2
PerformanceTester
System has acceptable throughput
Story #2
Developer
Doh, one singleton unit is not thread safe
Story #2
Performance test
does not check that system works
correctly
Story #3
Forgotten branches
Story #3
PerformanceTester
Performance degraded in the last release candidate
Story #3
Developer
What branch exactly impacted performance?
Story #3
PerformanceTester
I've never heard about the half of this branches
Story #3
Large projects need a comprehensive
performance testing process
Story #4
Did you test that?
Story #4
VP of Engineering
Production system failed this morning
Story #4
Developer
Two nodes were restarted
simultaneously
Story #4
Operations
Those nodes consumed to much memory
Story #4
QA Lead
This is a complex scenario we can't do it continuously
Story #4
Simulation of
maintenance operations and failures was not a
part of testing
Performance Testing
Revised
Performance Testing
Automatically test builds
Continuous
Performance Testing
Rise alert flag in case of performance degradation
Continuous
Performance Testing
Store all testing results
Traceable
Performance Testing
Guaranty that system works correctly under workload
Validated
Performance Testing
Result should include monitoring and profiling information
Transparent
Performance Testing
Modules should be tested in isolation where possible
Transparent
Performance Testing
Simulate potential maintenance
operations and failure
Pessimistic
