continuous integration - a performance engineer's tale
DESCRIPTION
Andrew Harding from Betfair's presentation on web performance testing in a continuous integration environment. Covers some good reasons why and why not to do perf testing during continuous integration.TRANSCRIPT
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
London Web Meetup – Dec 2011Continuous Integration – A Performance Engineers Tale
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
2
“WE WILL DO CONTINUOUS INTEGRATION TESTING”1128th commandment - The Management
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
INITIAL THOUGHTS
•Automate all of our Performance Tests
•Add tests to the Build process
3
CI
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
WHY IT WONT WORK
•Performance testing needs a functional application
•The length of time needed to gather significant amount of data points
•Performance testing requires analysis of results to find meaningful conclusions
4
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
MORE INFORMATION
5
we'll receive immediate
feedback
continuous integration (CI) implements continuous processes of applying quality
control
For a build to be self-testing the failure of a
test should cause the build to fail.
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
GOALS FOR PERFORMANCE
•Measure Performance as part of every build
•Compare between builds (regression)•Alert if performance deviates from normal
•Fail the build if performance does not meet SLA’s
•Automatic process
6
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
7
Build
Delivery Team
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
8
Build
Environment
Delivery Team
Deploy
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
9
Build
Environment
Performance Measurement
s
Perf Trends
Delivery Team
Deploy
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
10
Build
Environment
Performance Measurement
s
Perf Trends
Delivery Team
Alert
Deploy
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
11
Build
Environment
ContinuousLoad
Injection
Production logs
Performance Measurement
s
Perf Trends
Delivery Team
Usage Profile
Alert
Deploy
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
12
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Delivery Team
Usage Profile
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
13
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
14
“THE POINT OF TESTING IS TO FLUSH OUT, UNDER CONTROLLED CONDITIONS, ANY PROBLEM THAT THE SYSTEM WILL HAVE IN PRODUCTION. A SIGNIFICANT PART OF THIS IS THE ENVIRONMENT WITHIN WHICH THE PRODUCTION SYSTEM WILL RUN. IF YOU TEST IN A DIFFERENT ENVIRONMENT, EVERY DIFFERENCE RESULTS IN A RISK THAT WHAT HAPPENS UNDER TEST WON’T HAPPEN IN PRODUCTION”
MARTIN FOWLER
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
15
Build
Production Environment
End User Response
Perf Trends
Perf Details
Delivery Team
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
16
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
17
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
LoadRunner
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
18
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
Keynote
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
19
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
Dynatrace Ajax Premium
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
ADVANTAGES
•Allows standard performance regression testing to be fully automated, freeing up time for more complex performance work
•Separating load injection from performance measurements enables performance to be included in build cycle
•Provides immediate feedback to delivery teams20
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
NOT IN SCOPE
•Capacity Testing•Soak Testing•Failure Testing•Stress Testing
21
Still Plenty of Traditional work to do
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THINGS I LEARNT
•Keep an open mind to new stuff
22
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THINGS I LEARNT
•Keep an open mind to new stuff•Give it more than 2 minutes of thought
23
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THINGS I LEARNT
•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work
24
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THINGS I LEARNT
•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work•Use Google
25
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THINGS I LEARNT
•Keep an open mind to new stuff•Give it more than 2 minutes of thought•Get past the reasons it won’t work•Use Google•Design for the ideal, compromise later
26
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
THANK YOU