performance challenges along the continuous delivery pipeline€¦ · performance challenges along...
TRANSCRIPT
1 #Dynatrace
Wolfgang Gottesheim @gottesheim
Performance Challenges along the Continuous Delivery Pipeline
2 #Dynatrace
Example of a “Bad” Web Deployment 282! Objects
on that page 9.68MB Page Size
8.8s Page
Load Time Most objects are images
delivered from your main
domain
Very long Connect time
(1.8s) to your CDN
3 #Dynatrace
Application problems in the wild 526s to render a financial transaction report
1 SQL running
210s!
Debug Logging with
log4j on outdated log4j
library (sync issue)
4 #Dynatrace
But now we‘re doing
Continuous Delivery
5 #Dynatrace
700 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
6 #Dynatrace
It‘s not about the pipeline… (only)
7 #Dynatrace
Commit Stage
Acceptance Testing
Release
Developers
Capacity Testing
UAT
8 #Dynatrace
Metrics
to the rescue!
9 #Dynatrace
We want to replace magic with data!
Commit Stage
Acceptance Testing
Release Capacity Testing
UAT
10 #Dynatrace
4 examples
11 #Dynatrace
#1
12 #Dynatrace
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
13 #Dynatrace
8MB background image for STPCon
14 #Dynatrace
Key Metrics
# Resources
Size of Resources
Page Size
15 #Dynatrace
#2
16 #Dynatrace
#1: Loading too much data
24889! Calls to the
Database API!
High CPU and High Memory
Usage to keep all data in Memory
17 #Dynatrace
#2: On individual connections
Classical N+1 Query Problem
Individual SQL really <1ms
12444! individual connections (and no pooling…)
18 #Dynatrace
#3: Homegrown Caching Framework
Lots of time spent in Hashtable.get
Called from their Entity Objects
19 #Dynatrace
Key Metrics
# Functional Errors # of SQL Calls # of same SQL Execs (1+N) # of Connections Rows/Data Transferred
20 #Dynatrace
#3
Simply splitting web service calls is not a migration to a microservice architecture
21 #Dynatrace
3136! Calls to H2 mostly executed on async background
threads
40! internal Web Service Calls that do all these DB Updates
21671! Calls to Oracle
DB Exceptions on both Databases
DB Exceptions on both Databases
33! Different connections used
22 #Dynatrace
Key Metrics
# Service Calls # of Threads Timing (execution, wait) # SQL executions # of SAME SQLs
23 #Dynatrace
#4
24 #Dynatrace
Is this a successful new Build?
25 #Dynatrace
Look at Resource Usage: CPU, Memory, …
26 #Dynatrace
Memory? Look at Heap Generations
27 #Dynatrace
Root Cause: Dependency Injection
28 #Dynatrace
Prevent: Monitor Memory Metrics for every Build
29 #Dynatrace
Key Metrics
# of Objects per Generation
# of GC Runs
Total Impact of GC
30 #Dynatrace
1. # Resources
2. Size of Resources
3. Page Size
4. # Functional Errors
5. # SQL Executions
6. # of SAME SQLs
Where to start?
31 #Dynatrace
P A M
I
ick gree easure ntegrate
Allow me to introduce…
32 #Dynatrace AND MANY MORE Kieker
33 #Dynatrace
We successfully replaced magic with data!
Commit Stage
Acceptance Testing
Release Capacity Testing
UAT
34 #Dynatrace
Just one more thing…
35 #Dynatrace
It‘s also about the pipeline…
Define pipeline metrics!
36 #Dynatrace
Connect your Tests with Quality
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Exceptions probably reason for
failed tests Problem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regression Now we have the functional and
architectural confidence
Let’s look behind the
scenes
37 #Dynatrace
#1: Analyzing each Test
#2: Metrics for each Test
#3: Detecting Regression based on Measure
38 #Dynatrace
Quality-Metrics based
Build Status
39 #Dynatrace
Pull data into Jenkins, Bamboo ...
40 #Dynatrace
Make Quality a first-class citizen
„Too hard“
„we‘ll get round to this later“
„not cool enough“
41 @Dynatrace
Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial
Twitter @gottesheim