performance challenges along the continuous delivery pipeline€¦ · performance challenges along...

1 #Dynatrace

Wolfgang Gottesheim @gottesheim

Performance Challenges along the Continuous Delivery Pipeline

2 #Dynatrace

Example of a “Bad” Web Deployment 282! Objects

on that page 9.68MB Page Size

8.8s Page

Load Time Most objects are images

delivered from your main

domain

Very long Connect time

(1.8s) to your CDN

3 #Dynatrace

Application problems in the wild 526s to render a financial transaction report

1 SQL running

210s!

Debug Logging with

log4j on outdated log4j

library (sync issue)

4 #Dynatrace

But now we‘re doing

Continuous Delivery

5 #Dynatrace

700 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

6 #Dynatrace

It‘s not about the pipeline… (only)

7 #Dynatrace

Commit Stage

Acceptance Testing

Release

Developers

Capacity Testing

UAT

8 #Dynatrace

Metrics

to the rescue!

9 #Dynatrace

We want to replace magic with data!

Commit Stage

Acceptance Testing

Release Capacity Testing

UAT

10 #Dynatrace

4 examples

11 #Dynatrace

#1

12 #Dynatrace

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:

230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

13 #Dynatrace

8MB background image for STPCon

14 #Dynatrace

Key Metrics

# Resources

Size of Resources

Page Size

15 #Dynatrace

#2

16 #Dynatrace

#1: Loading too much data

24889! Calls to the

Database API!

High CPU and High Memory

Usage to keep all data in Memory

17 #Dynatrace

#2: On individual connections

Classical N+1 Query Problem

Individual SQL really <1ms

12444! individual connections (and no pooling…)

18 #Dynatrace

#3: Homegrown Caching Framework

Lots of time spent in Hashtable.get

Called from their Entity Objects

19 #Dynatrace

Key Metrics

# Functional Errors # of SQL Calls # of same SQL Execs (1+N) # of Connections Rows/Data Transferred

20 #Dynatrace

#3

Simply splitting web service calls is not a migration to a microservice architecture

21 #Dynatrace

3136! Calls to H2 mostly executed on async background

threads

40! internal Web Service Calls that do all these DB Updates

21671! Calls to Oracle

DB Exceptions on both Databases

DB Exceptions on both Databases

33! Different connections used

22 #Dynatrace

Key Metrics

# Service Calls # of Threads Timing (execution, wait) # SQL executions # of SAME SQLs

23 #Dynatrace

#4

24 #Dynatrace

Is this a successful new Build?

25 #Dynatrace

Look at Resource Usage: CPU, Memory, …

26 #Dynatrace

Memory? Look at Heap Generations

27 #Dynatrace

Root Cause: Dependency Injection

28 #Dynatrace

Prevent: Monitor Memory Metrics for every Build

29 #Dynatrace

Key Metrics

# of Objects per Generation

# of GC Runs

Total Impact of GC

30 #Dynatrace

1. # Resources

2. Size of Resources

3. Page Size

4. # Functional Errors

5. # SQL Executions

6. # of SAME SQLs

Where to start?

31 #Dynatrace

P A M

I

ick gree easure ntegrate

Allow me to introduce…

32 #Dynatrace AND MANY MORE Kieker

33 #Dynatrace

We successfully replaced magic with data!

Commit Stage

Acceptance Testing

Release Capacity Testing

UAT

34 #Dynatrace

Just one more thing…

35 #Dynatrace

It‘s also about the pipeline…

Define pipeline metrics!

36 #Dynatrace

Connect your Tests with Quality

12 0 120ms

3 1 68ms

Build 20 testPurchase OK

testSearch OK

Build 17 testPurchase OK

testSearch OK

Build 18 testPurchase FAILED

testSearch OK

Build 19 testPurchase OK

testSearch OK

Build # Test Case Status # SQL # Excep CPU

12 0 120ms

3 1 68ms

12 5 60ms

3 1 68ms

75 0 230ms

3 1 68ms

Test Framework Results Architectural Data

We identified a regresesion

Problem solved

Exceptions probably reason for

failed tests Problem fixed but now we have an

architectural regression

Problem fixed but now we have an

architectural regression Now we have the functional and

architectural confidence

Let’s look behind the

scenes

37 #Dynatrace

#1: Analyzing each Test

#2: Metrics for each Test

#3: Detecting Regression based on Measure

38 #Dynatrace

Quality-Metrics based

Build Status

39 #Dynatrace

Pull data into Jenkins, Bamboo ...

40 #Dynatrace

Make Quality a first-class citizen

„Too hard“

„we‘ll get round to this later“

„not cool enough“

41 @Dynatrace

Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial

Twitter @gottesheim

wolfgang.gottesheim@dynatrace.com