1

Checking Performance Along your Build Pipeline

@grabnerandihttp://apmblog.compuware.com

http://bit.ly/dttrial

2

Today we are here for …

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Cont. Delivery / Deploy

3

Who is doing it? How many successful deployments can they do?

300 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

4

More on Amazons Story

75% fewer outages since 2006

90% fewer outage minutes

~0.001% of deployments cause a problem

Instantaneous automatic rollback

Deploying every 11.6s

5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Recommended BookRecommended Book

https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/

66 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Status Quo: Unreliable Software Impacts BusinessStatus Quo: Unreliable Software Impacts Business

7

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

8

The “War Room” – NOW

Facebook – December 2012

The “War Room” – NOW

Facebook – December 2012

9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Status Quo: Bugfixing in Production happensStatus Quo: Bugfixing in Production happens

~80% of problems

caused by ~20% patterns

YES we know this

80% Dev Time in Bug Fixing

$60B Defect Costs

BUT

10

4 Situations on

WHY this happens,

HOW to avoid it

4 Situations on

WHY this happens,

HOW to avoid it

11

12

“Blindly” (Re)use Existing

Components

13

Requirement: We need a report

14

Using Hibernate results in 4k+ SQL Statements to display 3 items!

Hibernate Executes 4k+ Statements

Individual Execution VERY

FAST

But Total SUM takes 6s

15

Requirement: We need a fancy UI

16

Using Telerik Controls Results in 9s for Data-Binding of UI Controls

#1: Slow Stored ProcedureDepending on Request

execution time of this SP varies between 1 and 7.5s

#2: 240! Similar SQL StatementsMost of these 240! Statements are

not prepared and just differ in things like Column Names

17

Metrics: # Total SQLs# SQLs / Web Request# Same SQLs / Request

Transferred Rows

Test: With realistic Data

Dev: “Learn” Frameworks

18

19

Implementation Flaws

20

Business Impact requires Action!

21

Solution: Cache to the RESCUE!!

22

Implementation and Rollout

Implemented InMemory Cache

Worked well in Load Testing

23

Result: Out of Memory Crashes!!

Still crashes

Problem fixed!Fixed Version Deployed

24

Metrics: Heap Size, # Objects Allocated,# Objects in Cache

Cache Hit Ratio

Test: With realistic Data

25

12 000 000 $

26

#No “Agile” Deployment

27

Ad on air

Availability dropped to 0%

Load Spike resulted in Unavailability

28

Alternative: “GoDaddy goes DevOps”

Response time improved 4x

1h before SuperBowl KickOff

1h after Game ended

29

Behind the Scenes

30

Metrics: AvailabilityPage Size, # Objects

# Hosts, # Connections

DevOps: “Feature” Switches

31

32

#Push without a Plan

33

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

34

m.store.com redirects to www.store.com

ALL CSS and JS files are redirected to the www domain

This is a lot of time “wasted” especially on high latency mobile

connections

35

Metrics: Load Time, # Resources (Images, …),

# HTTP 3xx, 4xx, 5xx

Dev: Build for Mobile

Test: Test on Mobile

Ops: Monitor Mobile

36

37

#1: Define Measures# of Requests / User

# of Log Messages

# of Exceptions

# Objects Allocated

# Objects In Cache

Cache Hit Ratio

# of Images

# of SQLs

# SQLs per RequestAvailability

# HTTP 3xx, 4xx

Page Size

38

#2: Automate

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Unit & Integration Tests

Functional Tests

Performance TestsProductionMonitoring

Functional Tests

How? Performance Focus in Test AutomationAnalyzing All Unit / Performance Tests

Analyze Perf Metrics

Identify Regressions

How? Performance Focus in Test Automation

How? Performance Focus in Test Automation

Cross Impact of KPIs

Identify Regressions

4242

#3: Share Results

43

#4: Integrate

44

If we do all that

CANCELLED

45

Want MORE of these and more details?

http://apmblog.compuware.com

46

FREE Products & More Info

dynaTrace Free Trial30 Days Free Trial on http://bit.ly/dttrial After that totally free for local usage

Our Blog: http://apmblog.compuware.com

Follow Me: @grabnerandi