the 80/20 rule: top purepath problem patterns of 2015! · the 80/20 rule: top purepath problem...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Andreas Grabner, Performance Advocate

The 80/20 Rule: Top PurePathProblem Patterns of 2015!

2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Where do your

Stories come

from?

3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

#1: Real Life & Real User Stories

4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

#2: http://bit.ly/sharepurepath

5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

#3: http://bit.ly/onlineperfclinic

20%

80%

8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE8

Frontend PerformanceWe are getting FATer!

11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Make F12 or Browser Agent your friend!

12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Online Tools for you!

Dynatrace Performance Center

www.dynatrace.com

PageSpeed Insights

developers.google.com/speed

13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE13

Frontend AvailabilityBack to Basics Please!

17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Online Services for you: Is it down right now?

18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Online Services for you: Outage Analyzer

19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE19

Backend PerformanceThe Usual Suspects

• Symptoms

• HTML takes between 60 and 120s to render

• High GC Time

• Developer Assumptions

• Bad GC Tuning

• Probably bad Database Performance as rendering was simple

• Result: 2 Years of Finger pointing between Dev and DBA

Project: Online Room Reservation System

Developers built own monitoring

void roomreservationReport(int officeId)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForOffice(officeId);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, officeId);

}

Result:

Avg. Data Load Time: 45s!

DB Tool says:

Avg. SQL Query: <1ms!

#1: Loading too much data24889! Calls to the

Database API!

High CPU and High Memory Usage to keep all

data in Memory

#2: On individual connections 12444!individual

connections

Classical N+1 Query Problem

Individual SQL really <1ms

#3: Putting all data in temp Hashtable

Lots of time spent in

Hashtable.get

Called from their Entity Objects

• … you know what code is doing

• Challenge the developers

• Explore Tools that “might seem” out of your league!

• Built-In Database Analysis Tools

• “Logging” options of Frameworks such as Hibernate, …

• JMX, Perf Counters, … of your Application Servers

• Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics, …

Lessons Learned – Don’t Assume …

26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE26

Backend PerformancePools and Queues

Online Banking: Slow Balance Check

1.69m (=101s!) To Check Balance!

87% spent in IIS 600! SQL Executions

#1 Time really spent in IIS?

Tip: Elapsed Time tells us WHEN a Method was executed!

Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

Tip: Thread# gives us insight on Thread Queues / Switches

#2 What about these SQL Executions?

Finding: EVERY SQL statement is executed on

ITS OWN Connection!

Tip: Look at “GetConnection”

#2 SQL Executions! continued …

#1: Same SQL is executed 67! times

#2: NO PREPARATIONbecause everything

executed on new Connection

Lessons Learned!

ASP.NET Worker Thread Pool Sizing!

DB Connection PoolsMore Efficient SQL

32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE32

Backend Performance(Micro)Services

33 @Dynatrace

26.7sExecution Time 33! Calls to the

same Web Service

171! SQL Queries through LINQ by this Web Service – request

similar data for each call

Architecture Violation: Direct access to DB instead from frontend logic

34 @Dynatrace

21671! Calls to Oracle

3136! Calls to H2 mostly executed on async background

threads

33! Different connections used

DB Exceptions on both Databases

DB Exceptions on both Databases

40! internal Web Service Calls that

do all these DB Updates

35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE35

Tips & Tricks

36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Web Server Health

Do we have enough WORKER THREADS on the

Web Server?

Sanity Check on BANDWIDTH requirements per Request

Can we handle the LOAD and BANDWIDTH requirements?

37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: App Server Health over TimeHow is the LOAD DISTRIBUTION

on that JVM/CLR?

Any MEMORY ISSUES?

Do we have enough WORKER THREADS?

Do we have enough CPU?

38 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Host Health over Time

Do we have ENOUGH RESOURCES on these hosts?

Which processes are consuming these

resources? Need to RE-DEPLOY processes?

39 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Layer Breakdown over Time

With increasing load: Which LAYER doesn’t SCALE?

40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Exceptions and Log Messages

How are # of EXCEPTIONSevolving over time?

How many SEVERE LOG messages to we write in relation to Exceptions?

41 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Failed Transactions

Are more TRANSACTIONS FAILING (HTTP 5xx, 4xx, …)

under heavier load?

42 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Database Activity

Do we see increased in AVG # of SQL Executions over Time?

Do TOTAL # of SQL Executions increase with load? Shouldn’t

it flatten due to CACHES?

43 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Database History Dashboard

How many SQL Statements are PREPARED?

What’s the overall Execution Time of different SQL Types (SELECT, INSERT, DELETE, …)

44 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: DB Connection Pool UtilizationDo we have enough DB

CONNECTIONS per pool?

45 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Group-By + Show All Options

46 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Tip: Advanced Filtering

47 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Memory, Thread, CPU Diagnostics

48 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Server Timeline Dashlet (Community Supported)

49 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

• http://bit.ly/onlineperfclinic

• http://bit.ly/dttutorials

• http://bit.ly/sharepurepath

• http://answers.dynatrace.com

• http://apmu.dynatrace.com

• Follow me: @grabnerandi

More Resources

50 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE