the 80/20 rule: top purepath problem patterns of 2015! · the 80/20 rule: top purepath problem...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Andreas Grabner, Performance Advocate

The 80/20 Rule: Top PurePathProblem Patterns of 2015!


Where do your

Stories come

from?


#1: Real Life & Real User Stories


#2: http://bit.ly/sharepurepath


#3: http://bit.ly/onlineperfclinic

20%

80%

8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE8

Frontend PerformanceWe are getting FATer!


Make F12 or Browser Agent your friend!


Online Tools for you!

Dynatrace Performance Center

www.dynatrace.com

PageSpeed Insights

developers.google.com/speed

http://www.dynatrace.com/


Frontend AvailabilityBack to Basics Please!


Online Services for you: Is it down right now?


Online Services for you: Outage Analyzer


Backend PerformanceThe Usual Suspects

• Symptoms

• HTML takes between 60 and 120s to render

• High GC Time

• Developer Assumptions

• Bad GC Tuning

• Probably bad Database Performance as rendering was simple

• Result: 2 Years of Finger pointing between Dev and DBA

Project: Online Room Reservation System

Developers built own monitoring

void roomreservationReport(int officeId)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForOffice(officeId);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, officeId);

}

Result:

Avg. Data Load Time: 45s!

DB Tool says:

Avg. SQL Query: <1ms!

#1: Loading too much data24889! Calls to the

Database API!

High CPU and High Memory Usage to keep all

data in Memory

#2: On individual connections 12444!individual

connections

Classical N+1 Query Problem

Individual SQL really <1ms

#3: Putting all data in temp Hashtable

Lots of time spent in

Hashtable.get

Called from their Entity Objects

• … you know what code is doing

• Challenge the developers

• Explore Tools that “might seem” out of your league!

• Built-In Database Analysis Tools

• “Logging” options of Frameworks such as Hibernate, …

• JMX, Perf Counters, … of your Application Servers

• Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics, …

Lessons Learned – Don’t Assume …


Backend PerformancePools and Queues

Online Banking: Slow Balance Check

1.69m (=101s!) To Check Balance!

87% spent in IIS 600! SQL Executions

#1 Time really spent in IIS?

Tip: Elapsed Time tells us WHEN a Method was executed!

Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

Tip: Thread# gives us insight on Thread Queues / Switches

#2 What about these SQL Executions?

Finding: EVERY SQL statement is executed on

ITS OWN Connection!

Tip: Look at “GetConnection”

#2 SQL Executions! continued …

#1: Same SQL is executed 67! times

#2: NO PREPARATIONbecause everything

executed on new Connection

Lessons Learned!

ASP.NET Worker Thread Pool Sizing!

DB Connection PoolsMore Efficient SQL


Backend Performance(Micro)Services

33 @Dynatrace

26.7sExecution Time 33! Calls to the

same Web Service

171! SQL Queries through LINQ by this Web Service – request

similar data for each call

Architecture Violation: Direct access to DB instead from frontend logic

34 @Dynatrace

21671! Calls to Oracle

3136! Calls to H2 mostly executed on async background

threads

33! Different connections used

DB Exceptions on both Databases

DB Exceptions on both Databases

40! internal Web Service Calls that

do all these DB Updates


Tips & Tricks


Tip: Web Server Health

Do we have enough WORKER THREADS on the

Web Server?

Sanity Check on BANDWIDTH requirements per Request

Can we handle the LOAD and BANDWIDTH requirements?


Tip: App Server Health over TimeHow is the LOAD DISTRIBUTION

on that JVM/CLR?

Any MEMORY ISSUES?

Do we have enough WORKER THREADS?

Do we have enough CPU?


Tip: Host Health over Time

Do we have ENOUGH RESOURCES on these hosts?

Which processes are consuming these

resources? Need to RE-DEPLOY processes?


Tip: Layer Breakdown over Time

With increasing load: Which LAYER doesn’t SCALE?


Tip: Exceptions and Log Messages

How are # of EXCEPTIONSevolving over time?

How many SEVERE LOG messages to we write in relation to Exceptions?


Tip: Failed Transactions

Are more TRANSACTIONS FAILING (HTTP 5xx, 4xx, …)

under heavier load?


Tip: Database Activity

Do we see increased in AVG # of SQL Executions over Time?

Do TOTAL # of SQL Executions increase with load? Shouldn’t

it flatten due to CACHES?


Tip: Database History Dashboard

How many SQL Statements are PREPARED?

What’s the overall Execution Time of different SQL Types (SELECT, INSERT, DELETE, …)


Tip: DB Connection Pool UtilizationDo we have enough DB

CONNECTIONS per pool?


Tip: Group-By + Show All Options


Tip: Advanced Filtering


Memory, Thread, CPU Diagnostics


Server Timeline Dashlet (Community Supported)


• http://bit.ly/onlineperfclinic

• http://bit.ly/dttutorials

• http://bit.ly/sharepurepath

• http://answers.dynatrace.com

• http://apmu.dynatrace.com

• Follow me: @grabnerandi

More Resources

http://bit.ly/onlineperfclinic

http://bit.ly/dttutorials

http://bit.ly/sharepurepath

http://apmu.dynatrace.com/

http://apmu.dynatrace.com/

50 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

the 80/20 rule: top purepath problem patterns of 2015! · the 80/20 rule: top purepath problem...

Documents