from zero to performance hero in minutes - agile testing days 2014 potsdam

65
Performance Clinic Workshop From Zero to Performance Hero Andreas Grabner (@grabnerandi) [email protected] http://bit.ly/atd2014challenge

Upload: andreas-grabner

Post on 02-Jul-2015

1.148 views

Category:

Software


3 download

DESCRIPTION

As a Tester you need to level up. You can do more than functional verification or reporting Response Time In my Performance Clinic Workshops I show you real life exampls on why Applications fail and what you can do to find these problems when you are testing these applications. I am using Free Tools for all of these excercises - especially Dynatrace which gives full End-to-End Visibility (Browser to Database). You can test and download Dynatrace for Free @ http://bit.ly/atd2014challenge

TRANSCRIPT

Page 1: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Performance Clinic Workshop

From Zero to Performance HeroAndreas Grabner (@grabnerandi)

[email protected]://bit.ly/atd2014challenge

Page 2: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Why Applications Fail and/or are Slow!

Page 3: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Performance areas we cover today

• Frontend

• Backend

• Deployment

Page 4: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Frontend

Page 5: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Page 6: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/

Page 7: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

#Overloaded Web Pages• #1: Too many resources loaded on web page

– Images: can be “sprited”– CSS and JS: can be merged

• #2: Large Content– Images: do you need high-res? Compress!– CSS and JS: Minify, remove comments, …

• #3: 3rd Party Content– Slow or too much

• #4: AJAX to the MAX!– Too many AJAX Calls requesting too much data

Page 8: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Examples on Overloaded

Page 9: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

This is a heavy page

Page 10: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

This is a light page

http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/

Page 11: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

softdrink.com during SuperBowl

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/

Page 12: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Fifa.com during Worldcup

http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

Page 13: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Kia.com during SuperBowl

Ad

on

air

Page 14: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

GoDaddy.com during SuperBowl

1h before SuperBowl KickOff

1h after Game ended

Page 15: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Kia vs GoDaddy: The Facts!

http://apmblog.compuware.com/2014/02/19/dns-tcp-and-size-application-performance-best-practices-of-super-bowl-advertisers/

Page 16: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

3rd Party Content

Page 17: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Do you really need all bells and whistles?

# of Domains # of Resources Total Bytes DNS [ms] Connect [ms]

With Third Party Content

26 176 2856 Kb 1286,82 1176,09

Without Third Party Content

2 59 897 Kb 0,91 22,25

Page 18: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Too heavy AJAX/JavaScript

1.1s on my IE 10 to

execute magicSpanLinks()

The each loop calls this block of JavaScript for every span node

759 span nodes are processed

by the anonymous function

It adds the dynamically generated link and removes the

old spanWe can see all the DOM

Modifications and how this sums up in execution time

Page 19: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

3rd Party JavaScript Problems

Slow 3rd Party

Libraries

Slow jQuery

Lookukps

Page 20: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

15 Minute Sanity Check – Live Demo

• Dynatrace Performance Test Center– http://www.dynatrace.com/en_us/application-performance-

management/products/performance-center.html

• Dynatrace Browser Agent (formerly AJAX Edition)– http://apmblog.compuware.com/2014/07/01/can-monitor-web-

performance-free/

• Fiddler – simulate other browsers– http://www.telerik.com/fiddler

• PerfMap – HeatMap for a Website– https://github.com/zeman/perfmap

Page 21: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Summary – WPO Best Practices• Additional Blog Posts

– http://apmblog.compuware.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/

– http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/

– http://apmblog.compuware.com/2010/08/25/top-10-client-side-performance-problems-in-web-2-0/

• Recommended Books from Steve Souders covering things like– Make fewer HTTP Requests

– Proper Cache Settings

– Optimize/Compress Content

– Use CDNs

– Watch out for 3rd Parties

Page 22: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Tooling Support• http://bit.ly/dttrial• http://ajax.dynatrace.com• http://yslow.org• https://developers.google.com/speed/pagespeed• http://www.webpagetest.org/• http://www.sitespeed.io/• http://www.showslow.org• http://www.telerik.com/fiddler• https://github.com/zeman/perfmap• https://chrome.google.com/webstore/detail/user-agent-switcher-for-

c/djflhoibgkdhkhhcedjiklpkjnoahfmg

Page 23: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Hands-On

Page 24: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Backend Performance

http://bit.ly/atd2014challenge@grabnerandi

Page 25: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Page 26: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Page 27: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Bad Architectural Decisions#1: Database Access

N+1 Query Problem

Loading Too Much Data

Connection Leaks

#2: External Frameworks

Bad Configuration of O/R Mappers, e.g: Hibernate

Worked well in Sample App Attitude

#3: Excessive Logging & Exceptions

Debug Logging turned on

Using outdated logging libraries

Exception Overload

#4: Memory Leaks

Keeping objects for too long

Bad Cache Implementations

#5: Bad Coding

Synchronization to Death

High on CPU

Page 28: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Examples

Page 29: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Database Access

Page 31: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Querying too much DataAll of these requests with large size have the same problem in common:

DB Access

24889 Calls to the Database!

Tomcat needs to process all this data!SideEffect: High Memory Usage to process

data -> results in high GC

High GC is not the problem. It is just the symptom of too much data

loaded!

Page 32: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Too many connections 12444 individual connections to execute 12444 individual SQL

Statements

Classical N+1 Query Problem. The same SQL is executed many times

with different WHERE ClauseOptimize this by only calling it once

with a better WHERE Clause

Individual executions are fast. But VOLUME is

killing you

Page 33: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Filtering Data in Memory instead of DB

Most lookups are done by getRoomCapacity

Assumption: All the data loaded ends up in Hashtable. The lookups are then very expensive because these Tables

are so huge

Page 34: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

External Frameworks

Page 35: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Non Optimized Hibernate

http://apmblog.compuware.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/

Page 36: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Non Optimized Telerik Controls

http://apmblog.compuware.com/2014/04/03/database-access-patterns-gone-wild-inside-telerik-sharepoint-and-asp-net/

Page 37: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Logging

Page 38: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Too much Logging#1: Top Problem: log4j.callAppenders

#2: Most of logging done from fillDetail method

#3: Doing “DEBUG” log output: Is this necessary?

Page 39: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Exceptions vs. Log Messages

http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/

Page 40: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Exception Logging Overload

Tomcat logAbandoned=true flag causes many exceptions objects to be created

http://apmblog.compuware.com/2012/08/01/top-performance-mistakes-when-moving-from-test-to-production-excessive-logging/

Page 41: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Exception Performance Overhead

http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/

Page 42: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Memory

Page 43: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Oracle JDBC Driver BugEach of the 10 JVMs per Host consumes up to 4.1GB until they crash (41GB per Host)

It is a reoccurring pattern over months

http://apmblog.compuware.com/2014/01/07/hunting-a-oracle-jdbc-memory-leak-crashing-an-80jvm-websphere-cluster/

Page 44: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

No Object Cleanup Code!

http://apmblog.compuware.com/2014/02/26/memory-leaks-load-balancing-and-deployment-settings-testing-lessons-learned-from-the-ecommerce-industry/

Page 45: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Bad Coding

Page 46: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Slow Custom RegEx

http://apmblog.compuware.com/2014/10/16/15-minutes-spent-optimizing-performance-save-millions-lost-revenue/

Page 47: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Slow Content Rendering

Rendering Methods take very long!

Intializing GlyphLayout takes very long

These slow rendering methods are called very frequently!

Page 48: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Synchronization

http://apmblog.compuware.com/2013/09/24/100-performance-overhead-by-websphere-activity-log-when-dev-is-not-aware-of-settings-in-production/

Page 49: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

15 Minute Sanity Check – Live Demo

• Dynatrace

Page 50: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Summary – Backend Best Practices• Additional Blog Posts

– http://apmblog.compuware.com/2013/04/10/top-8-application-performance-landmines/

– http://apmblog.compuware.com/2010/06/15/top-10-performance-problems-taken-from-zappos-monster-and-co/

• Online Java Enterprise Performance Book: http://javabook.compuware.com

• Key Takeaways– Educate Developers

– Understand Frameworks you are using

Page 51: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Hands-On

Page 52: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Deployment

Page 53: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam
Page 54: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Common Deployment Mistakes• Missing Resource Files

– Many HTTP 4xx– Many HTTP 3xx Redirects -> overhead!

• Missing Configuration Files– Web Server Access Rules -> Leads to HTTP 4xx– Web Server -> App Server: Connection & Thread Pools

• Bad Modules or Configuration Problems– Leading to bad requests and overhead– Rewrite and Redirect Modules: Long chains of redirects

• 3rd Party: – CDN Configuration Issues leads to outdated content or HTTP 4xx– Slow 3rd Party calls impact performance

• Delivery Problems– Web Site Up – but not available to the outside world

Page 55: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Missing Resource Files

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

Page 56: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Wrong Access Right Configuration

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

Page 57: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Bad Connection Pool Configuration

http://apmblog.compuware.com/2014/02/04/when-it-really-is-the-database-to-blame-for-bad-performance-a-story-about-slow-statements-and-resulting-connection-pool-issues/

Page 58: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Bad Web Server Modules

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

Page 59: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

CDN Configuration Problems

http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/

Page 60: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Monitor your CDNs

http://apmblog.compuware.com/2014/03/20/when-cdns-and-ssl-bring-down-your-site-lessons-learned-from-doritos-and-esurance-during-the-super-bowl/

Page 61: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Monitor your other 3rd Parties

http://apmblog.compuware.com/2011/11/21/ecommerce-business-impact-of-3rd-party-address-validation-service/

Page 62: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

www.outageanalyzer.com

Page 63: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

15 Minute Sanity Check – Live Demo

• Dynatrace Application Monitoring– http://www.dynatrace.com/en/products/application-

monitoring.html

• Dynatrace Synthetic Monitoring– http://www.dynatrace.com/en/products/synthetic-

monitoring.html

• Outage Analyzer– www.outageanalyzer.com

Page 64: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Tooling Support

• http://bit.ly/atd2014challenge

• http://www.outageanalyzer.com

• http://blog.dynatrace.com

• http://de.slideshare.net/grabnerandi

• @grabnerandi

Page 65: From Zero to Performance Hero in Minutes - Agile Testing Days 2014 Potsdam

Hands-On