from zero to performance hero in minutes - agile testing days 2014 potsdam
DESCRIPTION
As a Tester you need to level up. You can do more than functional verification or reporting Response Time In my Performance Clinic Workshops I show you real life exampls on why Applications fail and what you can do to find these problems when you are testing these applications. I am using Free Tools for all of these excercises - especially Dynatrace which gives full End-to-End Visibility (Browser to Database). You can test and download Dynatrace for Free @ http://bit.ly/atd2014challengeTRANSCRIPT
Performance Clinic Workshop
From Zero to Performance HeroAndreas Grabner (@grabnerandi)
[email protected]://bit.ly/atd2014challenge
Why Applications Fail and/or are Slow!
Performance areas we cover today
• Frontend
• Backend
• Deployment
Frontend
http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
#Overloaded Web Pages• #1: Too many resources loaded on web page
– Images: can be “sprited”– CSS and JS: can be merged
• #2: Large Content– Images: do you need high-res? Compress!– CSS and JS: Minify, remove comments, …
• #3: 3rd Party Content– Slow or too much
• #4: AJAX to the MAX!– Too many AJAX Calls requesting too much data
Examples on Overloaded
This is a heavy page
This is a light page
http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/
softdrink.com during SuperBowl
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/
Fifa.com during Worldcup
http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/
Kia.com during SuperBowl
Ad
on
air
GoDaddy.com during SuperBowl
1h before SuperBowl KickOff
1h after Game ended
Kia vs GoDaddy: The Facts!
http://apmblog.compuware.com/2014/02/19/dns-tcp-and-size-application-performance-best-practices-of-super-bowl-advertisers/
3rd Party Content
Do you really need all bells and whistles?
# of Domains # of Resources Total Bytes DNS [ms] Connect [ms]
With Third Party Content
26 176 2856 Kb 1286,82 1176,09
Without Third Party Content
2 59 897 Kb 0,91 22,25
Too heavy AJAX/JavaScript
1.1s on my IE 10 to
execute magicSpanLinks()
The each loop calls this block of JavaScript for every span node
759 span nodes are processed
by the anonymous function
It adds the dynamically generated link and removes the
old spanWe can see all the DOM
Modifications and how this sums up in execution time
3rd Party JavaScript Problems
Slow 3rd Party
Libraries
Slow jQuery
Lookukps
15 Minute Sanity Check – Live Demo
• Dynatrace Performance Test Center– http://www.dynatrace.com/en_us/application-performance-
management/products/performance-center.html
• Dynatrace Browser Agent (formerly AJAX Edition)– http://apmblog.compuware.com/2014/07/01/can-monitor-web-
performance-free/
• Fiddler – simulate other browsers– http://www.telerik.com/fiddler
• PerfMap – HeatMap for a Website– https://github.com/zeman/perfmap
Summary – WPO Best Practices• Additional Blog Posts
– http://apmblog.compuware.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/
– http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/
– http://apmblog.compuware.com/2010/08/25/top-10-client-side-performance-problems-in-web-2-0/
• Recommended Books from Steve Souders covering things like– Make fewer HTTP Requests
– Proper Cache Settings
– Optimize/Compress Content
– Use CDNs
– Watch out for 3rd Parties
Tooling Support• http://bit.ly/dttrial• http://ajax.dynatrace.com• http://yslow.org• https://developers.google.com/speed/pagespeed• http://www.webpagetest.org/• http://www.sitespeed.io/• http://www.showslow.org• http://www.telerik.com/fiddler• https://github.com/zeman/perfmap• https://chrome.google.com/webstore/detail/user-agent-switcher-for-
c/djflhoibgkdhkhhcedjiklpkjnoahfmg
Hands-On
Bad Architectural Decisions#1: Database Access
N+1 Query Problem
Loading Too Much Data
Connection Leaks
#2: External Frameworks
Bad Configuration of O/R Mappers, e.g: Hibernate
Worked well in Sample App Attitude
#3: Excessive Logging & Exceptions
Debug Logging turned on
Using outdated logging libraries
Exception Overload
#4: Memory Leaks
Keeping objects for too long
Bad Cache Implementations
#5: Bad Coding
Synchronization to Death
High on CPU
Examples
Database Access
Database: N+1 Query Problem
Querying too much DataAll of these requests with large size have the same problem in common:
DB Access
24889 Calls to the Database!
Tomcat needs to process all this data!SideEffect: High Memory Usage to process
data -> results in high GC
High GC is not the problem. It is just the symptom of too much data
loaded!
Too many connections 12444 individual connections to execute 12444 individual SQL
Statements
Classical N+1 Query Problem. The same SQL is executed many times
with different WHERE ClauseOptimize this by only calling it once
with a better WHERE Clause
Individual executions are fast. But VOLUME is
killing you
Filtering Data in Memory instead of DB
Most lookups are done by getRoomCapacity
Assumption: All the data loaded ends up in Hashtable. The lookups are then very expensive because these Tables
are so huge
External Frameworks
Non Optimized Hibernate
http://apmblog.compuware.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/
Non Optimized Telerik Controls
http://apmblog.compuware.com/2014/04/03/database-access-patterns-gone-wild-inside-telerik-sharepoint-and-asp-net/
Logging
Too much Logging#1: Top Problem: log4j.callAppenders
#2: Most of logging done from fillDetail method
#3: Doing “DEBUG” log output: Is this necessary?
Exceptions vs. Log Messages
http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/
Exception Logging Overload
Tomcat logAbandoned=true flag causes many exceptions objects to be created
http://apmblog.compuware.com/2012/08/01/top-performance-mistakes-when-moving-from-test-to-production-excessive-logging/
Exception Performance Overhead
http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/
Memory
Oracle JDBC Driver BugEach of the 10 JVMs per Host consumes up to 4.1GB until they crash (41GB per Host)
It is a reoccurring pattern over months
http://apmblog.compuware.com/2014/01/07/hunting-a-oracle-jdbc-memory-leak-crashing-an-80jvm-websphere-cluster/
No Object Cleanup Code!
http://apmblog.compuware.com/2014/02/26/memory-leaks-load-balancing-and-deployment-settings-testing-lessons-learned-from-the-ecommerce-industry/
Bad Coding
Slow Custom RegEx
http://apmblog.compuware.com/2014/10/16/15-minutes-spent-optimizing-performance-save-millions-lost-revenue/
Slow Content Rendering
Rendering Methods take very long!
Intializing GlyphLayout takes very long
These slow rendering methods are called very frequently!
Synchronization
http://apmblog.compuware.com/2013/09/24/100-performance-overhead-by-websphere-activity-log-when-dev-is-not-aware-of-settings-in-production/
15 Minute Sanity Check – Live Demo
• Dynatrace
Summary – Backend Best Practices• Additional Blog Posts
– http://apmblog.compuware.com/2013/04/10/top-8-application-performance-landmines/
– http://apmblog.compuware.com/2010/06/15/top-10-performance-problems-taken-from-zappos-monster-and-co/
• Online Java Enterprise Performance Book: http://javabook.compuware.com
• Key Takeaways– Educate Developers
– Understand Frameworks you are using
Hands-On
Deployment
Common Deployment Mistakes• Missing Resource Files
– Many HTTP 4xx– Many HTTP 3xx Redirects -> overhead!
• Missing Configuration Files– Web Server Access Rules -> Leads to HTTP 4xx– Web Server -> App Server: Connection & Thread Pools
• Bad Modules or Configuration Problems– Leading to bad requests and overhead– Rewrite and Redirect Modules: Long chains of redirects
• 3rd Party: – CDN Configuration Issues leads to outdated content or HTTP 4xx– Slow 3rd Party calls impact performance
• Delivery Problems– Web Site Up – but not available to the outside world
Missing Resource Files
http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/
Wrong Access Right Configuration
http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/
Bad Connection Pool Configuration
http://apmblog.compuware.com/2014/02/04/when-it-really-is-the-database-to-blame-for-bad-performance-a-story-about-slow-statements-and-resulting-connection-pool-issues/
Bad Web Server Modules
http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/
CDN Configuration Problems
http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/
Monitor your CDNs
http://apmblog.compuware.com/2014/03/20/when-cdns-and-ssl-bring-down-your-site-lessons-learned-from-doritos-and-esurance-during-the-super-bowl/
Monitor your other 3rd Parties
http://apmblog.compuware.com/2011/11/21/ecommerce-business-impact-of-3rd-party-address-validation-service/
www.outageanalyzer.com
15 Minute Sanity Check – Live Demo
• Dynatrace Application Monitoring– http://www.dynatrace.com/en/products/application-
monitoring.html
• Dynatrace Synthetic Monitoring– http://www.dynatrace.com/en/products/synthetic-
monitoring.html
• Outage Analyzer– www.outageanalyzer.com
Tooling Support
• http://bit.ly/atd2014challenge
• http://www.outageanalyzer.com
• http://blog.dynatrace.com
• http://de.slideshare.net/grabnerandi
• @grabnerandi
Hands-On