evidence based performance tuning enterprise java applications · performance issues • instructs...
TRANSCRIPT
Evidence based performance tuning of
enterprise Java applicationsenterprise Java applications
By Jeroen [email protected]
Evidence based performance tuning of
enterprise Java applications
By Jeroen [email protected]
enterprise Java applications
Bad performance can lead to complete failure
3
Bad performance: hard to fix
• Call center agent took 20 minutes to work through five screens with customer– Customer un-friendly
– High cost
• Speedup needed before rush of number porting
4
• Speedup needed before rush of number porting– Major app redesign was needed
• Result– very slow system when up, mostly system down
– damage to company image
– loss of thousands of customers and $100,000,000
– company crash
Goal
Provide an answer to the question:
How can I speedup my enterprise Java app?
5
Who is Jeroen Borgers?
• sr. consultant with Xebia
• helps customers on enterprise Java performance issues
• instructs Java performance tuning courses• instructs Java performance tuning courses
• doing Java since 1996
• various roles in several industries
– developer, architect, team lead, quality officer, auditor, performance tester and tuner
www.xebia.com
Who is Xebia?
• international IT organization
– NL, FR, IN
• specialized in enterprise Java
• consultancy and projects• consultancy and projects
• 5 years old
• ±100 employees and growing
• share knowledge
• focus on quality
www.xebia.com
Agenda
• Big picture architecture vs. implementation
• How to achieve bad performance
• Evidence based tuning for high speed
• Tools overview
8
• Tools overview
• Largest Dutch web shop case
• Conclusions
Enterprise Java performance big picture
• Java is not slow anymore
• Importance how you do it: algorithms
• Typical enterprise app: most time spent in calls out of Java process– Remote calls
9
– Remote calls
– Database access
for (int i = 0; i < size; i++) {
order = orderDao.getOrder(i);
if (order.isDelivered()) { DB
>~10 ms~0,1µs
factor 100.000 !
Architecture enables performance and scalability
• Key choices– Application distribution
– Persistent data access approach: O-R mapping
– Data conversion: XML, encodings
– Presentation tier technologies
10
– Presentation tier technologies
– Use of proprietary features of db, app server
– Data caching
• It is all about:– Remote calls
– Database access• More general: data conversion between representations
Architecture enables, implementation makes it happen
• Minimize remote calls and other I/O
• Speed-up data conversion
11
• Code optimizations involving remote calls and data conversion
?
Architecture enables, implementation makes it happen
• Minimize remote calls and other I/O– Direct JDBC: use of batch updates
– Hibernate: use of query caching
• Speed-up data conversion– XML usage: use JiBX instead of JAXB
12
– XML usage: use JiBX instead of JAXB
– URL encoding: use commons codec instead of java.net
• Code optimizations involving remote calls and data conversion– Algorithms, collections
– Threading and synchronization
Architecture enables performance & scalability
• The architecture
13
Architecture enables performance & scalability
• The architecture is scalable
14
?
Architecture enables performance & scalability
• The architecture is scalable
15
• Add horse power: more load with same speed
Architecture enables performance & scalability
• How one implements the architecture makes a difference
16
• But the requirement of 120 km/h will never be met!
• Major redesign needed
Architecture enables performance & scalability
• A more suitable architecture for 120 km/h:
17
Architecture enables performance & scalability
• More demanding requirements can be met with the right implementation
• Bugatti EB Veyron 16.4
18
16.4
• Top 407 km/h
• 0-300 in 14 s.
How to achieve bad performance
• Many JavaEE apps have hard to solve performance problems
• How to achieve this?
– Architecture not suitable for requirements
19
– Architecture not suitable for requirements
– Ignore the problems too long
– Don’t follow best practices
– Optimize counterproductively
Architecture not suitable
• Architect: “Most important is a pure, flexible, scalable architecture: we distribute our application components and loosely couple with XML and web services.”
20
• Nonsense!
– Those who pay the bill care about efficient working, not purity;
– Speed is traded for scalability: remote calls and data conversion can be killing;
– How is flexibility gained?
Ignore problems too long
• Project leader: “First we get the functionality right, then we start worrying about performance.”
• Performance is often ignored until the final test, a week before production, or later!
• To meet requirements you may need a major re-
21
• To meet requirements you may need a major re-design: costly
• Solved by:– Verify the architecture in a POC: test a vertical slice
– Test performance continuously
How to achieve bad performance
• How to achieve this?
– Architecture not suitable for requirements
– Ignore the problems too long
– Don’t follow best practices
22
– Don’t follow best practices
– Optimize counterproductively
Don’t follow best practices
• Best practices are like:
– Use Map instead of List for lookups
– Use I/O buffering: InputStreamBuffer
– Use SQL batch updates
23
– Use SQL batch updates
– Use db table indexes
– Use StringBuffer for String concatenation?
?
Don’t follow best practices
• Best practices are like:
– Use Map instead of List for lookups
– Use I/O buffering: InputStreamBuffer
– Use SQL batch updates
24
– Use SQL batch updates
– Use db table indexes
– Use StringBuffer for String concatenation?
• Be aware: a best practice may not work in your situation
Best practices can be premature optimizations with trade-offs
• Developer: “String performs badly, we use StringBuffer for concatenation everywhere.”
• Many books: “use StringBuffer fortoString() { return “[“ + getName() + “]”; }
• faster istoString() {
StringBuffer buf = new StringBuffer(“[“);
buf.append(getName()).append(“]”);
25
buf.append(getName()).append(“]”);
return buf.toString();
} ”• However: same bytecode!• Maintainability and developer time is traded for an assumed, non-
existing performance gain• “Premature optimization is the root of all evil” - Hoare and Knuth
Optimize counterproductively
• Updating state remotely by sending only delta’s like RemoveAttribute, AddAttribute– many small objects with much serialization and
communication overhead
• Object and thread pooling depending on JVM
26
• Object and thread pooling depending on JVM
• Optimizations are very time and environment dependent
• So: don’t trust your knowledge or intuition, but measure that your optimization solves the problem!
Bad performance
• Architecture not suitable for requirements
• Ignore the problems too long
• Don’t follow best practices
• Optimize counterproductively
27
• Optimize counterproductively
How to speedup?
Evidence based tuning for high speed
• Don’t make assumptions but measure• Have quantified requirements, prove they are met or not• Prove the architecture performs in a POC• Continuously test performance during development
– to get a first impression– for quick feedback on changes
28
– for quick feedback on changes
• Test representatively in a dedicated environment– with production-like load– on a production-like environment– with production-like data
• Fix problems when found• Prove the effect of each optimization
• Monitor the performance of your app in production
Evidence based tuning cycle
Tools for measuring
• OS profilers to see resource usage: CPU, memory, I/O, threading • Database analyzers to see query behavior in the database• Network analyzers to see network traffic• App server monitors to see app server resource usage: heap,
connection pool, stmt cache,..• J2EE analyzers to see J2EE behavior under load• Instrumentation monitors to see timing at the instrumented
30
• Instrumentation monitors to see timing at the instrumented locations
• Java profilers to see detailed CPU and memory behavior in JVM• Continuous test tools to quickly get timing feedback of code
changes• Load test tools to generate usage load• Web site analyses tools to optimize page resource usage
Largest Dutch web shop case
www.xebia.com ?
Case: speeding up a web shop
32
Wehkamp TRC architecture
web server: ASP/.NetTuxedo
DB
33
TRC-appsrv: Java
DB
Services here
DB
Speedup results at Wehkamp
• Several services optimized to meet requirements
• Most expensive: voegKlantToe (addCustomer); verwerkenNieuweOrder(processNewOrder): from 7 s. to 1.5 s.
• 45% db-queries, 45% tux-calls, 10% java code
• Top optimizations:
34
• Top optimizations:– Reduce number of db queries
– Optimize most expensive db queries
– Reduce number of Tux/mainframe calls
– Optimize most expensive Tux/mainframe calls
• App orchestrates: many Java code changes needed for this
Tools used
• Load test tool: Apache JMeter• App level monitor: JAMon API• History and trends reporter: JARep• Continuous test tool: cruisecontrol+maven+JMeter• Java profiler: Quest JProbe
35
• Java profiler: Quest JProbe• App server monitor: Tivoli Performance Monitor• J2EE analyzer: Quest PerformaSure• Database analyzer: Quest Toad, Oracle Statspack• Unix OS profiler: vmstat, nmon• Network analyzer: Wireshark
?
JMeter for generating load and black box testing
36
JMeter mail to stakeholders
37
JAMon for monitoring statistics
added
service name
38
Instrumenting with JAMon API
39
JAMon API call
JARep reporting for history
40
41
42
43
JVM / app server node 4
JVM / app server node 3
deployment of JARep
your app
jamon.warjamonadmin.jsp
ShowJamonCounters.jsp DataFetcher
DataPersister
every 5 minutes
JVM / app server node 3
JVM / app server node 2
JVM / app server node 1
www.xebia.com
jamonapi.jar
JVM / off-line box
RDBMS
JaReporter.jsp
PerformaSure for analyses under load
45
Analysis tools compared
• Profiler JProbe– Pro: High detail: code line level
– Con: High overhead, cannot deal with load
• JAMon API + JARep– Pro: Low overhead < 1%
46
– Pro: Low overhead < 1%
– Pro: Enables comparing over time, see trends
– Con: No call tree, cannot drill down
– Con: Limited to built-in measure points
• PerformaSure– Pro: reasonable overhead, slick UI, measures all on JVM
– Con: not cheap, configuration, wait time
Conclusions & recommendations
• Architecture and implementation both determine performance
• Consider whole application life cycle: requirements, POC, continuous testing, representative testing, monitoring
47
representative testing, monitoring
• Biggest gains are in– Remote calls and other I/O
– Database access or other data conversions
• Tuning based on evidence is key– measure and prove by using the right tools
– JMeter, JAMon and JARep can help
“Meten is weten” (measuring means knowing)
48
References
• http://www.xebia.com• http://blog.xebia.com• http://podcast.xebia.com• http://www.javaperformancetuning.com• http://sourceforge.net/projects/jarep
49
• http://sourceforge.net/projects/jarep• http://www.jamonapi.com• http://jakarta.apache.org/jmeter/• http://www.quest.com/performance_management
Q & A
50