java monitoring stuff you can get for free (and stuff you ... · java monitoring stuff you can get...
TRANSCRIPT
Java Monitoring Stuff You Can Get For Free (And Stuff You Can’t) Paul Jasek Sales Engineer
A Bit About Me
• Current: • Past:
• Pre-Sales Engineer (1997 – present) • WaveMaker • Wily • Persistence • GemStone
• Application Developer (1988 – 1997) • Dow Jones • Dun & Bradstreet • General Electric / RCA
• Hobbies • Vinyl • Lionel trains • iPhone Development
2 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Why should I Pay for a Monitoring Tool ?
The changing Application Landscape
Try out the Free Monitoring Tools
How people are benefitting from APM?
3
4
2
Today’s Agenda
53 Copyright © AppDynamics. All rights reserved. "
Let’s debug a Slow Application 1
App is running Slow … AND … • It’s running in Production
• Can’t restart the servers
• Can’t reproduce in non prod environment • Data sets and load are not alike
• Just released a new version and it includes • several refactoring, a new deployment architecture
• Revenue critical app, can’t make it any slower by • overloading it with intrusive monitors, additional logging
• Development team is at a different location and time zone
4 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Home grown - Logs, JMX Dashboards etc.
• When it works • First level of defense
• Often added based on learning from outages and critical infrastructure components (pools, caches)
• Systemic issues, e.g. recurring error / warn messages
• When it doesn’t • Sheer volume, don’t know where to start, after all logs are flat • The issue is triggered by specific users, data, regions, etc. • Logging libraries are often synchronous, and heavy on string
operations, hence can lead to performance issues • What if you forgot to add “isDebugEnabled()”
• Time and Server correlation is not easy • Server Time vs. Actual Outage time
5 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Let’s try some free tools
• JConsole – ships with all JVMs
• Visual VM – Exclusive to Oracle (Sun) JVM
• Eclipse Memory Analyzer – Heap Dump Analysis
The Game Changer – AppDynamics Lite 2.0
6 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Let’s go classic - JConsole JVM Health
7 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Let’s go classic - JConsole Thread Dumps Mbean Browser
8 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Let’s go classic - JConsole Mbean Browser
9 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
JConsole – Summary
• When it Works • JVM needs tuning
• Get metrics for Memory, CPU, Threads and Garbage Collection • Experiment with Memory Pools and GC Algorithms
• Systemic slowness, thread dumps point at areas • Server MBean and Custom JMX provides immediate clues
• When it doesn’t • Metrics have no contextual information so no clue why thread is
stuck on a socket ! • You don’t know / own the source code and dev is not next to you • It’s not a JVM tuning issue ? • Passive, always have to be on watch • Overhead
10 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
New Age - VisualVM How are my threads doing ?
11 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
New Age - VisualVM Where are my Threads spending their time ? A Thread dump to show what’s behind the socket read 12 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
New Age - VisualVM What are my Threads munching on? My App Classes
13 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Visual VM– Summary
• When it Works • JVM Metrics overview : Memory, CPU, Threads, GC etc. • Thread Summary View – Wait, Sleep, Monitor, Run Time • Sampling provides some initial clues to the problems
• Thread Sampling – Where time is spent most often • Memory Sampling – What objects are created most often
• When it doesn’t • All the drawbacks of JConsole • The profiler option (instruments all classes and methods) has way
too much overhead even for trivial apps, so not an option • Ships only for Sun (Oracle) JVM
• JRockit ships with a similar tool named JRA • Nothing notable for IBM JVMs for free
14 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Out of Memory ! - Eclipse Memory Analyzer
Leak Suspects Retained Heap Table
15 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Eclipse Memory Analyzer - Summary • When it works
• You have a Heap Dump generated though –XX options • Open up Leak Suspect Report and
• Check out the actual content of the Object (Map <Key, Value>) • Trace the outgoing and incoming references on the object (Who is
holding the map) • It happens often and you know why it happens (you have some
smart engineers working with you !)
• When it doesn’t ! • Knowing what’s big is less important, what you need is
• How it got big ? • Who is adding / putting elements
• Doesn’t happen all the time and you can’t associate with a specific event, it grows in lock step, not linear !
16 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
AD LITE 2.0 The Game Changer
AppDynamics Lite 2.0 • Application flow map
• Identifies key components – Servlets, Struts, EJBs • External calls – HTTP, RMI, JMS, JDBC, THRIFT etc.
• Top Down View of Application • Recognizable artifacts e.g. URI vs. Class Name • Aggregated at the top, as experienced by real user
• Dynamic Thresholds • Learns by itself what’s slow for Login vs. Checkout
• Continuous visibility and Deep Dive when needed • Every click is monitored, capture outliers – Slow, Errors, Stalls
• Your very own dashboard – App, BT, JMX Metrics • Always ON, alerts when things go wrong
Built for Production < 2% Overhead
18 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Shameless Plug
19 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
http://www.appdynamics.com/products-free-download.php
When does it make sense to Pay for a Monitoring Tool?
If your app … • Deployed on more than 5 JVMs
• Need to see data at one place, be able to navigate easily
• Request processing is multi threaded, distributed • Need correlation across threads and services
• Agile releases, Now vs. Then • Last 5 mins ! Last Week ! Last Quarter
• Custom Data Capture, Slicing and dicing of the data • Capture Parameters, Select where session_id =‘’, All slow SQLs
• Advanced Baselines and Policy Engine • Static / Dynamic / Seasonal Baselines • Making sense from the metrics deluge
• Dashboards, Business Metrics • Pre built, custom widgets, live business metrics
• Moving towards Cloud Architecture 21 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
The Commercial Landscape
• The Big 5s
• IBM • CA • Compuware • HP • BMC
• The New Age Players • AppDynamics • New Relic
22 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Must haves for your Paid Tool • No professional services
• Plug and Play, no experts needed to configure and tune the tool
• You don’t tell the Tool, the Tools tells you what you got • Self adaptive learning
• Always ON in production with very low overhead • Not a factor of how much traffic
• Easy to learn the tool and extend too • Start with the basics and be an expert over time
• Easy to share and collaborate across the Organization • Operation, Dev, QA, Performance, Architects
• Total Cost of Ownership – include everything • Software, Hardware, Deployment, Learning, Maintenance
• Continuous Innovation • Cloud, NO SQL, Async Transactions
23 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
AppDynamics Pro
Custom Data Collection API / Scripts
Custom Action API / Scripts
Instrumentation Point API
Business Metric Collection API
Custom Header Correlation API
Controller
Application Server Agent
(JVM/CLR)
Machine Agent Alert&
Integra+on&API&/&Scripts&
Data&Feed&API&
Data&Collec+on&
API&
Cloud&Connector&
API&
How is APM Helping Organizations ?
• A nice problem to have... • http://techblog.netflix.com/2011/02/redesigning-netflix-api.html
• A “Blockbuster in the Cloud” • World’s largest Java public cloud deployment
AppDynamics in Action: Netflix
37x Growth Jan 2010-Jan 2011
26 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
1ST CLICK: THE APP TOPOLOGY
27 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
2ND CLICK: REVIEW SLOW TRANSACTIONS
28 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
3RD CLICK: TRIAGE LOCATION
29 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
4TH CLICK: FIND THE LINE OF CODE
30 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
• E – Travel Site
• Tomcat, Active MQ, My SQL
• 200 Business Transactions
AppDynamics in Action: Karavel, France
31 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
CUSTOM DASHBOARDS
32 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
DRILL DOWN ON SLOW TRANSACTIONS
33 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
FIND THE CODE AND THE BAD SQL
34 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
VALIDATE THE FIX !
35 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
FUTURE PROOF WITH ALERTS
36 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "
Q&A
Q&A
37 March 5, 2012 | Copyright © 2010 AppDynamics. All rights reserved. "