become a rum runner
DESCRIPTION
Life should be good: server's jumping, and the traffic is high. So where are all the sales and signups you expect? A healthy dash of RUM will reveal myriad answers to your conversion problems. Learn to apply Real User Monitoring to your application, so you can find out: Why "average response time" is misleading Why milliseconds can be expensive – even fatal – to your business How many of your visitors get frustrated when they use your site How many of them are satisfied – and how many are just hanging in there How to identify your most crucial outliers Whether your slowdowns are on the back or front end What the performance of your application feels like in real life Don't get lost in a sea of bewildering statistics! Become a RUM runner and navigate the numbers to your advantage. You will learn how to use metrics to discover buried treasure. Find hidden information about your application – information that your users already know, and that either keeps them clicking or drives them away for good. Learn how to interpret the Apdex (Application Performance Index) standard, understand and apply best practices from ITIL (IT Information Library), create transaction groups to meaningfully segregate data from different areas of your website, and more. Avoid common pitfalls – know what these measurements do and do not tell you, and how to recognize the difference. Shop for services that provide readable statistics about your application. Leverage the power of A/B tests. Communicate with management about real-life consequences regarding upgrades and repairs. Go home with a tool set for making the best decisions for providing your visitors with a hassle-free experience that encourages them to become enthusiastic customers.TRANSCRIPT
Become a RUM RunnerUse Real User Monitoring
to learn what life is like
as one of your own customers
Vesna Vuynovich Kovach@Vesna_V_K
What it is
What it’s not
Why you need it
How you can start doing it
Real-User Monitoring (RUM)
PHP developer at OfficeSupply.com
Freelance writer and consumer tech editor eHow Tech, Chron.com (Houston Chronicle), SFGate
(San Francisco Chron.), LIVESTRONG.com
Isthmus, Wisconsin Trails, Corporate Report Wisconsin
Maintain a few sites: how-to-cook-with-vesna.com
atheist-books.com
About me
What is Application Performance Management?
What do my end users experience?
How to interpret charts and terminology
What is Apdex?
How to shop for a Web monitoring service
Ideas and resources for rolling your own
What you’ll learn
Network bandwidth
Network latency
Database access time
CPU usage levels
Memory usage
What do you measure?
The folks in IT
Who cares about these things?
The end user’s experience
What matters?
“In application performance, the end user’s experience is king”
-- The Five Essential Elements of Application Performance Monitoring
Don Jones
Quest Software
www.realtimepublishers.com/book?id=168
What matters?
Concerned with metrics of End User Experience (EUE)
Also known as:
Passive top-down monitoring
Real-time application monitoring
An element of application performance management
Real User Monitoring
APM
A.K.A. “application performance monitoring”
Part of website monitoring
APM asks:
how is an instance of an application performing?
Application Performance Management
Anatomy of APM
Top-Down Monitoring
Real User
Humans (or wild users)
Passive
Find problems that haveoccurred
All events, all the time
Only works if your site has traffic
Synthetic
Robots
Active
Find problems before they occur
Scheduled events
No real visitor traffic needed
Measure
Performance
Availability
Diagnose
Individual incidents
Track
Impact of change
What RUM can do
“Is it broken for anyone, anywhere?”
-- Complete Web Monitoring
Alistair Croll & Sean Power
O’Reilly
books.google.com/books?isbn=0596551355
The question for RUM
Browser differences
Unexpected navigation
Untested site area
Brief outage
Different network connection
Different data
Intermittent problem
RUM Over Robots
RUM and synthetic testing are complementary
A cautionary tale from Website Monitoring
The synthetic testing looked great
But no one was buying the product
Honchos: “Fire the bum!”
RUM revealed the truth
The product was SO popular, the purchase requests were timing out
Company beefed up infrastructure
RUM’s Robot Friend
Apdex
What the heck is
Application Performance Index
Metric for measuring user satisfaction
An open-source standard
Apdex Alliance
Collaboration of several companies
Developed Apdex spec by consensus
Has become a standard measure since 2008
Apdex
A metric of user satisfaction
For transactional applications
Unit-neutral
Operates within a fixed range
0 (worst) to 1 (best)
Illuminates equivalence
Apdex
Specific values of the index report the same user experience across any application, user group, or enterprise
Example: 0.5
Equivalent user experiences observed by different measurement and reporting tools will report the same value
- From the Apdex Specification V1.1, Apdex Alliance, Inc.
Apdex Objectives
Apdex term General Def. HTML/HTTP equiv
Session Continuous period of interaction time
Ex: cookie assignment (not really HTML or HTTP)
Process A series of user interactions + User Think Times. “Transaction” “App Use Case”
A series of Web pages for a defined app function (ex: buy a book)
Task Each user interaction with the app during a Process. “User Wait Time” “Application Response Time.”
A container and 0 or more component objects, and the time required to download.
Task Chain A defined sequence of Tasks. Sum of Task times only. (No user think time.)
A defined sequence of Web pages and their download times.
Turn Each interaction needed to generate a user response or Task. A client-server request-driven round trip. “Chattiness”
HTTP GETs for parts of a Web page. Each object will generate a Turn.
Protocol DNS lookups, TCP ACKS, etc. TCP is the protocol supporting HTTP
Packet Smallest unit of interaction --
Transactional App Taxonomy
Three performance zones
Satisfied
Tolerating
Frustrated
Apdex Zones
Two thresholds (each one a measure of time, usually in seconds)
T
Defines the boundary between satisfied and tolerating
F
Defines the boundary between tolerating and frustrated
F is four times T, by definition
Apdex Thresholds
Count the number of samples in each zone relative to all samples
If T = 4 seconds and all 100 users get transactions <4 sec, then A[T] = 1
T = 4 sec means F = 16 sec
If all 100 users get transactions >16 seconds, then A[T] = 0
Apdex Index
- Source: New Relic
Computing Apdex
Compute Apdex
New Relic newrelic.com
SharePath RUM Express (free) real-user-monitoring.com
Compuware APM Real-User Monitoring Formerly Gomez, DynaTrace compuware.com
Pingdom pingdom.com
SOASTA mPULSE soasta.com/products/mpulse
Products that offer RUM
RUM API
Vesna_V_K
Slides will be on joind.in
Q & A