©2008, NetForecast, Inc., all rights reserved.
How Apdex Helps Performance Management Benchmarks
Apdex Symposium 2008December 9, 2008
Voice, Video and DataApplication Performance Experts
Apdex Symposium, Las VegasCMG Session 308
Peter SevcikNetForecast, Inc. [email protected] Emerson Drive www.netforecast.comCharlottesville, VA 22901 434 249 1310
CMG Session 308
Outline
Performance Management Model
Benchmarks Verify Model
How Apdex Improves Benchmarks
Best Practices: State of the Art
©2008, NetForecast, Inc., all rights reserved. Slide 2
© 1990-2008 Peter Sevcik and NetForecast, Inc., All rights reserved.
©2008, NetForecast, Inc., all rights reserved.
Performance Management Defined
Performance Management (PM):
A holistic approach to managing applicationA holistic approach to managing application performance that integrates all of your organizational resources with one overarching goal — to deliver application performance levels that meet the needs of your business
©2008, NetForecast, Inc., all rights reserved. Slide 3
PM Layers
CorporateManagement
Requirements
Actions Information
Data
People
ProcessPerformanceManagement(PM) Layers
©2008, NetForecast, Inc., all rights reserved. Slide 4
UsersInfrastructure
Corporate Assets
Product
©2008, NetForecast, Inc., all rights reserved.
Filling in the Layers
Requirements
Actions Information
Data
Best Practices
Processes
People PerformBest Practices
Essential ProcessesSupport the People
O ti l T l S t
©2008, NetForecast, Inc., all rights reserved. Slide 5
ToolsOptimal Tools SupportProcesses and People
Define the Strategy First and Architecture Second
StrategyMost tool vendors
Best Practices
Processes
Co
rrec
t W
ay t
o B
uild
pri
ses
s P
ath
ignore Processesor Best Practices
“Don’t worry it isall automatic”
©2008, NetForecast, Inc., all rights reserved. Slide 6
Architecture
Tools
C
Mo
st E
nte
rpA
re o
n T
his
©2008, NetForecast, Inc., all rights reserved.
Performance Management Model
Lin
k
Understand
Measu
rePM
©2008, NetForecast, Inc., all rights reserved. Slide 7
Communicate
Best Practices
Understand
People – Is your staff focused on the correct goals?
Understand Know your applications,
users, and requirements
Measure Properly measure key aspects
of application performance
Communicate Lin
k
Understand
Measu
r
ContinualService
I t
©2008, NetForecast, Inc., all rights reserved. Slide 8
Provide relevant performance reports to management
Link Show specific business-
performance links
Communicate
reImprovement
©2008, NetForecast, Inc., all rights reserved.
Processes
Incident
Turn events into actionable alarms with good diagnostic capability
Process – Do you have proper procedures in place?
good diagnostic capability
Goal is to restore normal service operations as quickly as possible
Availability
Fundamental reporting of system health
Historic data determines trends
Capacity
Match costs to business needs by adding or removing resources
ServiceAssurance
Management
Capacity Management
©2008, NetForecast, Inc., all rights reserved. Slide 9
g g
“What if” analysis and planning are key
Service Assurance
Service levels are tailored to meet business needs
Continually monitor and coordinate corrective action
Availability Management
Incident Management
Outline
Performance Management Model
Benchmarks Verify Model
How Apdex Improves Benchmarks
Best Practices: State of the Art
©2008, NetForecast, Inc., all rights reserved. Slide 10
© 1990-2008 Peter Sevcik and NetForecast, Inc., All rights reserved.
©2008, NetForecast, Inc., all rights reserved.
NetForecast PM Benchmarks
Second annual survey of enterprises to characterize their current APM practices and assess results Surveys completed in February 2007 and July 2008y p y y
Sample size of approximately 300 enterprises both years
Survey had questions on performance management methodology along with effectiveness metrics How problems are found
Time to solve problems
Performance meeting business needs
A il bilit
©2008, NetForecast, Inc., all rights reserved. Slide 11
Availability
Response time
Benchmark and effectiveness scores were determined for each enterprise
Best Practices Results
7
8
ss M
etrics
Best Practices51% Gain
k
Understand
Me
5
6
Perform
ance Effectivene
(0=p
oor, 10=excellent)
Gain is the slopeof the curve.
Or the return inperformance delivered(vertical axis)
Communicate
Lin
k
ea
su
re
©2008, NetForecast, Inc., all rights reserved. Slide 12
3
4
1 2 3 4 5 6 7 8 9 10
Composite P
NetForecast Benchmark Score
(0=poor, 10=excellent)
(vertical axis)for the investment inthe benchmark(horizontal axis)
©2008, NetForecast, Inc., all rights reserved.
Processes Results
7
8
ss M
etrics
5
6
Perform
ance Effectivene
(0=p
oor, 10=excellent) Processes
24% Gain
©2008, NetForecast, Inc., all rights reserved. Slide 13
3
4
1 2 3 4 5 6 7 8 9 10
Composite P
NetForecast Benchmark Score
(0=poor, 10=excellent)
The PM Model Works
Strategy
Best Practices
Processes
51% Gain
24% Gain
©2008, NetForecast, Inc., all rights reserved. Slide 14
Architecture
Tools
©2008, NetForecast, Inc., all rights reserved.
Outline
Performance Management Model
Benchmarks Verify Model
How Apdex Improves Benchmarks
Best Practices: State of the Art
©2008, NetForecast, Inc., all rights reserved. Slide 15
© 1990-2008 Peter Sevcik and NetForecast, Inc., All rights reserved.
Apdex Defined
Apdex is a numerical measure of user satisfaction with the performance of enterprise applications
It defines a method that converts many measurements into one number Uniform 0-1 scale
0 = no users satisfied
1 = all users satisfied
©2008, NetForecast, Inc., all rights reserved. Slide 16
Standardized method that is a comparable metric across Applications,
Measurement approaches, and
Enterprises
©2008, NetForecast, Inc., all rights reserved.
How Apdex Works
Existing TaskResponse TimeMeasurement 7
1. Define T for the applicationT = Target time (satisfied-tolerating threshold)
F = 4T (tolerating-frustrated threshold)2. Define a Report Group
Frustrated
Good
Fair
Poor0.50T
1.00T
0.85T
0.94T
0.70T
ExcellentReport Group:ApplicationUser GroupTime Period
easu e e tSamples2
3
5
7
6
p p3. Extract data set from existing measurements4. Count number of samples in three zones5. Calculate the Apdex formula6. Display Apdex value showing T7. Optionally display value using quality colors
©2008, NetForecast, Inc., all rights reserved. Slide 17
Tolerating
ApdexT=Total samples
Satisfied2+
Satisfied
Tolerating
Un
ac
ce
pta
ble
0.00T
T1
4F
6
0.91 [6]
Value T
14%
16%
Tim
e
Apdex Example
Major eCommerce site ($4B annual on-line sales)North American broadband users accessing the San Francisco data center
6%
8%
10%
12%
y o
f E
xper
ien
cin
g t
he
52% Satisfied
This site had an average response timeof 4 seconds, so it looked like all was well
But: Apdex = 0.7310 = Fair
©2008, NetForecast, Inc., all rights reserved. Slide 18
0%
2%
4%
0 20 40 60 80 100 120Load Time of a Typical Business Page (sec)
Pro
bab
ilit
y
42% Tolerating
6% Frustrated
©2008, NetForecast, Inc., all rights reserved.
Apdex Methodology
Gather Agree Improve
• Measure• Gather baseline data• Test Apdex Parameters
A T
1) Work withusers to set T
2) Work with businessmanagers to define theservice objective for A
• Apdex reports & trends• Improve performance
where needed• Continual performance
& process improvement
©2008, NetForecast, Inc., all rights reserved. Slide 19
Keys to PM Success
ProcessesBest Practices
AssuranceLink
Collaboration Dependability
©2008, NetForecast, Inc., all rights reserved. Slide 20
Assurance
Capacity
Availability
Incident
Link
Communicate
Measure
Understand
©2008, NetForecast, Inc., all rights reserved.
Leveraging Apdex for Success
ProcessesBest Practices
The Apdex methodology forces organizations to achieve better PM
AssuranceLink
Collaboration Dependability
Apdex Benefit Zone
©2008, NetForecast, Inc., all rights reserved. Slide 21
Assurance
Capacity
Availability
Incident
Link
Communicate
Measure
Understand
Apdex is a numerical measure of application performance relative to business objectives.The Apdex open standard is managed by the Apdex Alliance – see apdex.org.
The Apdex Alliance has more than 800 individual and 10 vendor members.
How Apdex Helps PM
ProcessesBest PracticesSelection of T and theApdex score is the Excellent foundation
Apdex dialogsupports collaboration
Open standardSupports dependability
AssuranceLink
Collaboration DependabilityFacilitates technologistsand non-technologistscommunications
Apdex score is thelink to business needs
Defines
ce e t ou dat oFor meaningful SLAs
Trend analysis helpsfind capacity problems
©2008, NetForecast, Inc., all rights reserved. Slide 22
Assurance
Capacity
Availability
Incident
Link
Communicate
Measure
UnderstandForces you tounderstand yourapplications and users
Definesmeasurementtaxonomy
Discovers hardto find incidents
Can be applied toavailability analysis
©2008, NetForecast, Inc., all rights reserved.
Outline
Performance Management Model
Benchmarks Verify Model
How Apdex Improves Benchmarks
Best Practices: State of the Art
©2008, NetForecast, Inc., all rights reserved. Slide 23
© 1990-2008 Peter Sevcik and NetForecast, Inc., All rights reserved.
What Are Best Practices?
Understand Have a good definition of which technical parameters are important
Good understanding of which applications are mission criticalg
Good understanding of which users are more important
How well the above is documented
How well these understandings are distributed within the IT organization
Measure How well the important technical parameters are measured
How well the measurements are tracked over time
©2008, NetForecast, Inc., all rights reserved. Slide 24
How well the measurements are tracked over time
Setting critical thresholds for the important parameters
Formal processes to ensure that these measurements are carried out
Automation for efficient data gathering and correlation
©2008, NetForecast, Inc., all rights reserved.
What Are Best Practices? (con’t)
Communicate Provide relevant performance reports to management
Ensure that the reports a meaningful to non-IT staffg
Measurements are communicated throughout the enterprise
Distribute information on a regular basis
Don’t limit information thinking the recipient won’t understand it
Link Performance targets are agreed upon with business managers
Applications monitored are confirmed to be business critical
Performance targets are confirmed to be relevant to the business
©2008, NetForecast, Inc., all rights reserved. Slide 25
Performance targets are confirmed to be relevant to the business
Application-level SLAs are in place
Meet with business managers periodically to review the above
Best Practices BenchmarksIndustry State of the Art
20%
25%
OverallBenchmark
Medians
10%
15%
Per
cen
t D
istr
ibu
tio
n
20072008
Medians2007: 42008: 5
State of theart improved10% in a year
©2008, NetForecast, Inc., all rights reserved. Slide 26
0%
5%
1 2 3 4 5 6 7 8 9 10
Best Practices Benchmark
©2008, NetForecast, Inc., all rights reserved.
Private/Public Traffic in 2008
chm
arks
8
9
10
Private vs. Public makes no differenceAll within a tight 0.3 point range
n B
est
Pra
ctic
es B
enc
3
4
5
6
7
©2008, NetForecast, Inc., all rights reserved. Slide 27
Med
ian
0
1
2
100/0 75/25 50/50 25/75 0/100
Private/Public RatioPublicPrivate
Impact of Enterprise Size in 2008
8
9
10
chm
arks
Enterprise size has a 2 point range
3
4
5
6
7
n B
est
Pra
ctic
es B
enc
©2008, NetForecast, Inc., all rights reserved. Slide 28
0
1
2
<50 50-100 100-500 500-1K 1K-10K 10K-40K >40K
Med
ian
Number of Employees in the EnterpriseBigSmall
©2008, NetForecast, Inc., all rights reserved.
Benchmarks Compared
Understand 20072008
Communicate
Measure
©2008, NetForecast, Inc., all rights reserved. Slide 29
0 1 2 3 4 5 6 7 8 9 10
Link
Median Best Practices Benchmark
Why Measure is Less Than Understand
Contribution to the Difference Between Understand and Measure
End-User Response Time
Server Errors
Server Query-Response Time
Availability
Server Utilization
TCP Transaction Response Time
Transactions ProcessedOverall shift of2 benchmark points
©2008, NetForecast, Inc., all rights reserved. Slide 30
- 1 2 3 4 5
Best Practices Benchmark
Traffic Flows by Application
WAN Performance
Bandwidth Utilization
2 benchmark pointsfrom 6 to 4 onprevious slide
©2008, NetForecast, Inc., all rights reserved.
Real Company Examples
UnderstandBank
HospitalityGovernment
ComputersConsumer Website Pharma
Individual Enterprise Best Practices Profiles Are Diverse
Healthcare
Communicate
Measure
Telecom Eq. Airline
©2008, NetForecast, Inc., all rights reserved. Slide 31
0 1 2 3 4 5 6 7 8 9 10
Link
Best Practices Benchmark
NetForecast Benchmark Service
Benchmark how well your enterprise is delivering the Best Practices and essential Processes
Strategy
Rank your benchmark scores against industry norms (database of more than 600 enterprises)
Gap Analysis shows where you Best Practices and Processes excel or are deficient
Best Practices
Processes
©2008, NetForecast, Inc., all rights reserved. Slide 32
Workshop meeting that defines your successful performance management strategy
Tools
©2008, NetForecast, Inc., all rights reserved.
Thank You
More information is available at:
Free articles and reports on performance
measurement analysis and managementmeasurement, analysis, and management
are available at www.netforecast.com