how apdex helps performance management benchmarksapdex.org/documents/session308.2sevcik.pdf · how...

©2008, NetForecast, Inc., all rights reserved.

How Apdex Helps Performance Management Benchmarks

Apdex Symposium 2008December 9, 2008

Voice, Video and DataApplication Performance Experts

Apdex Symposium, Las VegasCMG Session 308

Peter SevcikNetForecast, Inc. [email protected] Emerson Drive www.netforecast.comCharlottesville, VA 22901 434 249 1310

CMG Session 308

Outline

Performance Management Model

Benchmarks Verify Model

How Apdex Improves Benchmarks

Best Practices: State of the Art

©2008, NetForecast, Inc., all rights reserved. Slide 2

© 1990-2008 Peter Sevcik and NetForecast, Inc., All rights reserved.


Performance Management Defined

Performance Management (PM):

A holistic approach to managing applicationA holistic approach to managing application performance that integrates all of your organizational resources with one overarching goal — to deliver application performance levels that meet the needs of your business


PM Layers

CorporateManagement

Requirements

Actions Information

Data

People

ProcessPerformanceManagement(PM) Layers


UsersInfrastructure

Corporate Assets

Product


Filling in the Layers

Requirements

Actions Information

Data

Best Practices

Processes

People PerformBest Practices

Essential ProcessesSupport the People

O ti l T l S t


ToolsOptimal Tools SupportProcesses and People

Define the Strategy First and Architecture Second

StrategyMost tool vendors

Best Practices

Processes

Co

rrec

t W

ay t

o B

uild

pri

ses

s P

ath

ignore Processesor Best Practices

“Don’t worry it isall automatic”


Architecture

Tools

C

Mo

st E

nte

rpA

re o

n T

his



Lin

k

Understand

Measu

rePM


Communicate

Best Practices

Understand

People – Is your staff focused on the correct goals?

Understand Know your applications,

users, and requirements

Measure Properly measure key aspects

of application performance

Communicate Lin

k

Understand

Measu

r

ContinualService

I t


Provide relevant performance reports to management

Link Show specific business-

performance links

Communicate

reImprovement


Processes

Incident

Turn events into actionable alarms with good diagnostic capability

Process – Do you have proper procedures in place?

good diagnostic capability

Goal is to restore normal service operations as quickly as possible

Availability

Fundamental reporting of system health

Historic data determines trends

Capacity

Match costs to business needs by adding or removing resources

ServiceAssurance

Management

Capacity Management


g g

“What if” analysis and planning are key

Service Assurance

Service levels are tailored to meet business needs

Continually monitor and coordinate corrective action

Availability Management

Incident Management

Outline








NetForecast PM Benchmarks

Second annual survey of enterprises to characterize their current APM practices and assess results Surveys completed in February 2007 and July 2008y p y y

Sample size of approximately 300 enterprises both years

Survey had questions on performance management methodology along with effectiveness metrics How problems are found

Time to solve problems

Performance meeting business needs

A il bilit


Availability

Response time

Benchmark and effectiveness scores were determined for each enterprise

Best Practices Results

7

8

ss M

etrics

Best Practices51% Gain

k

Understand

Me

5

6

Perform

ance Effectivene

(0=p

oor, 10=excellent)

Gain is the slopeof the curve.

Or the return inperformance delivered(vertical axis)

Communicate

Lin

k

ea

su

re


3

4

1 2 3 4 5 6 7 8 9 10

Composite P

NetForecast Benchmark Score

(0=poor, 10=excellent)

(vertical axis)for the investment inthe benchmark(horizontal axis)


Processes Results

7

8

ss M

etrics

5

6

Perform

ance Effectivene

(0=p

oor, 10=excellent) Processes

24% Gain


3

4

1 2 3 4 5 6 7 8 9 10

Composite P

NetForecast Benchmark Score

(0=poor, 10=excellent)

The PM Model Works

Strategy

Best Practices

Processes

51% Gain

24% Gain


Architecture

Tools


Outline







Apdex Defined

Apdex is a numerical measure of user satisfaction with the performance of enterprise applications

It defines a method that converts many measurements into one number Uniform 0-1 scale

0 = no users satisfied

1 = all users satisfied


Standardized method that is a comparable metric across Applications,

Measurement approaches, and

Enterprises


How Apdex Works

Existing TaskResponse TimeMeasurement 7

1. Define T for the applicationT = Target time (satisfied-tolerating threshold)

F = 4T (tolerating-frustrated threshold)2. Define a Report Group

Frustrated

Good

Fair

Poor0.50T

1.00T

0.85T

0.94T

0.70T

ExcellentReport Group:ApplicationUser GroupTime Period

easu e e tSamples2

3

5

7

6

p p3. Extract data set from existing measurements4. Count number of samples in three zones5. Calculate the Apdex formula6. Display Apdex value showing T7. Optionally display value using quality colors


Tolerating

ApdexT=Total samples

Satisfied2+

Satisfied

Tolerating

Un

ac

ce

pta

ble

0.00T

T1

4F

6

0.91 [6]

Value T

14%

16%

Tim

e

Apdex Example

Major eCommerce site ($4B annual on-line sales)North American broadband users accessing the San Francisco data center

6%

8%

10%

12%

y o

f E

xper

ien

cin

g t

he

52% Satisfied

This site had an average response timeof 4 seconds, so it looked like all was well

But: Apdex = 0.7310 = Fair


0%

2%

4%

0 20 40 60 80 100 120Load Time of a Typical Business Page (sec)

Pro

bab

ilit

y

42% Tolerating

6% Frustrated


Apdex Methodology

Gather Agree Improve

• Measure• Gather baseline data• Test Apdex Parameters

A T

1) Work withusers to set T

2) Work with businessmanagers to define theservice objective for A

• Apdex reports & trends• Improve performance

where needed• Continual performance

& process improvement


Keys to PM Success

ProcessesBest Practices

AssuranceLink

Collaboration Dependability


Assurance

Capacity

Availability

Incident

Link

Communicate

Measure

Understand


Leveraging Apdex for Success

ProcessesBest Practices

The Apdex methodology forces organizations to achieve better PM

AssuranceLink

Collaboration Dependability

Apdex Benefit Zone


Assurance

Capacity

Availability

Incident

Link

Communicate

Measure

Understand

Apdex is a numerical measure of application performance relative to business objectives.The Apdex open standard is managed by the Apdex Alliance – see apdex.org.

The Apdex Alliance has more than 800 individual and 10 vendor members.

How Apdex Helps PM

ProcessesBest PracticesSelection of T and theApdex score is the Excellent foundation

Apdex dialogsupports collaboration

Open standardSupports dependability

AssuranceLink

Collaboration DependabilityFacilitates technologistsand non-technologistscommunications

Apdex score is thelink to business needs

Defines

ce e t ou dat oFor meaningful SLAs

Trend analysis helpsfind capacity problems


Assurance

Capacity

Availability

Incident

Link

Communicate

Measure

UnderstandForces you tounderstand yourapplications and users

Definesmeasurementtaxonomy

Discovers hardto find incidents

Can be applied toavailability analysis


Outline







What Are Best Practices?

Understand Have a good definition of which technical parameters are important

Good understanding of which applications are mission criticalg

Good understanding of which users are more important

How well the above is documented

How well these understandings are distributed within the IT organization

Measure How well the important technical parameters are measured

How well the measurements are tracked over time


How well the measurements are tracked over time

Setting critical thresholds for the important parameters

Formal processes to ensure that these measurements are carried out

Automation for efficient data gathering and correlation


What Are Best Practices? (con’t)

Communicate Provide relevant performance reports to management

Ensure that the reports a meaningful to non-IT staffg

Measurements are communicated throughout the enterprise

Distribute information on a regular basis

Don’t limit information thinking the recipient won’t understand it

Link Performance targets are agreed upon with business managers

Applications monitored are confirmed to be business critical

Performance targets are confirmed to be relevant to the business


Performance targets are confirmed to be relevant to the business

Application-level SLAs are in place

Meet with business managers periodically to review the above

Best Practices BenchmarksIndustry State of the Art

20%

25%

OverallBenchmark

Medians

10%

15%

Per

cen

t D

istr

ibu

tio

n

20072008

Medians2007: 42008: 5

State of theart improved10% in a year


0%

5%

1 2 3 4 5 6 7 8 9 10

Best Practices Benchmark


Private/Public Traffic in 2008

chm

arks

8

9

10

Private vs. Public makes no differenceAll within a tight 0.3 point range

n B

est

Pra

ctic

es B

enc

3

4

5

6

7


Med

ian

0

1

2

100/0 75/25 50/50 25/75 0/100

Private/Public RatioPublicPrivate

Impact of Enterprise Size in 2008

8

9

10

chm

arks

Enterprise size has a 2 point range

3

4

5

6

7

n B

est

Pra

ctic

es B

enc


0

1

2

<50 50-100 100-500 500-1K 1K-10K 10K-40K >40K

Med

ian

Number of Employees in the EnterpriseBigSmall


Benchmarks Compared

Understand 20072008

Communicate

Measure


0 1 2 3 4 5 6 7 8 9 10

Link

Median Best Practices Benchmark

Why Measure is Less Than Understand

Contribution to the Difference Between Understand and Measure

End-User Response Time

Server Errors

Server Query-Response Time

Availability

Server Utilization

TCP Transaction Response Time

Transactions ProcessedOverall shift of2 benchmark points


- 1 2 3 4 5


Traffic Flows by Application

WAN Performance

Bandwidth Utilization

2 benchmark pointsfrom 6 to 4 onprevious slide


Real Company Examples

UnderstandBank

HospitalityGovernment

ComputersConsumer Website Pharma

Individual Enterprise Best Practices Profiles Are Diverse

Healthcare

Communicate

Measure

Telecom Eq. Airline


0 1 2 3 4 5 6 7 8 9 10

Link


NetForecast Benchmark Service

Benchmark how well your enterprise is delivering the Best Practices and essential Processes

Strategy

Rank your benchmark scores against industry norms (database of more than 600 enterprises)

Gap Analysis shows where you Best Practices and Processes excel or are deficient

Best Practices

Processes


Workshop meeting that defines your successful performance management strategy

Tools


Thank You

More information is available at:

Free articles and reports on performance

measurement analysis and managementmeasurement, analysis, and management

are available at www.netforecast.com

how apdex helps performance management benchmarksapdex.org/documents/session308.2sevcik.pdf · how...

Documents