Establishing an Application Performance Management

Framework

Philip PoochPerformance Test Specialist

Where are we now?

• Still Performance Test Specialists

• Still do performance testing

• Risk-based approach is working well

• Our processes are mature

• More ‘take up’, earlier engagement

• Have become ‘consultants’ as well as ‘testers’

• Now moved into a Systems Management Team

• Involved in establishing a Business Process Monitoring & Alerting service for critical business applications

Agenda

• Application Performance Management

– A Definition

– Core Elements

• Our current progress and challenges

• Removing artificial constraints

• A way to proceed

• Changing mindsets

• Questions

Application Performance Management– a definition

A business-centric approach

for measuring application performance with the objective of maintaining it

at a level that meets or exceeds

end-user expectations

Core elements of APM

• Performance Risk Analysis

• Business Objectives & End-user Requirements

• Governance (Planning, Process, Documentation)

• Business Process Monitoring

• Application Status Visualization

• Real-time and Historical Data Analysis

• Availability & Response Time Exception Handling

– Alerting

– Analysis/Diagnosis

– Remediation

• Reporting

Our Current Progress

Critical Applications List

Implementation process documented

Monitoring & Alerting agreement for each application

Automated business processes for each application

Basic application status displayed on NOC dashboard

Monitoring data stored centrally and available via intranet

Exception Handling

Availability and Response time alerting on NOC display

Analysis/Diagnosis by App/Infra Support Areas

Remediation by App/Infra Support Areas

Automatic daily reports published to the intranet

Challenges

• Can become a ‘blame game’ in silo’ed application and infrastructure teams

• Requirements and targets set by IT rather than the end-user

• Resourcing and Application owner commitment

• Script creation & maintenance an intensive ongoing process

• Tool limitations, vendor-driven, fragmentation of toolsets

• Product focus to the neglect of Process and People

• Ownership of exception handling

• Focus on changing the statistics rather than improving the customer experience

Removing Artificial Constraints

Performance TestingApp Risk Assessment

Test Requirements

Test Planning & Documentation

Script Creation & Execution

Test Status During Execution

Test Results & Analysis

Defect Handling & Resolution

Test Summary Reporting

Removing Artificial Constraints

Performance Testing Production MonitoringApp Risk Assessment App Grading & Prioritisation

Test Requirements Monitoring Requirements

Test Planning & Documentation Monitoring & Alerting Agreement

Script Creation & Execution Business Process Automation

Test Status During Execution Application Status (dashboard)

Test Results & Analysis Real & Historical Data Analysis

Defect Handling & Resolution Event Handling & Resolution

Test Summary Reporting Availability & Performance Reporting

Removing Artificial Constraints

Application Performance ManagementApplication Performance Risk Assessment

Application Performance & Availability Requirements

Test Planning Success Criteria Monitoring Agreement

Business Process Automation (Performance Measuring)

Test Performance Status/Dashboard Prod Performance

Test Results Data Store/Analysis Real-time & Historical

Application Exception Handling & Resolution

Test Summary Reporting Availability & Performance

Business Process Automation

D E V E L O P M E N T

A p p lic a tio n fu n c tio n a lity d e v e lo p e d

P e rfo rm a n c e m o n ito rfu n c tio n a lity d e v e lo p e d .

* E m b e d d e d w ith in a p p lic a tio n c o d e ,* T h ird -p a r ty m o n ito r in g p ro c e sse s ,

* P a s s iv e m o n ito rs ,* C u s to m ise d te s t s c r ip ts ,

* S c r ip ts fo r a c tiv em o n ito r in g

P e r fo rm a n c e m o n ito r u se d in re a l-w o r ld c o n d itio n s

P e r fo rm a n c e s ta ts re p o r te da n d c o m p a re d a g a in s t S L A s

P e rfo rm a n c e s ta ts u se d to v e r ifyp e r fo rm a n c e te s t a c c u ra c y

S ta ts u se d to fo rm b a se lin e fo r te s t in g n e x t re le a se

E x e rc is ep e r fo rm a n c e

m o n ito r fu n c tio n a lity u n d e rs im u la te d lo a d le v e ls

P e r fo rm a n c e s ta ts re p o r te d a n dc o m p a re d a g a in s t s im u la te d lo a d

ta rg e ts

P e r fo rm a n c e m o n ito rfu n c tio n a lity te s te d w ith

m in im a l lo a d

P e rfo rm a n c e s ta ts re p o r te da n d c o m p a re d a g a in s t

m in im a l lo a d ta rg e t

P R O D U C T IO N

A p p lic a tio n fu n c tio n a lity u se d in re a l-w o r ld c o n d itio n s

F U N C T IO N A L T E S T IN G

A p p lic a tio n fu n c tio n a lity te s te d w ith m in im a l lo a d

A p p lic a tio n fu n c tio n a litye x e rc ise d u n d e r s im u la te d lo a d le v e ls

P E R F O R M A N C E / S O C IA B IL IT YT E S T IN G

Package anddeploy under

change control

Package anddeploy under

change control

Package anddeploy under

change control

Retrieve fromProd under

change control

A Way to Proceed– Establish a Framework

A Way to Proceed (Production)- First ‘Function’ then ‘Toolset’

TestPartner

Client Vantage

Administrator

Agent-less

Monitoring

Vantage

NetIQ

HP OpenView

HP OpenView

Client Vantage

Administrator

Vantage

Visualiser

Vantage View

Vantage View

HP OpenView

HP OpenView

HP OpenView

Client Vantage

Administrator

Client Vantage

Administrator

MOM

HP OpenView

Vantage ?

?

A Way to Proceed (Production)- First ‘Function’ then ‘Toolset’

Agent-less

Monitoring

Vantage

Client Vantage

Administrator

Vantage

Visualiser

Vantage View

Vantage View

Vantage

Tivoli

TestPartnerClient Vantage

Administrator

Vantage

MOM

Client Vantage

Administrator

Client Vantage

Administrator

Tivoli

Tivoli

Tivoli

Tivoli

Tivoli

?

?

A Way to Proceed (Production)- First ‘Function’ then ‘Toolset’

Real User

Monitor

SiteScope

Business

Process

Monitor

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

MOM

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

Business

Availability

Centre

?

?

A Way to Proceed (Pre-Production)

LoadRunner

Vuser Scripts

Performance

Centre

Performance

Centre

Performance

Centre

Performance

Centre

Analysis Deep

Diagnostics

Performance

Centre

LoadRunner

Monitors

Performance

Centre

Performance

Centre

Performance

Centre

A Way to Proceed

• Establish an APM team independent of IT App Owners, and IT Infrastructure Owners.

• Provide a pre & post-production application performance measurement service.

• Define clear service boundaries, roles & responsibilities.

• Focus on Process & Function over Product.

• Move accountability for hardware/tool software setup & maintenance to an Infrastructure Support group.

• Determine constraints, and work within them, but keep the goal in mind.

Changing Mindsets- APM is:

• First about business outcomes - not technical solutions.

• About Process & People - not just Product.

• A specialist discipline that requires adequate resources.

• First Strategic, then Operational.

• About business disruption management – not component downtime.

• About improving the customer experience – not an IT blame game.

• Only going to work when the business value of establishing it is perceived.

Questions

Questions?

Contact:Philip Pooch - Performance Test SpecialistInfrastructure ServicesSuncorp IT

e-mail: philip.pooch@suncorp.com.au