beyond monitoring - d1o3t0kvuxq0hm.cloudfront.net · beyond monitoring delete & insert headshot...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Performance Prediction

Using RETIT & Dynatrace App Mon

Beyond Monitoring

Delete &

Insert

Headshot

here

Delete &

Insert

Headshot

here

2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE2

Agenda

Performance Prediction

Use cases, Benefits and examples

Demo


Performance prediction

Take your Continuous delivery to the next level.


Performance Modeling - Why?

A new version of your application is on the way:

• … review realtime application performance.

• … evaluate realtime performance metrics.

• … deep dive into single components of your environment.

Enterprise ApplicationHardwareSoftware

Measurements Measurement Data


Performance Modeling - Why?

Capacity

Planning

Performance

Analysis


Measurement Measurement Data

Capacity

Management

What happens if…

• … you change yur deployment topology?

• … you migrate to a different hardware environment?

• … the workload changes?

• … you reduce the number of CPU cores?


Performance Modeling - What?

Performance

Model

Resource

Demand

Software

Architecture

Hardware

Environment

Workload

Simulation

Response Time

Throughput

Resource

Utilization


A Brief History of Performance Modeling

Queuing Theory(Erlang, ~1906)

Queuing Networks (Jackson, ~1963)

LayeredQueuing Networks (Neilson / Woodside, ~1995)

Architecture-Level Performance Models (Reussner / Becker ~2009)

w

t

S

λ

Arriving

Jobs

Departing

Jobs

Queue Service

Station

1

4

CPU

HDD

Application Server

Arriving

Transactions

Completed

Transactions

HTTPRequest

[Z=2 s]

User

{m=1200}

WebService

[s=15 ms]

HTTP Server

Thread

{m=50}

Purchase

[s=80 ms]

AppServer

Thread

{m=25}

Browse

[s=20 ms]

(y=1)

(y=0.6) (y=0.4)

User Device

{m=1200}

User Device

{m=1200}

HTTP Server

Machine

HTTP Server

Machine

AppServer

Machine

{m=2}

AppServer

Machine

{m=2}

Repository Model

Resource Environment

System Model

Allocation Model

Usage Model

Pa

llad

io C

om

po

nen

tM

od

el (P

CM

)

Foundation for tools such

as CA Capacity Manager

(was: Hyperformix) or

Sumarian Capacity Planner

Foundation for tools such

as Performance

Assurance‘s ePASA

Foundation for

RETIT solutions


Performance Modeling – How?

Knowledge

Generation

Capacity

Planning

Performance

Analysis


Performance

Model Repository

Measurement

Modeling

Simulation

Measurement Data

Model Parameter

Simulation Results

Capacity

Management


Performance Modeling – When?

OpsDev

Implementation

Operations

1 4.5 10 27.5 50

500

0

100

200

300

400

500

600

Rel

aive

cost

fact

or

(mea

n)*

Fix performance-related bugs earlier

and for less costs by managing

performance knowledge using models.

Source: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

*The cost factor for fixing a performance-related problem is normalized in

the different phases relative to the cost of fixing a defect in the requirements phase.

Extrapolate your (load/performance) test

results and reduce labor, machinery and

license costs by reducing the amount tests.

Detect performance change in every version

created in a continuous delivery pipeline without

the need for expensive performance tests.

Provide models along with your

application binaries to simplify

capacity planning activities.

Right-size your environment for seasonal peaks or to

reduce license / operating cost (e.g., reduce the

amount of cloud instances).


Dynatrace-Compatible RETIT Solutions

• RETIT Capacity Manager (RCM)

• Performance Modeling Environment

• Connect to any of your Dynatrace deployments to

generate performance models

• “What-If” scenario simulations:

• Hardware changes

• Workload changes

• Software architectural changes


Dynatrace-Compatible RETIT Solutions

• RETIT Continuous Delivery (RCD)

• Plugin for the Jenkins CI server

• Detects performance changes automatically

• Uses Dynatrace data collected in previous test phases

(e.g., acceptance tests) to generate models

• Can evaluate performance for multiple hardware

environments and workloads


Agenda

Performance Modeling


Demo


Use Cases – Design Time Performance Evaluations

Implementation

Operations

Existing

Component 1 Existing

Component …

Existing

Component N

New

Application

Design

Performance-

relevant aspects

known and depicted

in performance models

Structure and

behaviour

defined in design

models

e.g., minimum

response time –

even you assume

that a new

application

consumes no

time


Benefits – Design Time Performance Evaluations

1 4.5 1027.5

50

500

0

100

200

300

400

500

600

Requirements Design Implementation Testing Deployment Operations

Rel

ativ

e co

stfa

cto

r(m

ean

)*

Fix performance problems in your software architecture earlier with a lot less cost than during test, deployment or operations!

Source: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

*The cost factor for fixing a performance-related problem is normalized in the

different phases relative to the cost of fixing a defect in the requirements phase.

Implementation

Operations


Real Example - Design Time Perf. Evaluations

• Evaluating Performance in a Service-oriented Architecture (SOA)

Implementation

Operations

• Service-Consumers:

• Process-oriented user interfaces

• Orchestrated by a BPM Engine

• Service-Providers:

• Common data sources and application services

Enterprise Service Bus (ESB)

Business

Dev

Ops




Implementation

Operations






• What happens if we introduce/automate new business processes?• Can you achieve the required response time and business process lead

time goals?• How much does it cost to increase the IT system performance to improve

the business process lead times?

Business

Dev

Ops




Implementation

Operations






Business

Dev

Ops

• Can we achieve the desired performance and process lead

time goals using our existing service-level agreements?

• Which service needs to improve most in order to achieve the

business goals?

• Should we negotiate new SLAs with multiple service proviers

and ask them to improve their performance slidely or with a

few providers but as for more radical changes?




Implementation

Operations






• How much additional load will new business processes generate? • Will the existing systems be able to handle the load?• Do we need to increase our capacity?• Who pays for the additional capacity?

Business

Dev

Ops




Implementation

Operations






Business

Dev

Ops

APM Data

APM Data

Business

process

descriptions

Software

designs for

new processes




Implementation

Operations






Business

Dev

Ops

• You should talk toservice provider XYZ!

• Your workload will increase by X!• You should buy … new servers!

• You can achieve the desiredbusiness process lead times but itwill cost you …!



Modeling improves the collaboration of all parties involved in thesoftware lifecycle (Business, Dev and Ops)!

Business has a level of granularity (business processes) which easesthe communication with the IT department

Service consumers (Dev) can better estimate the expected responsetimes for new business processes

Service providers (Ops) have early access to workloadinformation when new business processes are released

Implementation

Operations


Use Cases – Model-based Evaluations in CD

Implementation

Operations

Evaluate the performance impact of feature additions and bugfixes

For multiple hardware environments and workloads

Without the need to own corresponding test systems!

Deployment Pipeline in a

Continuous Delivery Process

Model-based Performance

Change Detection

Commit

Stage

Automated

Acceptance

Test

Manual

TestingRelease

Developer

checks in

Notify Developer about Performance Change


Benefits – Model-based Evaluations in CD

Implementation

Operations

Ensure that no version gets released with performance regressions

Leverage cost-benefits of fixing performance problems early in thedevelopment process

Increase the performance awareness of developers by immediate feedback on check-ins

Avoid the need to setup and prepare load/performance testenvironments for each and every project

Leverage your existing acceptance/regression testinginvestments for performance evaluations


• A customer (software vendor) has 5 major enterprise applications (EA):

• 1 CI system per EA (Jenkins)

• 5 customer scenarios (workload/hardware environment combinations) should be tested for each new build

• A small performance test environment costs 10 k € / year

• 250.000 € / year

• RETIT Continuous Delivery (RCD) can realise this scenario withone license per EA!

• Saves the huge investment in test environments

Real Example – Model-based Evaluations in CD

Implementation

Operations


Use Cases – Integrating Load Tests w/ Predictions

Implementation

Operations

Setup testenvironment

Define testscenarios

Execute testsAnalyzeresults

Enterprise Applications HardwareSoftware


0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

600 Users 800 Users 1000 Users

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT


0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT


0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

Generate model and

predict performance for

multiple scenarios


Benefits – Integrating Load Tests w/ Predictions

Save costs by reducing the amount of load/performance tests

Increase the coverage of your tests

Evaluate scenarios without buying the corresponding hardware

Easily grow the coverage as the deployment count of yourapplication increases

Implementation

Operations


• Save costs by reducing the amount of load/performancetests• Effort for load tests (real customer example, incl. script development, test setup,

execution and result analysis):

• Small scale: 13 person days (PD), medium scale 23 PD, large scale: 41 PD

• Replacing one medium or large scale test by a small scale test with predictionssaves between 8 and 26 PD (assuming 2 PD for predictions)

Real Example – Integrating LT with Predictions

Implementation

Operations

0

100

200

300

400

500

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Savi

ngs

in

per

son

day

s(P

D)

Number of performance tests

PD savings when replacing large scale tests withsmall scale tests and predictions

PD savings when replacing medium scale testswith small scale tests and predictions


Use Cases – Model-based Capacity Management

Implementation

Operations



0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT


0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT


0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

0,00 ms

20,00 ms

40,00 ms

60,00 ms

80,00 ms

100,00 ms

120,00 ms

140,00 ms

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT

B MRT

B SRT

M MRT

M SRT

P MRT

P SRT


0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9

Diagrammtitel

Throughput

Response

Time

Resource

Utilization

Generate model based on

production data and predict

performance for future scenarios.

What happens if we

migrate to AWS?

How much capacity is required if

we have ten-times more users

during Christmas season?

What happens if we change

our deployment topology?


Right-size your environments to pay only for what you really need

Avoid the need to setup expensive test environments to evaluatechanges

Reduce risk for hardware environment (e.g., cloud) migrations

Reduce the time for capacity management activities

Increased accuracy as the simulations avoid the need forlinear assumptions

Benefits – Model-based Capacity Management

Implementation

Operations


10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

12:00 12:15 12:30 12:45 13:00 13:15 13:30 13:45 14:00 14:15 14:30 14:45

€/MWh

time

EPEX SPOT INTRADAY AUCTION - 2014-12-11

pi

Smart Grid Capacity Planing for several millionhouseholds

Real Example – Model-based CM

maximum 15 min

Implementation

Operations


Agenda

Performance Modeling


Demo


RETIT Solution Demo

• Demo


• What did we learn today?

• Performance modeling…

Summary

Improves cross-

team collaboration

…by integrating multiple

data sources

Increases performance

awareness

… through immediate feedback

during development.

Extends test

coverage

… by allowing you to test more

workloads and hardware

environments.

Saves cost

… by taking the guess-work

out of capacity planning

activities.


Andreas Brunnert

[email protected]

Peter [email protected]


Backup Evaluation Results


Beispiel – Erhebung der Daten

IBM System X3755M3

4 AMD Opteron 6172

12 cores and 2,1 GHz

256 GB RAM

VMWare ESXi 5.1.0

Benchmark Driver

Faban Harness

16 Virtual CPUs

8 GB RAM

openSuse 12.3 64bit

UI Server

GlassFish AS 4.0

2 CPUs Enabled

10 GB Heap

openSuse 12.2 64bit

Dynatrace / RETIT APM

WS Server

JBoss 7.1.1

6 CPUs Enabled

6 GB Heap

openSuse 12.2 64bit

Dynatrace / RETIT APM

DB Server

PostgreSQL 9.2.7

4 CPUs Enabled

96 GB RAM

openSuse 12.3 64bit

SAR

IBM System X3755M3

4 AMD Opteron 6172


256 GB RAM

IBM System X3755M3

4 AMD Opteron 6172


256 GB RAM

IBM System X3550

2 Intel Xeon

6 cores and 2,4 GHz

96 GB RAM


Beispiel - Ergebnisse

• Measurement

• Simulations using data collected by dynatrace + SAR

• Simulations using data collected by RETIT APM + SAR

IBM System X3755M3

4 AMD Opteron 6172


256 GB RAM

VMWare ESXi 5.1.0

Benchmark Driver

Faban Harness

16 Virtual CPUs

8 GB RAM

openSuse 12.3 64bit


Beispiel - Ergebnisse

UI Server

GlassFish AS 4.0

2 CPU Enabled

10 GB Heap

openSuse 12.2 64bit

WS Server

JBoss 7.1.1

6 CPU Enabled

6 GB Heap

openSuse 12.2 64bit

DB Server

PostgreSQL 9.2.7

4 CPU Enabled

96 GB RAM

openSuse 12.3 64bit

SAR

IBM System X3755M3

4 AMD Opteron 6172


256 GB RAM

IBM System X3755M3

4 AMD Opteron 6172


256 GB RAM

IBM System X3550

2 Intel Xeon

6 cores and 2,4 GHz

96 GB RAM

RETIT APM RETIT APM


• Ergebnisse -> VM zu realer Hardware

Modellgenerierung mit RCM & RETIT APM


• Ergebnisse -> VM zu realer Hardware



• Änderungen des Nutzerverhaltens:

• Interarrival Anpassen

• Variation der Transaktionsverteilung

• Neue Usage Szenarien anlegen

• z.B. Closed Workloads



• Variation der Nutzer, ihrer Verteilung und der Think Time / Arrival Rate



• Beispiele – AMD Opteron 16 Cores -> Meas. Vs. Simulation (SPECjEnterprise2010 – Java EE 5.0 Benchmark)

Höhere Skalierungen Testen


• Beispiele – Intel Xeon 8 Cores (adapted from AMD) -> Meas. Vs. Simulation (SPECjEnterprise2010 –Java EE 5.0 Benchmark)

Höhere Skalierungen Testen


Persona Icons – For your use

Recolor these icons as

needed to match

Dynatrace brand

colors. PictureTools/Format/Color

Customer Conversion

Fanatic

for

Digital Business

Owners

Deployment-obsessed

Releasearaptor

for

Development

War Room

Peacemaker

for

Operations

53 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

beyond monitoring - d1o3t0kvuxq0hm.cloudfront.net · beyond monitoring delete & insert headshot...

Documents