simulating production beforehand mining statistics afterward

Simulating Production Beforehand, Mining Statistics After

Mario AndreoliSolution & Infrastructure Architect

Performance Analysis , Testing & Tuning

Simulating Production BeforehandMining Statistics Afterward

May 28 2013

TASSQ Presentation

@MarioAndreoli

LinkedIn.com/in/MarioAndreoli

Mario@acs1.com

28 May 2013| 2

� Introduction

� Challenge - Certifying performance on a mission critical financial application

� Solution to the “Performance Certification” challenge

� Inception – understand what needs to be done

� Elaboration – explain how the solution will work

� Construction – build the test components and a data factory

� Test Execution and Test Results

� Additional Benefits

� Where are we today?

Agenda

28 May 2013| 3

� Historically a Developer, Solution Architect

� Strengths (certifications) in RUP, OOAD, J2EE

� “Reverse Engineering” strengths

� Created a bridge between Technical & Testing

� Teams consist of T/BSAs, Developers, Testers

� Certified the performance of a $250 million financial application

— not in the negative “Gartner Statistic” group

Introduction

28 May 2013| 4

Questions:

� How will it perform in Production?

— Is the Hardware sized appropriately?

— Is the Software tuned efficiently?

—Will the solution handle the anticipated volume?

—Are we confident transaction processing will have a significant percentage of straight through processing?

—Can the communication infrastructure handle the load?

—Will the end users have an acceptable experience?

—We can’t wait for Production to catch issues!

� How do I build confidence for the Stakeholders?

— Top down support is crucial. How do we get it?

— How can we build a plan that Stakeholders buy into?

General Challenges

Answers:

� How will it perform in Production?

—Design and create a working model thataccurately simulates the solution before it is in Production

� How do I build confidence for the Stakeholders?

—Provide tools Stakeholders can use to be part of the process and monitor business healththroughout and after

28 May 2013| 5

We Can Address It

28 May 2013| 6

Technical Challenges:

� Vendor Application

— no source access, little architectural information, no data model

� Complex Workflow Engine

— Senior RBT resource estimated an extremely high possible number of paths (test cases)

� 40+ interfaces / 250+ concurrent users

� Multiple functional streams

— Each with their own critical and complex requirements

� Global access (interfaces & users)

� Millions of varying types of messages daily

— exponentially more transactions

Inception

28 May 2013| 7

Financial Challenges:

� Market close deadlines

� Financial critical transactions

� Vital upstream and downstream SLAs

� This has to work, $000,000,000’s depend on it

— We do not want to be (negatively) in the newspaper the next day

� Manual functional testing of all production (financial) data scenarios almost impossible

� Distributed geographical markets

Inception

28 May 2013| 8

Approach:

� Senior QA resource team came to a conclusion that in addition to a good test strategy, a different approach was needed

� A strong technical resource with ‘out of the box’ thinking

� Iterative with quick and frequent deliveries

� Built in layers – with agile yet controlled processing

� Quick iterations and deployments to:

— start the testing early…learn early

— easily repetitive (building the regression suite),

— no (or minimal) manual interventiontests

statistics

learning

decisions

Resolution

Inception

28 May 2013| 9

Iterative Approach:

Inception

� Build layers of tests that can all talk to each other and be used in harmony to do full performance, load, regression testing etc.

28 May 2013| 10

White-Boxing a Black-Box:

� Understand business concepts?

� Vendor + Business Stream

� Flows, Begin-Status, End-Status, Rules

� Sample Messages (in/out)

� Understand the architecture through Reverse Engineering

� Structural Diagrams:

� Deployment, Package, Component

� Hardware, Software, Communication

� Interfaces and Message Transportation

� Behaviour Diagrams:

� Activity, State, Sequence

� Major Flows for System Behaviour

� Guide other teams/groups

� Functional coverage

� Code coverage

Elaboration

Performance Test Scope:

Elaboration

Performance

TestingTypes

Overall

Program

Components

Dependencies

Endurance

Repetitive

Stress

H/W, Network,

Database

(optional)

Timing to

Volumes and SLA

(Performance NFR)

FinancialApp User

Interface

Response Time

Interface:

SI Processing Time

Batch Cyle Time

FinancialApp

User Interface

Interface

Processing

Volume

Importance

EnvironmentFunctional

Testing

Tuning

ESB/FTP Transportation

Failover & DR

Performance

External System

Performance

Automation

User Interface

Loadrunner

Interface

Harness

Schedule

Management

Database

(Flashback)

Stability over long periods

H/W Resource Management

Finding Ceiling of

Architectural Component

Realistic Production

Volumes & Timelines

Anticipated growth

Exact same execution,

repeated to determine

transaction accuracy

Stream:

Trade Processing

Entitlement

Integration

Corporate Actions

User Types:

Inquiry (Detail, Index, Wildcard)

Update

40+ Interfaces

Day in the Life

FinancialApp

Database:

--Start

--Growth

IntradayEOD/BOD

FinancialApp version

Core version

FinancialApp Deployment Footprint

All other infrastructure components

Technical Test Team

Functional Test Team

Vendor

IT Support

Environment Coordinator

LoadRunner

Infrastructure Resources (DBA,

Capacity & Planning, Websphere etc.)

Geographic:

Europe users

Entitlement &

Position

Generation

Out of Scope

Supporting

Resources

“A Day In

The Life”

All Overall Program

components running

together over a 24 hour

period

Measurements

Prioritization

Statistics

Monitoring

28 May 2013| 12

Business Concepts: Prioritize & Understand� Select your high volume, critical business flows to understand and build

harnesses for

� The scenario below is 1of 4 built (there were 10 scenarios in total, but the 4 selected consisted of 90% of the overall volume)

Elaboration

28 May 2013| 13

Architecture � We take each scenario and perform a volume analysis

� Map volume per time period

� Analyze peaks

� Confirm your findings with business resources

Elaboration

28 May 2013| 14

Hardware: Technical Understanding & Architecture

Elaboration

Reverse engineer the system and create the diagrams to help you understand the application:

Deployment: The physical layout of servers/nodes and their roles

It may also depict any load balancing strategies

Communication amongst servers/nodes/components should be depicted as well

28 May 2013| 15

Software: Technical Understanding & Architecture

Elaboration

Reverse engineer the system and create the diagrams to help you understand the application:

Component: Determine the ‘pieces’ of the system and map the relationships between these different ‘pieces’

28 May 2013| 16

Interface Transportation & Processing� Understand the major points and paths whereby messages will flow in and out

of the application

� Understand scope, which caveats you need to state in test results

� Point of binding for test harness depends on purpose, capability, constraints

Elaboration

28 May 2013| 17

Tying It All Together:A Day in the Life (DIL) of the Financial Application:

� Map all business layers on a 24 hour clock

� Each layer can have many components running to execute the pertinent tests

Elaboration

24 Hour Clock (EST)

Activity 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00

Batch Processing Intraday Batches EOD Batches

Online - Europe

Online - North America

Trade Transactions

Annoucement Interfaces

Other Interfaces

Entitlements Processing

BI Data Replication

Application

Component

1..* *

DIL Profile

28 May 2013| 18

Tying It All Together:A Day in the Life of the Financial Application:

� At this point we have a good understanding of critical:

— functions, flows, volumes, processesand timing

� We now need a profile that unifies all our data and understanding

� Circulate this profile for agreement and sign off

� This will be our contract starting point for our physical design, prototyping and construction

� If we can emulate a 24 hour period, we now have a baseline to test with

Elaboration

28 May 2013| 19

Tying It All Together:A Day in the Life of the Financial Application:

� A component:

— can be a user scenario, interface feed, business process etc.

— is its own “thread” in the final simulator

— can be adjusted:

� On/Off

� Volume increase/decrease

� Duration time

� Period spike/lull

� Volume over Duration

—will be recordingstatistics and pertinent data

Elaboration

DIL Profile

28 May 2013| 20

Day in the Life - Profile:

Elaboration

24 hour clock, broken up in 30 minute intervals

Interfaces, Business Processes,

GUI Lo

ad Profile

24 Hour Clock 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00GUI Load Time Line 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00GUI Load - Europe 15 15 15 15 15 15 15 15 15 15

GUI Load - North America

20 20 20 20 20 20 20 20 20 20

GUI Load - North America 2

200 200 200 200 200 200 200 200 200 200

Entitlement S

Entitlement S-V 2750 2750 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

28 May 2013| 21

Day in the Life - Profile:

Elaboration

Map your volume pattern over a 24 hour clock

28 May 2013| 22

Test Harness Component Pattern� With our requirements solidified in the Day in Life profile, we now can start the

development of the Test Harness Components

� Each component:

— will do one thing and will do it well

— will follow the same general patternand specialize where needed

— Is responsible for managing its behaviour and parameters but will communicate with others when necessary

— will take the DIL profile as its input

— will persist statistics and data for analysis as its output

Construction

Application

Component

1..* *

DIL Profile

28 May 2013| 23

Test Harness Component� Physical implementation of a component depends on its role

� The example below shows an interface processing Test Harness component alongside a user load scenario

Construction

28 May 2013| 24

Data Factory� Messages used for testing are provided via one of three methods:

— Existing Production Data

— Data on any of the other systems on the network

— Custom message production based on: Rules & Templates

� Used by business users and functional testers to review data and use for their own tests and investigation

Construction

28 May 2013| 25

Statistics Monitor� During a test, statistics of the execution are critical in determining results and

effectiveness of each component and overall test success

� Transactions may be processed, but proper test execution and code coverage can only be confirmed by end state of the transactions

� Throughout the test, ongoing variances in expected results need to be proactively pushed to the respective parties so as to not compromise the overall test.

— Cost of running a test vs. a restart dictates level of granularity for notification

Construction

28 May 2013| 26

Statistics Monitor – Application Health� Application health comes from two distinct locations:

— Internal Application Health

— External Infrastructure

� Internal Application Health

— Transaction status (per business stream), user load, messages in/out etc.

— Anything critical to predict variances in business trends

— Watermarks can be set up for proactive notifications

� External Infrastructure

— CPU, disk, memory, network, queues, database health

— Prognosis, Dynatrace, Groundworks etc.

Test Execution

28 May 2013| 27

Test Execution

Statistics Monitor – Internal Health� Dashboard with all statistics

� historical comparisons and trends

Functional A

Search CriteriaVersion,Feedback

Daily breakdown of data

History by hour

Graphical History

28 May 2013| 28

Success is Difficult, Iterations Make it Achievable

Test Results

In this scenario, a test harness component was injecting Trade messages

The application settles the trades based on business rules and sends a confirmation message when complete

Above: “message injected” (blue line) versus “messages processed” (pink line) had a significant delay due to poor processing performance.

Right: eight iterations later (with fine tuning and vendor fixes) Trade settlement met NFR SLA times

28 May 2013| 29

Other Significant Benefits

Organizational & Stream Readiness� Organizational Readiness wanted to give the operational support team an

opportunity to

— experience a “production day” before going live

— review the system capabilities, identify key risk areas and an indication of STP rates

— The business unit can carry out their day-to -ay work in an accurate and timely fashion

� Certain business streams wanted to understand what rate of “Straight Through Entry and Processing” (STE/P) would be achieved for the new system and compare that to the existing system

— A specific date was used as the baseline

— Flows and message types were chosen and collected from production for that date

— Once the test was executed statistics were compared and led to rule tweaks (in the new system) and re-runs to match STE/P rates

� Production Support

— Able to recreate production scenarios using the Test Harness

— Step through transactions as they happened in Production to debug issues

Production

� Application has been in production since April 1 2013

� Hardware, environment, user load and infrastructure all operating as predicted

� All NFR SLAs being met

� Final piece being added is a reporting module on the Statistics database

— Custom reports driven by users

� Utilities and tools promoted to Production and used by the executive team and production support to monitor application health

� A completely reusable set of processes and components adaptable to a varying number of systems (small to large)

28 May 2013| 30

Where are we today?

28 May 2013| 31

Thank You

Contact Information

� Twitter: @MarioAndreoli

� LinkedIn.com/in/MarioAndreoli

� Mario@acs1.com

simulating production beforehand mining statistics afterward

Documents

meeting starts 7:15 pm, come along for dinner beforehand

adding modularity afterward with embedded osgi

simulating avian wingbeats

order birthday celebration product beforehand

simulating a satellite

afterward background information - the little...

simulating the interferometer

afterward summary - the little...

simulating queuing systems

day-and-night closed-loop insulin delivery in a broad...

simulating quadriceps muscle atrophy and...

modelling and simulating rf sensor systems -...

afterward: making meaning after a frightening near-death...

simulating extended reproduction

simulating urban transformation

simulating microelectromechanical systems

afterward by martin delaney, acceptable risks by jonathan...

simulating patient safety

simulating thermal conductivity

la tomatinala tomatina - the most exciting and...