continuous performance monitoring of a distributed application [con4730]

Insert Picture Here

Continuous Performance Monitoring of a Distributed Application

Ashish Srivastava Principal Member of Technical StaffDiana YuryevaSenior Member of Technical Staff

The following is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Session Goal

§ Components:– Design patterns

– Tools

§ Qualities:– Continuous

– Light-weight

– Recordable

Arrive at solution for extreme performance monitoring

Session Agenda

§ Use Case

§ Software Patterns

§ Tools

§ Pitfalls and Advice

§ Q&A

About Us

§ Oracle Billing and Revenue Management Elastic Charging Engine– 100% real-time charging application

– Java

– Distributed grid– Oracle Coherence

– Oracle NoSQL

– Focus on extreme performance

Operating Conditions

§ Low latency expectations

§ Heavy system load

§ Distributed environment

§ Multi-level software stack

Monitoring Requirements

§ Detailed insight about performance– Latency

– Throughput

§ View over time§ Reporting§ Bottleneck detection§ View of system as cohesive unit

Functional

Monitoring Requirements

§ Minimal impact on processing

§ Ease of use

§ Separation of concerns

Non-Functional

Session Agenda

§ Use Case

§ Tools

§ Q&A

Approach

§ Off-the-shelf software not sufficient

§ Custom development needed– Incorporate monitoring into system

– Collect, analyze and present metrics

How do I address these requirements?

Collecting Metrics

§ Goal– Incorporate metrics collection into general processing

§ Approach– Enhance domain model with monitoring-related data structures

Problem overview

Collecting Metrics

Model of sample system

ECE Client

Network Mediation

request

―Debatch the requests―Data lookups―Apply Tariff―Save Session―Prepare Response―..

response

Collecting Metrics

Solution

ECE Client

Network Mediation

request

―Debatch the requests―Data lookups―Apply Tariff―Save Session―Prepare Response―..

response

Client Node

Processing Node

Envelope

Routing ContextPayloadTracking Context Chronicler

– TimePointsStat Reporterharvest

Envelope

Collecting Metrics

##### Elapsed time = 3600 seconds

##### Avg throughput = 20000 ops/sec

##### Avg latency = 50 ms

Result

Granular Reporting

§ Goal– I need more granular reporting of performance over time

§ Approach– Enhance reporting of collected metrics

Problem overview

Granular Reporting

Solution – data structure

Chronicler removed

―A moving reporting window―100% reporting―Sampled reporting―Stats exposed over JMX―Fixed data set for a window

Chronicler added

Granular Reporting

Solution – class diagram

Granular Reporting

§ I can see min/max/avg latency and throughput over time

§ My throughput reporting is quite good: I can see whether I had stable or erratic throughput

Result

Latency Percentile Report

§ Goal– Latencies are still not detailed enough. I need to know more than the

average/min/max latencies

– Need to guarantee that 99.999% of the requests take less than 55ms

§ Approach– Introduce range bucketing to count latencies

Problem overview

§ Pre-defined buckets of latency percentiles§ Data set does not grow. Each bucket is updated§ Multiple percentile breakdown

– End-to-end

– Server side processing

– Per batch reporting

Solution

2013-03-21 08:44:29.112 PDT INFO ##### Latency statistics based on percentiles:

Percentile: 0.1, Latency: 1ms, Total Count: 1173148

Result – printed report

Result – heat map

Method Breakdown

§ Goal– I want to measure the impact of a new method under varying load

– End-to-end latency always ON

– Minimum performance impact

§ Approach– Method annotations

– Aspect

Problem overview

Method Breakdown

Solution

public enum LabelEnum { APPLY_TARIFF,

... DEBATCH } public class ClassToBeTracked { @Track(pointLabel = LabelEnum.APPLY_TARIFF) private <ReturnObject> method(<Parameters>) { ... } }

Method Breakdown

Result – detailed breakdown report

2013-07-15 16:29:24.953 PDT Chronicler Breakdown: DEBATCH -> 64149 nanoseconds LOOKUP_DATA -> 1056748 nanoseconds APPLY_TARIFF -> 99994 nanoseconds SAVE_SESSION -> 12989 nanoseconds PREPARE_RESPONSE -> 15998 nanoseconds

Session Agenda

§ Use Case

§ Tools

§ Q&A

Storing and Presenting Metrics

§ Goal– I collect detailed performance metrics, but I need to report them too

– I need a tool which stores these metrics and presents them in a unified view

§ Approach– Create monitoring dashboard

– Technologies: JRDS,RRD and in-house development

Problem overview

Result – monitoring dashboard

Configuration: Topology 24 servers Throughput 20000 ops/sec Duration 10 hrs

§ Graphical§ Supports various metrics

– Application-specific

– Machine-specific

– JVM-specific

§ Consolidated view– All graphs on one page

Solution qualities

§ Easy to use– Collects and saves data automatically

§ Easy to share– Includes configuration for future references

– Send links to web pages

– Print page as PDF

Solution qualities

§ Stores data without losing precision§ Supports drilling down§ Light-weight§ Customizable

Solution qualities

Session Agenda

§ Use Case

§ Tools

§ Q&A

Pitfalls and Advice

§ Distributed system monitoring != Single JVM monitoring– Consolidated view is critical

§ Consistency of tools across team is important– Same language across development, QE and Performance teams saves

§ Solution should enable you to be agile– Run monitoring on laptop AND realistic setup

Take into consideration

Pitfalls and Advice

§ These hide problems– Averaging

– Sampling

§ GC has big impact, so include it in your metrics§ Watch our for processes sharing the same host§ Always run long-duration tests

Some things to pay attention to

Session Summary

Detailed insight about performance– Latency

– Throughput

View over time Reporting Bottleneck detection View of system as cohesive unit

Let's see how we addressed original requirements

Session Summary

– Throughput

Session Summary

– Throughput

Session Summary

– Throughput

Session Summary

– Throughput

Session Summary

– Throughput

Session Agenda

§ Use Case

§ Tools

§ Q&A

§ ECE– http://www.oracle.com/us/products/applications/communications/elastic-

charging-engine

§ JRDS

– http://www.jrds.fr

continuous performance monitoring of a distributed application [con4730]

oracle andor

oracle billing

rights reser

sole discretion of oracle

throughput reporting

granular reporting goal

batch reporting solution

moving reporting window

Technology

main effects screening: a distributed continuous quality...

operating manual€¦ · by default, continuous monitoring...

distributed computing in sensor systems dcoss 2019 ·...

continuous monitoring deck

continuous monitoring in a risk management …...agenda •...

distributed context monitoring for the adaptation of...

fundamentals of continuous monitoring - nist · pdf...

continuous emission monitoring system - rayten · ⊙...

continuous auditing en continuous monitoring: levert het...

continuous monitoring and continuous auditing

distributed pv monitoring

continuous glucose monitoring

continuous auditing & continuous monitoring in a...

electronic reporting of continuous emission monitoring...

the lightweight distributed metric service: a scalable...

distributed intrusion monitoring system with fiber link...

continuous transaction monitoring

distributed continuous quality assurance process

continuous auditing & continuous monitoring...il continuous...

continuous monitoring · “continuous monitoring and...