Download - Klein - Measuring End-to-End Response Time through ALL the Tiers

www.hotsos.com Slide 1Copyright © 1999–2005 by Hotsos Enterprises, Ltd.

Measuring End-to-End Response TimeThrough ALL The Tiers

[email protected] March 2005


Agenda

• A Case Study• Lessons Learned• Generalizing a Method for End to End Performance

Optimization


A Case StudyClient XYZ

• Insurance company• Custom application

– under development for two years– nighttime batch load of transaction data– daytime online edits, reviews, approvals of transaction data– in “pilot production” phase for 10% of total online user base

• some users in geographically distributed corporate offices• other users work from home via dialup or broadband VPN

• Online response times unacceptable – “everything is slow”– editors and reviewers are clerical and paid by “piece work”

• “Please help us fix our Oracle Database”


You should diagnose Oracle performance problems like you would diagnose any other performance problem.

1. “What’s the most important thing to be working on?”– Business priority

2. “Where did the system spend my time?”– Resource profile– Which SQL is responsible?– Which SQL to blame for time spent between db calls?

3. “What’s the most effective way to fix it?”– Eliminate waste, eliminate competition– Stop when “fixing” isn’t worth it

4. Repeat

Source: Millsap, C.; Holt, J. 2003. Optimizing Oracle Performance.


I arrived on Monday. Client promised to pull together Transaction Information by Wednesday. Meanwhile…

?Oracle Database


AnalysisStart with the Oracle Database

Why start with Oracle?• Single element in common

across customers’ environments• Very well instrumented• All I knew about Client XYZ’s

environment

Here’s what I did:• v$session showed 200 JDBC

Thin Client sessions – Connection Pool!

• v$sess_io showed relatively “even” session usage

• picked 5 out of 200 sessions• dbms_system.set_ev

(sid,serial#,10046,12,’’)• captured and analyzed trace files


Trace File Analysis, Recommendations, Results

• “SQL*Net message from client” predominated

• “CPU service” for a handful of SQL statements taking one or a few seconds each

• Each SQL statement preceded by a dbms_describe

With much iterative, cut/paste testing:• ran dbms_stats at 30%• one new index• one code change• one hint

• individual SQL statements now subsecond• fixes implemented

• made notes – to pursue reason why/ possible

elimination of describe– to review init.ora CBO parm’s

• optimizer_index_cost_adj = 10 not 100• optimizer_index_cache = 80 not 0


Meanwhile, Client Feedback on Thursdaythen again on Friday after Fixes

User Action

Response TimeBefore Fixes

(seconds)

Response Time After Fixes (seconds)

Response TimeGoal

(seconds)

A 19 18 5

B 31 29 5

C 6 6 5

… … … 5

Clearly, not enough Progress!!!


Now What? What’s Happening End to End?

?Oracle Database

SQL*Net

msg from Client

User


The End to End Question is easierin the IBM Mainframe world

Graphical Command Center Mainframe

“The status light is red for the Catalog Center operation in Wichita because Carol Smith’s Order Entry transaction from there is taking 3.92 seconds which exceeds the SLA of 2.0 seconds; drilling down on the red status shows her online transaction waiting on the database, which is suffering from high I/O contention on file 23, due to large batch job XYZ that’s running right now at an inappropriate time of day.”

IBM Designed and Built Performance Considerationsthroughout its End to End Technology Stack


In the Open Systems WorldOur Strength is our Weakness…

• Lower cost of computing• Robust solutions from multiple vendors• No common or dominant performance management architecture

across the end to end technology stack of multiple vendors

How can I figure out

the End to End Here?


Hotsos Method RReprised

1. “What’s the most important thing to be working on?”– Business priority

2. “Where did the system spend my time?”– Resource profile– Which SQL is responsible?– Which SQL to blame for time spent between db calls?

3. “What’s the most effective way to fix it?”– Eliminate waste, eliminate competition– Stop when “fixing” isn’t worth it

4. Repeat

Source: Millsap, C.; Holt, J. 2003. Optimizing Oracle Performance.


Generalizing Method R for End to End Performance Optimization

• Identify and Prioritize Important Transactions• Determine Application Architecture• Establish Instrumentation Plan• Conduct Tests or Probe Production• Measure Activity• Compile Results• Identify Opportunities to Optimize


User

PC

Trans

Gateway

Interface Conn Pool Oracle

MQ Interface MQCICS/ Cobol

3rd Party Workflow Product

Workflows

EngineSOAP

SOAP

.Net Client

HTTP

Client XYZ’s Application ArchitectureAfter Many Conversations with Many “Silo” Teams


User

PC

Trans

Gateway

Interface Conn PoolOracle



Workflows

EngineSOAP

SOAP

.Net Client

HTTP

How to Instrument this End to End?

Stopwatch

Sniffer

.out file

.lis file

.log file

10046 traces.log file

Key for “Logging”

• Existing, custom written

• Existing, vendor provided


Log File Attributes, Issues, and Demonstration Tests

Attributes• Timestamps

– start– end– duration (calculated)

• Transaction/Step name– current process– called process

Issues• Awareness - Client’s Production Support team didn’t

know that logs were available or how to enable them– Original architects no longer involved with project– “Custom” logs built by silo teams for their own

purposes but not widely known to other teams– “Standard” logs supplied by vendors but not

widely known by project team• Inconsistent timestamp granularity across log files

Test 1 Fail UNIX servers not time synched

Test 2 Fail Windows servers not time synched

Test 3 Fail Logging filesystems filled up

Test 4 Fail Sniffer buffer filled up

Test 5 Success Collate activities cross platform by timestamp


The Logs were HelpfulTo Reveal Discrete Actions per Architecture Component

.Net Client User Actions

Trans Gtwy SOAP Calls

Called Workflows

Called Oracle Procedures

A S1 Wa GET…

B S2 Wb DO…

C S3 Wc xyzLKUP…

… … … …


The Logs when Collated by Timestamp were HelpfulTo Map the User Action/Transaction Flow

• User Action A– SOAP Call 1

• Workflow Wd– Oracle Proc xLKUP– Oracle Proc GET…

– SOAP Call 2• Workflow Wa

– Oracle Proc jLKUP• MQ Call 17

– SOAP Call 8• Workflow Wg

– Oracle Proc yLKUP

• User Action B– SOAP Call 7

• Workflow Wz– Oracle Proc jLKUP

• MQ Call 23– SOAP Call 9

• Workflow Wz– Oracle Proc Do…

– SOAP Call 1• Workflow Wd

– Oracle Proc xLKUP– Oracle Proc GET…


“A”User

PC

Trans

Gateway

Interface Conn PoolOracle



Workflows

EngineSOAP

SOAP

.Net Client

HTTP

The Logs were HelpfulTo Create a Resource Profile

18.00

.08

9.00

8.70

seconds seconds

seconds

seconds


The Logs were HelpfulTo Reveal “Too Much Work” and Other Issues

SOAP Call Type CountCount % of Total Count

Sum (msec) Sum % of Total Sum

Avg (msec)

S1 Lookup 24 18.18% 73980.00 21.36% 3082.50

LOGIN Other 1 0.76% 3213.00 0.93% 3213.00

S2 Lookup 81 61.36% 185327.00 53.51% 2287.99

S3 Other 14 10.61% 44598.00 12.88% 3185.57

S4 Other 12 9.09% 39222.00 11.32% 3268.50

Totals: 132 100.00% 346340.00 100.00% 2623.79


Client XYZStatus

• “Thanks – this is the best End to End View we’ve ever had!”• Using logging on an ongoing basis• Collapsing multiple, ongoing static lookups

– one static lookup at login– cache lookups in .Net client– realize tradeoff of one slightly longer login for many 6-8 second savings per edit or

review throughout the day• Reevaluating current Transaction Gateway Usage (XML Lovefest)

– currently maps .Net client inbound SOAP/XML to alternative SOAP/XML format before passing downstream to Workflow or MQ; receivers need to reparse for their own purposes

– substantial latency/cpu consumption from SOAP/XML parsing and manipulation– considering simpler non-XML map for messages “once in glass house” to eliminate

redundant construction/parsing overhead• Reevaluating current Workflow Engine Usage

– currently used as “database driver” but each workflow = 1000+ steps around db call– considering using only when “flow” is required otherwise call db from Gateway


Lessons LearnedIf you are an Oracle Performance Analyst…

• Current methods can quickly isolate problems to be inside or upstream of the database

• If the problems are “not Oracle”, you can still add HUGE value– you already are successful within an existing method framework– you are a big picture thinker– you know what questions need answers– you have the confidence to lead– you can be an objective facilitator across many people and

technologies– you can dig out the details from simple log file analysis

• You should check your eyeglass prescription before reviewing logs• You need to expand your past successes with the

Generalized Method R for End to End Optimization


Lessons LearnedIf you are Building or Managing an Application

• Performance needs to be architected into an Application by Design• Performance needs to be a “mindset” throughout a Project Life

Cycle, not an afterthought months or years later• When in “Design Doubt,” Performance Proofs of Concept can

channel development efforts in a positive and efficient direction• Performance methods and tools need to be

– well-tested– documented– implemented

• You need to Employ the Generalized Method R for End to End Performance Optimization


The Generalized Method R for End to End Performance Optimization

• Identify and Prioritize Important Transactions• Determine Application Architecture• Establish Instrumentation Plan• Conduct Tests or Probe Production• Measure Activity• Compile Results• Identify Opportunities to Optimize


Identify and Prioritize Important Transactions

• Critical to the business• Priority order• Use case

– user– location– navigation path– data

• Current and desired response time• Improvement’s value to the business


Determine Application Architecture

• Identify the major technology building blocks through which the business transactions flow

• Identify the service calls/interfaces among the blocks• Construct a “sequence diagram” template to map the blocks

User Web Server Middle Tier Oracle Database

User Time

Unacct Time

Web Time

Unacct Time

Middle Tier Time

Unacct Time

Database Time

Trans Service

Request

DB

Call


Establish Instrumentation Plan

For each transaction,• establish how to measure end-user time• establish how to measure response time consumed by each

building block in the architecture

User Web Server

Middle TierOracle Database

Trans Service

Request

DB

Call

Stopwatch? Sniffer?

Log File? Log File? Traces


Conduct Tests or Probe Production

• Define goals• Construct a capture plan

– How/who will drive transactions?– How/who will support the logging?

• Prepare to execute the plan– Time synch all platforms– Enough space for logging?– “Skinny down” test connection pool

• Execute the plan• Measure activity• Preserve the results, outputs


Measure Activity

Phase Steps

Before Enable data collection for each relevant block

During Confirm that data are being collected properly

After • Turn off data collection• Capture the results


Compile Results

• Use the Sequence Diagram Template• Sift through the collected data• Key individual event data into the template• Record or calculate total “accounted for” time• Derive “unaccounted for” time• Determine if “unaccounted for” requires refinement and a new

test


Identify Opportunities to Optimize

• Review sequence diagrams• Construct resource profiles• In descending order by time consumption

– identify the component consuming the most time– evaluate optimization opportunities

• eliminating unnecessary service calls• combining many service calls into fewer• tuning individual service calls or actions


References

Millsap, C.; Holt, J. 2003. Optimizing Oracle Performance. Sebastopol CA: O’Reilly.This book provides a full description of Method R, a detailed reference for Oracle’s extended SQL trace facility, an introduction to queueing theory for the Oracle practitioner, and a set of worked performance improvement example cases.

Download - Klein - Measuring End-to-End Response Time through ALL the Tiers

Top Related