Download - Code performance workshop - PUG Challenge
2
compile-listing ain’t enough
▪ Intro
▪ ABL Profiler: your new best friend
▪ Indexes: it's easier to find things quickly when you know where you left them
▪ Conclusion
4
one named Peter (so-called Judge)
Software Architect*
@ Progress since 2003
Integration-y stuff – Authentication Gateway, HTTP-Out,
Corticon et al
OE Best Practices / OERA / AutoEdge / CCS
4GL since 1996
* Aka programmer who knows PowerPoint
5
one named Paul (Koufalis)
Progress DBA and UNIX admin since 1994
Expert consulting related to technical aspects of
Progress and OpenEdge
Wide range of experience
Small 10 person offices to 3500+ concurrent users
AIX, HPUX, Linux, Windows…if Progress runs on it,
I’ve worked on it
Single malts and American bourbons enthusiast
6
▪ The oldest and most respected independent DBA consulting firm in the world
▪ Five of the world’s top OpenEdge DBAs
▪ Author of ProTop, the #1 FREE OpenEdge Database Monitoring Tool
http://protop.wss.com
7
compile-listing ain't enough
Everybody knows that there's a problem: "It's slow". Now what?
1. Find the source of the problem
2. Identify the errant code – HOW ?
3. Fix it. That's the easy part
8
why are we here?
▪ The purpose of this workshop is to help you identify and fix application performance
problems by
• Familiarising you with the ABL profiler
• (Re) Teaching you the index rules
▪ You can use these in dev, QA and prod environments
▪ We're not talking about
• How do you measure (relative) performance
• Database design
• Network traffic
• All the other crappy slow parts of your application
▪ Point of the labs is to IDENTIFY problems not necessarily fix 'em
▪ The code & db we have is intended to be 'realistic' aka crappy
▪ You can expect
• Some useful facts and info
• A lot of opinions
You know what you're doing
We expect your opinions
and input
expectations
11
your next best friend: the abl profiler
▪ What is it?
• "Unsupported" tool that provides line-by-line execution timings
• "Officially" available since v9 via PROFILER handle
• Ignores UI blocking
• 11.6.0 added tooling support to PDSOE
• Needs more love, especially wrt OOABL
▪ How do I use it?
• In DEV all-in-one tooling in PDSOE (via launch configs)
• In PROD
– PROFILER system handle
– Roll-your-own tools
15
profiling (production) code
▪ Set PROFILER attributes
ORDER MATTERS
• ENABLED: turns the feature on and off
• PROFILING: starts measuring timings
• LISTING: automatically generate
DEBUG-LIST for executed procedures
into DIRECTORY directory
• STATISTICS: dumps out various stats
• COVERAGE: If true, identify ALL
statements and IPs, whether run or not
▪ Add "magic" config
• Eight-finger CTRL-ALT-SHIFT-SPACE-
F7 in UI or a config file
• Turn it on/off, define output file etc
// Mandatory attributes. ORDER MATTERSprofiler:enabled = trueprofiler:file-name = session:temp-dir
+ '/profiler.out'profiler:description = 'working the shop'
// optional attributes. Listing require SOURCE// code, which shouldn't be on your PROD// systemprofiler:listings = falseprofiler:directory = session:temp-dirprofiler:trace-filter = '*':uprofiler:tracing = '':u
// from 10.2Bprofiler:statistics = true// coverage wins profiler:coverage = true
profiler:profiling = true
16
profiling (production) code
▪ When done, stop profiling and write out
the data
▪ Optionally add listings in later
if profiler:enabled then
do:
assign profiler:profiling = false
// optionally turn the feature off
profiler:enabled = false.
// profiling's not much use unless it's
// dumped to disk
profiler:write-data().
end.
17
profiling via command-line
▪ Specify values via a file on the command
–profile <profile-options-file>
• where profile-options-file contains
The profiler is enabled for the whole session
-COVERAGE-STATISTICS-DESCRIPTION "your description"-FILENAME "/path/to/profile.out"-LISTINGS "path/to/debug/listings/directory"-PROFILING-RAWDATA-TRACE-FILTER "*"-TRACING "procedure-name | line-number , ..."
18
profiling production faq1. What should I profile?
For an AppServer request, turn on in the activate event procedure to profile the entire ABL request. Don't forget to turn profiling off and/or dump the data
In UI, after data comes back from the AppServer, measure how long it takes to query and browse all 14 squillion records in the "thoroughly-endorse" table
See exactly which lines of code are run (code coverage). Works better with automated tests
You're in this workshop for a reason; I'm sure you have places in your apps that need profiling
2. Can I profile AppServer calls?
Yes, but it requires two profiler sessions (ie the profile data is not automatically measured on the server
3. I have no source on my production system, can I still profile my app?
Yes, absolutely!
4. What's the cost in using profiling at runtime?
Minimal / you shouldn't care
19
profiler output file format
The file contains 11 sections, which are in EXPORT format
• Separated by single-line . (dot / period)
1. Description of profiling session
2. Description of modules (e.g., procedures, functions, triggers)
3. Procedure call-tree information
4. Timing summaries for each executable line that was a part of the profiling session
5. Detailed tracing information
6. Coverage analysis
7. Operators info
8. Module detail
9. Session high-water marks
10. Session db info
11. Data written using the USER-DATA() method
new in v2
empty in v2
20
2 03/21/2017 "slow http call NO-SRC NO-LISTING" 12:43:46 "pjudge".254 "OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter" "DefaultRequestFilter.dbg" 30800235 "~HttpHeader OpenEdge.Net.HTTP.HttpHeader" "" 091 "OpenEdge.Net.HTTP.BuilderRegistry" "" 12769266 "KeySet OpenEdge.Core.Collections.KeySet" "" 0146 "EncodeString OpenEdge.Net.URI" "" 0189 "propGet_Registry OpenEdge.Net.HTTP.Filter.Writer.SetHeaderMessageWriterBuilder" "" 0126 "Example/dummy.r" "" 29514321 "GetByte OpenEdge.Core.Memptr" "" 0382 "~HttpClient OpenEdge.Net.HTTP.HttpClient" "" 0224 "AddUserAgent OpenEdge.Net.HTTP.HttpClient" "" 0213 "propGet_Registry OpenEdge.Net.HTTP.ResponseBuilder" "" 052 "OpenEdge.Logging.Filter.LoggerFilterNode" "" 52279385 "~ClientSocket OpenEdge.Net.ServerConnection.ClientSocket" "" 0
profiler output example (1)
1
2
3
21
coverage
119 "ReturnDataset" 3444647.119 "OutputDataset" 6525455566364.119 "OutputDataset" 56970717879.
22
119 "ReturnDataset" 3444647.119 "OutputDataset" 6525455566364.119 "OutputDataset" 56970717879.
coverage
23
profiler links
▪ https://community.progress.com/community_groups/openedge_general/w/openedgegeneral/
1980.profiler-control-tool
▪ Kbase: PROFILER Object attributes and methods
http://knowledgebase.progress.com/articles/Article/19495
• Lots of other kbases on the Profiler and related tooling
25
compile-listing ain’t enough
▪ Everybody knows that there's a problem: "It's slow".
• Now what?
▪ Last year we covered some of the tools available to help you find the errant code
• ProTop, LOG-MANAGER, VST data, -zqil
▪ Peter just covered profiler
▪ Now that we know where the problem is, let’s fix it
26
stupid index tricks
▪ We are going to cover the rules without going into all the dirty details
▪ There are a number of excellent presentations on index rules
• Mike Lonski’s numerous “Pick an Index” PPTs are my favourites
27
but before we get to the rules…
▪ This only applies to ABL and not SQL
▪ Rules are applied in hierarchical order to filter indexes
• This is important: Each rule is applied and the result is one or more remaining indexes
• Use a worksheet approach to make the rules "easy" to apply
▪ The first 5 rules only apply to a subset of indexes
• Compiler scans all fields in the query and selects all indexes that have leading components with those fields
▪ Field match rules must be contiguous
• “Equality” on fields 1 and 3 of the index counts as 1, not 2
Rules continue to be applied until there is only one index left
28
the rules
For the first-pass set of indexes, filter using the following rules:
1. Pre-select only indexes with leading components in the where clause
2. If CONTAINS use word-index
3. Unique index with all components involved in the equality matches
4. Most active equality matches
• Sorta kinda…full matches trump partial matches
• But only if more than 1 field (sometimes)
5. Most active range matches
6. Most active sort matches
If no "good" index is found, select one from
7. The primary index
8. First index alphabetically by name
30
order table indexes
Table Records Size Size Count LOBs Size Mean
PUB.Order 727285 63.9M 92 727285 -- -- --
Indexes
Flags Index Name Cnt Field Name
u CustOrder 2 + CustNum + Ordernum
OrderDate 1 + OrderDate
pu OrderNum 1 + Ordernum
OrderStatus 1 + OrderStatus
SalesRep 1 + SalesRep
w SRepW 1 + SalesRep
SRepDate 2 + SalesRep + OrderDate
DateSRep 2 + OrderDate + SalesRep
SDateOstat 2 + ShipDate + OrderStatus
for each order
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X X X X X
Unique index with all components
involved in the equality matchesX X X X X X X X X
Most active equality matches (Full
matches trump partial matches)X X X X X X X X X
Most active range matches X X X X X X X X X
Most active sort matches X X X X X X X X X
The primary index *
First index alphabetically by
name
where salesrep = "BBB"
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X
Unique index with all components
involved in the equality matches X X X
Most active equality matches (Full
matches trump partial matches) 1 / 1 X 1 / 2
Most active range matches
Most active sort matches
The primary index
First index alphabetically by
name
1
3
where salesrep = "DKP" and orderdate = 09/05/2011
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X
Unique index with all components
involved in the equality matches X X X X X
Most active equality matches (Full
matches trump partial matches) X X X 2 / 2 2 / 2
Most active range matches X X
Most active sort matches X X
The primary index X X
First index alphabetically by
name X
2
5
1
34
multiple index use
▪ Where clause includes “AND”
• ALL components of each index are involved in equality matches
• No unique indexes are involved
▪ Where clause includes “OR”
• Both sides of OR contain at least the lead component of an index
• Either equality or range match
▪ CAREFUL: return order not guaranteed
35
where salesrep = "BBB" and orderStatus = "Shipped"
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X
Unique index with all components
involved in the equality matches X X X X
Most active equality matches (Full
matches trump partial matches) 1 / 1 1/ 1 X 1 / 2
Most active range matches X X
Most active sort matches X X
The primary index
First index alphabetically by
name
2
4
2
36
gotchas
▪ Expressions break bracketing
for each order no-lock where month(orderDate) = 1 ...
▪ BEGINS does NOT break bracketing
• Considered a range bracket
for each order no-lock where salesRep begins "D"
• Uses the order.salesRep index
▪ MATCHES breaks bracketing
▪ Temp-table rules are subtly different
▪ When in doubt, test and verify
• LOG-MANAGER is your friend
where month(orderDate) > 6
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X X X X X
Unique index with all components
involved in the equality matchesX X X X X X X X X
Most active equality matches (Full
matches trump partial matches)X X X X X X X X X
Most active range matches X X X X X X X X X
Most active sort matches X X X X X X X X X
The primary index *
First index alphabetically by
name
9
38
test and verify? how !?!
▪ Using the “salesrep begins …” example
▪ ProTop of course. It’s free.
▪ Query object handle q:index-information(i)
▪ Log-manager - see next slides
39
log-manager
▪ A terribly underused but awesomely amazing tool
▪ Allows you to leave debug messages in your code
• No more /* Message here vValue. */
▪ Create some secret hotkey sequence to activate
• I.e. you can turn it on in production for one user
▪ Writes detailed info to a log file
assign log-manager:logfile-name = "c:\temp\wshop.log"
log-manager:logging-level = 3 // There are more levels than you think
log-manager:log-entry-types = "4GLTrace,4GLTrans,QryInfo".
41
test and verify with log-manager
▪ Still with our “for each order no-lock where salesrep begins ‘d’ ” example:
Type: Dynamically Opened Query
PREPARE-STRING: for each order no-lock where salesrep begins 'd'
Prepared at Runtime
Client Sort: N
Scrolling: Y
Table: sports2000.Order
Indexes: SalesRep
Times prepared: 1
Time to prepare (ms): 0
DB Blocks accessed:
sports2000 : 366074
DB Reads:
Table: sports2000.Order : 182913
Index: Order.SalesRep : UNAVAILABLE
43
About ProTop
▪ Download free version at
http://protop.wss.com
▪ Local ChUI for real-time data
▪ Web portal for trending data
• Free and $$
▪ Monitoring and Alerts ($$)
▪ Free version is open source (GNU
GPL). Take a look under the hood
if you want to write your own
monitoring code
▪ Hit “h” to get the help screen
Options are CaSE-sENSiTiVe!!
44
ProTop Web Dashboard
Graphical view of _TableStat/_Indexstat VST
These graphs are from the $$ part of the ProTop web
dashboard
45
conclusion
▪ The aim today was to show you areas you can look at for improving the performance of your
application
▪ From a code perspective, by
• coding appropriately to the indexes you have
• retrieving only data that's appropriate
• not leaving data "lying around"
▪ From a design/architecture perspective, by
• appropriately balancing the size and quantity of network requests
▪ From an infrastructure perspective, by
• making sure that you optimise the data transfers from the DB server
for each order where
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index
Unique index with all components
involved in the equality matches
Most active equality matches (Full
matches trump partial matches)
Most active range matches
Most active sort matches
The primary index
First index alphabetically by
name
where month(orderDate) > 6
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X X X X X
Unique index with all components
involved in the equality matchesX X X X X X X X X
Most active equality matches (Full
matches trump partial matches)X X X X X X X X X
Most active range matches X X X X X X X X X
Most active sort matches X X X X X X X X X
The primary index *
First index alphabetically by
name
6
where salesrep = "BBB"
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X
Unique index with all components
involved in the equality matches X X X
Most active equality matches (Full
matches trump partial matches) 1 / 1 X 1 / 2
Most active range matches
Most active sort matches
The primary index
First index alphabetically by
name
1
3
where salesrep = "BBB" and orderStatus = "Shipped"
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X
Unique index with all components
involved in the equality matches X X X X
Most active equality matches (Full
matches trump partial matches) 1 / 1 1/ 1 X 1 / 2
Most active range matches X X
Most active sort matches X X
The primary index
First index alphabetically by
name
2
4
where salesrep = "DKP" and orderdate = 09/05/2011
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X
Unique index with all components
involved in the equality matches X X X X X
Most active equality matches (Full
matches trump partial matches) X X X 2 / 2 2 / 2
Most active range matches X X
Most active sort matches X X
The primary index X X
First index alphabetically by
name X
2
5
1
where salesrep = "DKP" and orderdate = 09/05/2011 by salesrep
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X
Unique index with all components
involved in the equality matches X X X X X
Most active equality matches (Full
matches trump partial matches) X X X 2 / 2 2 / 2
Most active range matches X X
Most active sort matches X X
The primary index X X
First index alphabetically by
name X
2
5
1
where salesrep >= "DKP" and orderdate >= 09/05/2011 by salesrep
Index Selection Rule
U
CustOrder OrderDate
PU
OrderNum OrderStatus SalesRep
W
sRepW SRepDate DateSRep SDateOstatCarrier
If “CONTAINS”, use word-index X X X X X
Unique index with all components
involved in the equality matches X X X X X
Most active equality matches (Full
matches trump partial matches) X X X X X
Most active range matches 2 / 2 2 /2
Most active sort matches 1 / 2 X
The primary index
First index alphabetically by
name
2
5
1