performance tuning through iterations
TRANSCRIPT
Performance Tuning a CFML script
Gert FranzRasia GmbH
WHO AM I?
• Gert Franz
– Involved with Lucee and Railo since day 1
– Into CFML for 10 11 12 13 14 15 16 17 years
– DBA, System architect
– CTO Rasia Ltd, CIO Rasia CH
• Rasia is founding Member of the Lucee association
MY EVOLUTION
Assembler Basic Clipper Pascal Delphi Java CFML Lucee
WHAT IS THIS ABOUT
• Having some fun with CFML
• Discrediting long known approaches
• Looking for new ways of solving a given problem
TASK AT HAND
• The client reported that there was
– An import job that imports an .xlsx file
– Into a database table
– With some special rules
INITIAL CODE
• The code is around 2 years old
• Takes around 20 minutes to execute
• Simply imports a total of 25000 recordsinto a database with some specific checks
• So nothing really fancy
STARTING POINT
• Let's have a look at the source code that was used to fulfil the task
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
MAIN POINTS VERSION 1
• Uses a default datasource• Uses lists for lookups• Creates an array to hold fieldnames• Creates a query and imports all into the
query• Uses two QoQ to delete unnecessary
records• Uses a QoQ to add only needed records to
an array• Insert the array into the database table
IDEAS? PROPOSALS?
FIRST ITERATION
• Convert everything to CFScript
WHY???
ITERATION 1
• Use a function to load the XLSX content
• Optimize filling the fields
• Exit early in the cleanUpNumber() function
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript, reduced the field building effort
ITERATION 2
• Replace the main QoQ with a nested struct
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
QUERY OF QUERY EVIL
QUERY OF QUERY
• Lucee is no database server
• So nothing of the following
– Query analyzer
– Indexes
– Statistics
– Query Optimizations
– etc!
QUERY OF QUERY
• Avoid if possible at all times!
ITERATION 3
• Eliminated the evaluate approach
EVALUATE – SMALLER EVIL
EVALUATE
• Evaluate sometimes does a lot of parsing
• I mostly use it only on serialized complex objects
• VERY OFTEN SEEN:
<cfset b = evaluate("form.#fieldName#")>
Instead use:
<cfset b = form[fieldname]>
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
ITERATION 4
• Eliminated the SQL statements from getMarketID()
• Avoid too many unnecessary SQL statements
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
ITERATION 5
• Eliminated all QoQ
• Only added stuff when model > 0
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
ITERATION 6
• Why create a query in the first place?
• Why not create the resulting struct right away?
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminated creating a query and filling the struct
ITERATION 7
• What is always the thing that consumes the most time?
• Queries
• Let's get rid of many of them
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
ITERATION 8
• We only fill the struct in cases where it makes sense
• So why filter afterwards, if we can avoid filling these values in, in the first place
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Exiting early from filling the struct
ITERATION 9
• Create a bulk insert, instead of lots of individual inserts
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
ITERATION 10
• Instead of nested structs, use composite keys
• Nested structs use lots of recursions
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key
ITERATION 11
• Replaced lists with arrays
• Generally speaking: Lists are to be avoided if possible
• Used len() instead of isEmpty() and added an exit early strategy
INTERMEDIATE RESULT
# T (s) Δ (s) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key
11 0.32 0.09 21.90% 0.05% 2050.31 4'035 B Replaced isEmpty() with len() and eq 0, eliminated Trim
FINAL RESULT CHART
0
100
200
300
400
500
600
700
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
100
200
300
400
500
600
700
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
2
4
6
8
10
12
14
16
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
0.5
1
1.5
2
2.5
3
3.5
4
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 3 4 5 6 7 8 9 10 11 12
FINAL RESULT CHART
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1 2 3 4 5 6 7 8 9 10 11 12
AS A PIE CHART
3.6%
91.7%
0.9%1.5% 0.4%0.2%1.0%0.1%0.5%0.0%0.0%
3.6%
0.9%
1.5%
0.4%
0.2%
1.0%
0.1% 0.5%
0.0%0.0%
CONCLUSIONS
• DON'T USE QueryOfQuery
• Don't use heavily nested structs
• Avoid lists, use structs or arrays instead
• Avoid too many database calls
CONCLUSIONS
• If you know the data you're handling, performance tuning is a lot easier
• First generalize in order to understand the code
• Then specialize in order to tune it