![Page 1: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/1.jpg)
Understanding and Detecting Real-World Performance Bugs
Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz and Shan Lu
University of Wisconsin–Madison
1
![Page 2: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/2.jpg)
Software Efficiency is Critical
• No one wants slow and inefficient software– Frustrate end users– Cause economic loss
• Software efficiency is increasing important– Hardware is not getting faster (per-core)– Software is getting more complex– Energy saving is getting more urgent Still Not Finished?
![Page 3: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/3.jpg)
Performance Bugs
• Implementation mistakes causing inefficiency• An example
Bookmark All Tabs…
many open tabs
![Page 4: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/4.jpg)
Performance Bugs
• Implementation mistakes causing inefficiency• An example
for (i = 0; i < tabs.length; i++) { … tabs[i].doTransact();}
Buggy Code Snippet:
Bookmark one URL
number of tabs
![Page 5: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/5.jpg)
Performance Bugs
• Implementation mistakes causing inefficiency• An example
for (i = 0; i < tabs.length; i++) { … tabs[i].doTransact();}+ doAggregateTransact(tabs);
Buggy Code Snippet: Patch
![Page 6: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/6.jpg)
Need to Fight Performance Bugs
• Performance bugs are severe– Escaping compiler optimization– Large speed up after fixing small code regions• E.g. “hang disappear” after fixing in the last bug
• Performance bugs are common– 5 to 50 Mozilla perf bugs are fixed each month
6
![Page 7: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/7.jpg)
How to fight Performance Bugs
BugDetection
Bug Fixing
PerformanceTesting
Bug Avoidance
![Page 8: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/8.jpg)
How to fight Performance Bugs
BugDetection
Bug Fixing
PerformanceTesting
Bug Avoidance
What are the root causes and locations of performance bugs?
How are performance bugs fixed?
How are performance bugs introduced?
How do performance bugs manifest?
![Page 9: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/9.jpg)
Contribution
• 1st comprehensive study of real-world perf. Bugs– Study 109 bugs from 5 applications– Guide future performance bug research
• Rule-based bug detection– Build 25 checkers– Find 332 previously unknown PPPs– Guide future performance bug detection
9
![Page 10: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/10.jpg)
Outline
• Motivation• Methodology• Characteristics Study• Rule-based Bug Detection• Conclusion
10
![Page 11: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/11.jpg)
Outline
• Motivation• Methodology• Characteristics Study• Rule-based Bug Detection• Conclusion
11
• Motivation• Methodology• Characteristics Study• Rule-based Bug Detection• Conclusion
![Page 12: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/12.jpg)
Methodology
• Application and Bug Source
• Threats to Validity
Application
Apache
Chrome
GCC
Mozilla
MySQL
Software Type
Server Software
GUI Application
GUI Application
Compiler
Command-line Utility +Server + Library
Language
C/Java
C/C++
C/C++
C++/JS
C/C++/C#
MLOC
1.3
Bug DB History Tags
Compile-time-hog5.7
4.7
14.0
N/A
N/A
perf
S5
0.45
14 y
13 y
10 y
13 y
4 y
# Bugs
25
10
10
36
28
Total: 109
![Page 13: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/13.jpg)
Root causes and locations?
How performance bugs are introduced?
How to patch performance bugs?
How to expose performance bug?
Outline of Characteristics Study
![Page 14: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/14.jpg)
Root Causes of Performance BugsPerformance Bug Detection
![Page 15: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/15.jpg)
Uncoordinated Functions
Skippable Function
Synchronization Issue
Others0
1020304050
MySQLMozillaGCCChromeApache
Root Causes of Performance BugsPerformance Bug Detection
Dominating
![Page 16: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/16.jpg)
Uncoordinated Functions
Skippable Function
Synchronization Issue
Others0
1020304050
MySQLMozillaGCCChromeApache
Root Causes of Performance BugsPerformance Bug Detectionfor (i = 0; i < tabs.length; i++) {
… tabs[i].doTransact();}+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch
![Page 17: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/17.jpg)
Root Causes of Performance BugsPerformance Bug DetectionnsImage::Draw(…) {
…} Mozilla Bug 66461
+ if(mIsTransparent) return;
![Page 18: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/18.jpg)
Root Causes of Performance BugsPerformance Bug Detectionint fastmutex_lock (fmutex_t *mp){
- maxdelay += (double) random(); + maxdelay += (double) park_rng();…} MySQL Bug 38941 & Patch
![Page 19: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/19.jpg)
Root Causes of Performance BugsPerformance Bug DetectionImplication: Future bug detection
research should focus on these common root causes.
![Page 20: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/20.jpg)
Locations of Performance BugsPerformance Bug Detection
Implication: Detecting inefficiency in nested loops is critical.
![Page 21: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/21.jpg)
How Performance Bugs are Introduced
Performance Bug Detection
Performance Bug Avoidance
![Page 22: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/22.jpg)
How Performance Bugs are Introduced
Performance Bug Avoidance
Dominating
![Page 23: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/23.jpg)
How Performance Bugs are Introduced
Performance Bug Avoidance
int fastmutex_lock (fmutex_t *mp){ - maxdelay += (double) random(); + maxdelay += (double) park_rng();…} MySQL Bug 38941 & Patch
![Page 24: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/24.jpg)
How Performance Bugs are Introduced
Performance Bug AvoidancensImage::Draw(…) {
…} Mozilla Bug 66461
+ if(mIsTransparent) return;Not Born Buggy!
![Page 25: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/25.jpg)
How Performance Bugs are Introduced
Performance Bug AvoidanceImplication: Performance aware
annotation systems and change-impact analysis tools are needed.
![Page 26: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/26.jpg)
How Performance Bugs ManifestPerformance Bug AvoidancePerformance Testing
![Page 27: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/27.jpg)
How Performance Bugs ManifestPerformance Testing
Always Active Special Feature Special Scale Feature+Scale0
1020304050607080
MySQLMozillaGCCChromeApache
Unique, severe
![Page 28: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/28.jpg)
Always Active Special Feature Special Scale Feature+Scale0
1020304050607080
MySQLMozillaGCCChromeApache
How Performance Bugs ManifestPerformance Testing
Special Feature Large Scale
![Page 29: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/29.jpg)
How Performance Bugs ManifestPerformance TestingImplication: New input generation
tools are needed.
Always Active Special Feature Special Scale Feature+Scale0
1020304050607080
MySQLMozillaGCCChromeApache
![Page 30: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/30.jpg)
How Performance Bugs are FixedPerformance TestingPerformance Bug Fixing
![Page 31: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/31.jpg)
How Performance Bugs are FixedPerformance Bug Fixing
Change Call Sequence
Change Condition
Change A Parameter
Others0
10
20
30
40
50
MySQLMoziilaGCCChromeApache
• Patch sizes are small– 42 patches are no larger than 5 LOC– Median patch size = 8 lines of codesFixing perf. bugs does not hurt readability
![Page 32: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/32.jpg)
Other Characteristics
• Performance bugs hide for a long time
• Server vs. Client; Java vs. C/C++– More server bugs caused by synch. issues– Others are consistent
• Correlations among characteristic categories– skippable function root cause change condition fix
![Page 33: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/33.jpg)
Outline
• Motivation• Methodology• Characteristics Study• Rule-based Bug Detection• Conclusion
33
• Motivation• Methodology• Characteristics Study• Rule-based Bug Detection• Conclusion
![Page 34: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/34.jpg)
Efficiency Rules
• An efficiency rule includes two parts– A transformation that improves code efficiency– An applying condition of the transformation
Rules applicable to only one application Rules applicable cross application
for (i = 0; i < tabs.length; i++) { … tabs[i].doTransact();}+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch int fastmutex_lock (fmutex_t *mp){ - maxdelay += (double) random(); + maxdelay += (double) park_rng();…} MySQL Bug 38941 & Patch
![Page 35: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/35.jpg)
Heuristics to Evaluate
• Efficiency rules widely exist
• Rule violations can be statically checked
• Violations to efficiency rules widely exist
![Page 36: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/36.jpg)
Rule Extraction andChecker Implementation
• Identifying Efficiency Rules
Not Contain Rules
Dynamic Rules
LLVM Checkers
Python Checkers
![Page 37: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/37.jpg)
Rule-Violation Detection Results
• 17 checkers find PPPs in original buggy versions• 13 checkers find 332 PPPs in latest versions
Found by cross-application
checking
Inherits from buggy versions
Introduced later
* PPP: Potential Performance Problem
Efficiency rules and rule-based performance-bug detection is promising!
![Page 38: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/38.jpg)
Conclusions and Future Work
• First characteristics study on performance bugsPaper contains more details (contact us for more)
• Efficiency rules are useful
• Future work– Alabama: dynamically detect inefficiency in nested loopsAlabama has found severe previously unknown inefficiency
![Page 39: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/39.jpg)
39
Thanks a lot!
![Page 40: Understanding and Detecting Real-World Performance Bugs](https://reader036.vdocument.in/reader036/viewer/2022062501/56815c29550346895dc9ff7d/html5/thumbnails/40.jpg)
Questions?Findings Implications
Skippable Functions and Uncoordinated Functions are most common root causes. Inefficiency detection should focus on
these common patterns.
Most bugs are introduced by Workload Mismatch and API Misunderstanding.
29 out of 109 bugs were not born buggy.
Many bug-exposing inputs need special features and large scales.
Most patches are a few lines of code.Similar mistakes are made in many places by many developers in many software.
Inefficiency avoidance needs performance annotations and change-impact analysis.
Test-input generation should consider coverage + intensity.
Efficiency != bad readability
Efficiency rules are important!
Most bugs involve nested loops.
Bugs involve nested loops are common and severe.
Alabama detects inefficiency in nested loops.