identifying hotspots in the postgresql build process
DESCRIPTION
Software developers rely on a fast and correct build system to compile their source code changes and produce modified deliverables for testing and deployment. The scale and complexity of the PostgreSQL build process makes build performance an important topic to discuss and address. In this talk, we will introduce a new build performance analysis technique that identifies "build hotspots", i.e., files that are slow to rebuild (by analyzing a build dependency graph), yet change often (by analyzing version control history). We will discuss the identified hotspots in the 9.2.4 release of PostgreSQL.TRANSCRIPT
![Page 1: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/1.jpg)
Identifying Hotspots in the PostgreSQL Build Processes
Shane McIntosh
Ahmed E. Hassan
Bram Adams
Meiyappan Nagappan
![Page 2: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/2.jpg)
Source code
What is a build system?
!2
![Page 3: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/3.jpg)
Source code
Deliverable
What is a build system?
!2
![Page 4: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/4.jpg)
.tex
.c
.cc
.o
.o
.dvi
.a
.exe
.deb
Build systems describe how sources are translated into deliverables
!3
![Page 5: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/5.jpg)
(4) Test
(1) Think
(2) Edit
(3) Build
The developer’s work cycle
All builds are equal…
!4
![Page 6: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/6.jpg)
“
…But some builds are more equal than others
http://xkcd.com/303/
Why is the gtk build so abysmally slow? This hinders our ability to run gtk bots
and keep the gtk build green. :( ”!5
![Page 7: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/7.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 8: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/8.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 9: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/9.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 10: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/10.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 11: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/11.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 12: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/12.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
Before refactoring:!4 commands!
triggered
![Page 13: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/13.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 14: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/14.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 15: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/15.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 16: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/16.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 17: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/17.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 18: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/18.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
After refactoring:!2 commands!
triggered
![Page 19: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/19.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
!8
![Page 20: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/20.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
Graphanalysis results
(1)Dependency
Graph Construction (3)
Build Hotspot
DetectionQuadrant Plot
Build Activity
File
Chu
rn
(2)Dependency
GraphAnalysisDep. Graph
VersionControlSystem
An approach to detect slowly rebuilding files
!9
![Page 21: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/21.jpg)
MAKAO tool extracts build dependency graphs
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
Design recovery and maintenance of build systems B. Adams, Herman Tromp, Kris De Schutter, Wolfgang De Meuter
[ICSM 2007]
!10
![Page 22: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/22.jpg)
The cost of traversing an edge is derived by timing its build command
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
!11
Median of 10 repetitions
![Page 23: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/23.jpg)
Open source case studies!!
!
!
!12
![Page 24: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/24.jpg)
0 750,000 1,500,000 2,250,000 3,000,000
2,752,225
38,102
Build dependency graph properties
0 35,000 70,000 105,000 140,000
60,170
121,710
5,131
3,375
# Nodes# Edges
!13
![Page 25: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/25.jpg)
Incremental build performance
!14
0 100 200 300 400 500 600
02
46
810
12
File ID
Build
Tim
e (s
)
0 100 200 300 400
050
100
150
File ID
Build
Tim
e (s
)
.c files .h files
![Page 26: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/26.jpg)
Incremental build performance
!15
.c files .h files0 200 400 600 800 1000
0.0
0.5
1.0
1.5
2.0
File ID
Build
Tim
e (s
)
0 100 200 300 400 500 600
050
100
150
File ID
Build
Tim
e (s
)
![Page 27: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/27.jpg)
Incremental build performance
!16
.c files .h files0 2000 4000 6000 8000 10000
050
100
150
File ID
Build
Tim
e (s
)
0 2000 4000 6000 8000 10000
020
0040
0060
0080
00
File ID
Build
Tim
e (s
)
![Page 28: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/28.jpg)
So, refactoring should target the slow files!
!17
0 100 200 300 400 500 600
050
100
150
File ID
Build
Tim
e (s
)
![Page 29: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/29.jpg)
So, refactoring should target the slow files!
!17
0 100 200 300 400 500 600
050
100
150
File ID
Build
Tim
e (s
)
![Page 30: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/30.jpg)
So, refactoring should target the slow files!
!17
0 100 200 300 400 500 600
050
100
150
File ID
Build
Tim
e (s
)
![Page 31: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/31.jpg)
So, refactoring should target the slow files!
!17
0 100 200 300 400 500 600
050
100
150
File ID
Build
Tim
e (s
)
![Page 32: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/32.jpg)
!18
> Please answer the following questions for each source file listed above:!> (1) Is this source code file a performance bottleneck? Y/N!!For all of them no...
Let’s see what developers think!
![Page 33: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/33.jpg)
!19
> Please answer the following questions for each source file listed above:!> (1) Is this source code file a performance bottleneck? Y/N!!For all of them no...!!> (2) Why do you believe this file is or is not a performance bottleneck?!!...because none of these headers change often.
Let’s see what developers think!
![Page 34: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/34.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
!20
![Page 35: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/35.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
…but they may only rarely change!
!20
![Page 36: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/36.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
!20
![Page 37: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/37.jpg)
!21
Mining frequently changing files from version control
Terminal
$ git clone … … $ git checkout <some_tag> … $ git log --oneline <some_file>
![Page 38: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/38.jpg)
!21
Mining frequently changing files from version control
Terminal
$ git clone … … $ git checkout <some_tag> … $ git log --oneline <some_file>
But many of these files already build quickly!
![Page 39: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/39.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
!22
![Page 40: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/40.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
…but they may already be optimal!
!22
![Page 41: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/41.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
…but they may already be optimal!
!22
Focus on build hotspots:
Files that rebuild slowly and change often
![Page 42: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/42.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
Graphanalysis results
(1)Dependency
Graph Construction (3)
Build Hotspot
DetectionQuadrant Plot
Build Activity
File
Chu
rn
(2)Dependency
GraphAnalysisDep. Graph
VersionControlSystem
An approach to detect slowly rebuilding files
!23
![Page 43: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/43.jpg)
(3)Build
Hotspot Detection
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
Graphanalysis results
(1)Dependency
Graph Construction (3)
Build Hotspot
DetectionQuadrant Plot
Build Activity
File
Chu
rn
(2)Dependency
GraphAnalysisDep. Graph
VersionControlSystem
Hotspot detection approach
!24
![Page 44: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/44.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!25
![Page 45: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/45.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!25
![Page 46: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/46.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!25
![Page 47: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/47.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
Refactor
these first!
!25
![Page 48: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/48.jpg)
Open source case studies!!
!
!
Rebuild cost 90 seconds
Number of changes Median
Thresholds
!26
![Page 49: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/49.jpg)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●●
●●
●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●
●
●
●
●●
●●
●●●
●
●
●●
●
●
●●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●●●●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
0.25
0.50
0.75
1.00
0 50 100 150Build Time
Nor
mal
ized
File
Chu
rn
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
65 hotspots!(7% of
source files)
!27
glib/glib.hglib/glib-object.h
Main culprits
![Page 50: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/50.jpg)
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●●●
●
●
●
●●●●●
●●
●
●●
●
●●
●
●
●●●●●●●●●●●
●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●●●●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●●●
●●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●●●●
●
●
●●●●●
●
●
●
●●●●●
●
●
●
●
●●●●●●●
●
●●●●●
●●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●●●●●●●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●●●●●●
●
●
●●
●●●●●●●●
●
●
●●
●●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●●●●●●●●●●●
●
●
●
●●●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●●●●●●●●●●●●●●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●●
●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●●
●●
●
●
●●
●
●●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●●●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●●
●
●
●●
●
●●●
●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●
●●●●
●●
●
●●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●●
●
●●
●
●
●
●●●
●
●
●●●●●
●
●
●
●●●●●
●
●
●
●●●●●
●
●●
●
●●●●●●●●
●
●
●●●●
●
●●●●
●
●●
●
●
●
●
●●●
●●●
●
●●●●●●●●●●●●●●●●
●
●
●
●●●
●
●
●
●
●●
●●●●●●●●●●
●
●
●
●
●●
●●
●
●
●●●●●
●
●●●●●
●
●●●●●●
●
●
●●●●
●
●
●
●
●
●●
●●●●●●●●●●●
●
●
●●●●●
●
●
●●●●●●●●
●
●
●
●
●●
●
●●●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●●●●
●
●●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●●●●●
●●●●●
●
●●
●
●●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●●●●●
●
●●
●
●
●
●
●
●●●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●
●
●
●●●●●●●●●●●●●●●
●●
●
●
●●●●●
●●
●
●●●●
●
●●
●●
●
●●
●●●
●
●●
●●
●●
●
●
●
●●
●●
●●●●
●
●
●●●●●
●●●
●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●
●●●
●
●●●
●●
●●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●●●●●●
●●●●●
●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●●●●●
●●●●●●
●
●●●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●●●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●●●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●●●
●
●
●●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●●●●
●
●
●●●
●
●
●●
●●●●●
●●
●●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●●●
●
●
●
●●
●
●●
●●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●●●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
●●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●●
●●
●
●●●●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●
●●●
●●
●●●●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●●●●●
●
●
●
●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●●●
●
●●●
●
●●●●●●●●●●●●
●●
●
●
●
●
●●●●●●●●
●●●
●●
●
●●
●
●
●●●
●●
●●
●
●●●●●●●●●●●●●
●
●
●●●
●●
●●●●●●●●●●
●●●●●●●●●
●●●
●●●
●
●
●●
●●
●●●
●●
●●
●
●●
●
●●●
●
●
●●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●●
●●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●●
●●
●●
●●
●●
●●●
●●●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●●●
●●●
●●
●
●●●
●●●●●
●
●●●
●
●
●●
●
●
●
●●
●●●●
●●
●
●
●●
●●
●
●●●
●
●●
●
●
●●●
●
●●
●●●●
●
●●
●●
●●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●●●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●●●●
●
●●
●●●●
●
●
●
●
●
●
●●●●●
●
●●
●●●●●●●●●●●●●●●●●●●●●
●
●●●●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●
●●●
●
●
●●●
●
●
●●●●
0.25
0.50
0.75
1.00
0 2000 4000 6000 8000Build Time (s)
Nor
mal
ized
File
Chu
rn 732 hotspots!(8% of source files)
!28
qtxmlpatterns
High hotspot!concentration
qtbase/…/corelib
Main culpritstmtm
![Page 51: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/51.jpg)
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●●
●●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●●
●●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●
●
●
●
●
●●
●●
●●●●●●●●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●●
●●
●
●●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
0.25
0.50
0.75
1.00
0 50 100 150 200Build Time (s)
Nor
mal
ized
File
Chu
rn27 hotspots!
(2% of source files)
!29
postgres.haccess/htup.h
Main culprits
access/genam.haccess/xact.hutils/builtins.h
![Page 52: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/52.jpg)
hot.c
...1 2
hot.o
nn-1
!30
General trends Transitive property of build hotspots
![Page 53: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/53.jpg)
hot.c
...1 2
hot.o
nn-1
!30
General trends Transitive property of build hotspots
![Page 54: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/54.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
!30
General trends Transitive property of build hotspots
![Page 55: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/55.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
!30
General trends Transitive property of build hotspots
![Page 56: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/56.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!31
![Page 57: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/57.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!31
![Page 58: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/58.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!31
![Page 59: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/59.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!31
![Page 60: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/60.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!31
![Page 61: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/61.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!31
![Page 62: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/62.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
Limit internal use of “header file hubs” as much as possible!
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!31
![Page 63: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/63.jpg)
RELENG 2014
2nd International Workshop on Release Engineering
http://releng.polymtl.ca
?
?
![Page 64: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/64.jpg)
RELENG 2014
2nd International Workshop on Release Engineering
http://releng.polymtl.caDeadline: February 28, 2014
Event: April 11, 2014
?
talks on research
and practice
keynotes
poster session
and panel
?
![Page 65: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/65.jpg)
RELENG 2014
2nd International Workshop on Release Engineering
http://releng.polymtl.caDeadline: February 28, 2014
Event: April 11, 2014
?
talks on research
and practice
keynotes
poster session
and panel
hosted in Mountainview (CA, US)
by?
![Page 66: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/66.jpg)
RELENG 2014
2nd International Workshop on Release Engineering
http://releng.polymtl.caDeadline: February 28, 2014
Event: April 11, 2014
?
talks on research
and practice
keynotes
poster session
and panel
hosted in Mountainview (CA, US)
by
Dinah McNutt
(Google)
![Page 67: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/67.jpg)
![Page 68: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/68.jpg)
![Page 69: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/69.jpg)
![Page 70: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/70.jpg)
![Page 71: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/71.jpg)
![Page 72: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/72.jpg)
![Page 74: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/74.jpg)
!34
Bonus!
![Page 75: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/75.jpg)
!35
postgres.h
src/include/c.h src/include/port.h src/include/utils/elog.h src/include/utils/palloc.h src/include/pg_config_manual.h
![Page 76: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/76.jpg)
!36
htup_details.h
src/include/access/htup.h src/include/storage/itemptr.h src/include/storage/itemid.h src/include/storage/bufpage.h src/include/access/tupdesc.h src/include/catalog/pg_attribute.h src/include/access/tupmacs.h
![Page 77: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/77.jpg)
!37
genam.h
src/include/access/skey.h src/include/storage/lock.h src/include/storage/shmem.h src/include/utils/hsearch.h src/include/storage/lwlock.h src/include/utils/relcache.h
![Page 78: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/78.jpg)
!38
src/include/access/xlog.h src/include/access/xlogdefs.h src/include/nodes/pg_list.h src/include/lib/stringinfo.h
xact.h
![Page 79: Identifying Hotspots in the PostgreSQL Build Process](https://reader036.vdocument.in/reader036/viewer/2022081404/5598a9b51a28ab91268b4664/html5/thumbnails/79.jpg)
!39
src/include/fmgr.h src/include/nodes/parsenodes.h src/include/nodes/primnodes.h src/include/nodes/pg_list.h src/include/nodes/nodes.h
builtins.h