![Page 1: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/1.jpg)
Software: Good, Bad, or Just a Matter of Taste?
Arie van Deursen Delft University of Technology, The Netherlands
SBES 2012, Natal, Brazil, 27 September 2012
1
![Page 2: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/2.jpg)
Acknowledgements
• Marcio Delamaro & CBSoft organizers
• Joint work with Eric Bouwers & Joost Visser Software Improvement Group
• Co-workers TU Delft Software Engineering Research Group
2
![Page 3: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/3.jpg)
3
A. van Deursen and T. Kuipers. Source-Based Software Risk Assessments. ICSM, 2003.
www.sig.eu
Collect detailed technical findings about software-intensive systems
Translate into actionable information for high-level management
Using methods from academic and self-funded research
![Page 4: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/4.jpg)
Focus on Software Architecture
The organizational structure of a software system including components, connectors,
constraints, and rationale
-- Kogut & Clemens, 1994
“Architectures allow or preclude nearly all of a system’s quality attributes.”
-- Clements et al, 2005
4
![Page 5: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/5.jpg)
Early versus Late Evaluations
• Early: Making design decisions
– Design space exploration
– What-if analysis
• Late: Analyzing as-implemented architecture
– Drift & erosion
– Actual suitability
5
van Deursen, Hofmeister, Koschke, Moonen, Riva: Symphony: View-Driven Software Architecture Reconstruction. WICSA 2004
L. Dobrica and E. Niemela. A survey on software architecture analysis methods. IEEE Transactions on Software Engineering 2002.
![Page 6: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/6.jpg)
The Software Risk Assessment Process
6
ICSM 2009
![Page 7: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/7.jpg)
Lord Kelvin
Meten = Weten
Medição = Conhecimento
To measure = To know
“when you cannot measure it, (…), your knowledge is of
a meagre and unsatisfactory kind”
7
![Page 8: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/8.jpg)
8
Country Size Brazil - NL
![Page 9: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/9.jpg)
9
Population Brazil - NL
![Page 10: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/10.jpg)
10
Gross Domestic
Product
![Page 11: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/11.jpg)
11
GDP Per Capita
![Page 12: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/12.jpg)
12
Brazil – NL Football Wins
4-3
![Page 13: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/13.jpg)
13
IT Labor Force?
![Page 14: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/14.jpg)
14
IT Profitability?
![Page 15: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/15.jpg)
15
IT Quality??
![Page 16: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/16.jpg)
Software metrics for project management? Beware!
E. Bouwers, J. Visser, and A. van Deursen. Getting what you Measure Communications of the ACM, May 2012
16
![Page 17: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/17.jpg)
Pittfall 1:
Metric in a bubble:
No context, no goal
17
![Page 18: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/18.jpg)
Pittfall 2:
Metrics galore: Measuring everything
18
![Page 19: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/19.jpg)
Pittfall 3:
One-track metric: Narrow-minded
19
![Page 20: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/20.jpg)
Pittfall 4:
Treating the metric: Fighting symptoms
20
![Page 21: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/21.jpg)
21
ISO25010 -- The Many Faces of Software Quality
![Page 22: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/22.jpg)
22
What to measure?
Heitlager, Kuipers, Visser. A Practical Model for Measuring Maintainability. QUATIC 2007 J. Visser. SIG/TÜViT Evaluation Criteria Trusted Product Maintainability:
Guidance for producers Version 4.2, 2012
![Page 23: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/23.jpg)
Metric Value: Good or Bad?
• Benchmark of many industrial systems
– All metrics of interest collected
• Determine thresholds based on percentage of systems with given value.
– E.g.: 90% of systems have McCabe <= 15
– McCabe > 15 suggests high risk.
23
Tiago L. Alves, Christiaan Ypma, Joost Visser. Deriving metric thresholds from benchmark data. ICSM 2010.
![Page 24: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/24.jpg)
2009: Re-assessing Architectural Properties Study of 40 risk assessments Rethinking architectural properties Outcome: Metrics refinement wanted
24
Eric Bouwers, Joost Visser, Arie van Deursen: Criteria for the evaluation of implemented architectures. ICSM 2009
![Page 25: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/25.jpg)
Information Hiding
25
Things that change at the same rate belong together
Things that change quickly should be insulated from things that change slowly
Kent Beck. Naming From the Outside In. Facebook Blog Post, September 6, 2012.
![Page 26: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/26.jpg)
Encapsulation Metrics?
Which software architecture metrics can serve as indicators for the success of encapsulation of
an implemented software architecture?
26
Eric Bouwers, Arie van Deursen, and Joost Visser. Quantifying the Encapsulation of Implemented Software Architectures
Technical Report TUD-SERG-2011-033-a, Delft University of Technology, 2012
![Page 27: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/27.jpg)
Approach
• Identification of candidate metrics
• Quantitative approach (repository mining): – Which metric is the best predictor of good
encapsulation?
• Qualitative approach: – Is the selected metric useful in a late architecture
evaluation context?
27
![Page 28: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/28.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
Module (size) Component Module dependency
Lifted (comp) dependency
C1 C2
C3
28
![Page 29: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/29.jpg)
Literature Study: Candidate Metrics
29
![Page 30: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/30.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Commit in version repository results in change set 30
![Page 31: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/31.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set I: modules { A, C, Z } Affects components C1 and C3 31
![Page 32: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/32.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set II: modules { B, D, E } Affects components C1 only
Local change 32
![Page 33: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/33.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set III: modules { Q, R, U } Affects components C2 only
Local change 33
![Page 34: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/34.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set IV: modules { S, T, Z } Affects components C2 and C3
Non-Local change 34
![Page 35: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/35.jpg)
Observation 1: Local Change-Set Series are Good
• Combine change sets into series
• The more local changes in a series, the better the encapsulation worked out.
35
![Page 36: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/36.jpg)
Observation 2: Metrics may change too
• A change may affect the value of the metrics.
• Cut large set of change sets into sequence of stable change-set series.
36
![Page 37: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/37.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set I: modules { A, C, Z } Affects components C1 and C3 37
![Page 38: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/38.jpg)
U Z
C
E
A B
R
X
S
Y
P
T
Q
D
C1 C2
C3
Change set I: modules { A, C, Z } The Change Set may affect metric outcomes!! 38
![Page 39: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/39.jpg)
Solution: Stable Period Identification
39
![Page 40: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/40.jpg)
Approach
• Identify 10 long running open source systems
• Determine metrics on monthly snapshots
• Determine stable periods per metric: – Metric value
– Ratio of local change in this period
• Compute (Spearman) correlations [0, .30, .50, 1]
• Assess significance (p < 0.01)
• [ Assess project impact ]
• Interpret results
40
![Page 41: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/41.jpg)
Systems Under Study
41
![Page 42: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/42.jpg)
Results
42
![Page 43: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/43.jpg)
Best Indicator for Encapsulation: Percentage of Internal Code
Module types:
1. Internal
2. Inbound
3. Outbound
4. Transit
43
Eric Bouwers, Arie van Deursen, Joost Visser. Dependency Profiles for Software Architecture Evaluations. ICSM ERA, 2011.
![Page 44: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/44.jpg)
Threats to Validity
Construct validity • Encapsulation ==
local change? • Commit == coherent? • Commits too small? • Commits too large? • Architectural (meta)
model appropriate?
External • Open source, Java • IC behaves same on other
technologies Internal validity • Stable periods: Length, nr,
volume • Monthly snapshots Reliability • Open source systems • All data available
44
![Page 45: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/45.jpg)
Shifting paradigms
• Statistical hypothesis testing:
Percentage of internal change is valid indicator for encapsulation
• But is it of any use?
• Can people work with?
• Shift to pragmatic knowledge paradigm
45
![Page 46: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/46.jpg)
Software Risk Assessments
46
![Page 47: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/47.jpg)
Using “Percentage of Internal Change” For Evaluation Purposes
Setting
• SIG has included %IC in its revised maintainability model
• Used in risk assessments and monitors of many industrial systems
• Frequent discussion between clients, consultants, suppliers, and analysts.
47
![Page 48: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/48.jpg)
Evaluation Approach
• SIG consultants and analysts report any IC-related experience to investigator
• Collect “stories” of actual use in (anonymized) memos.
– 6 months, 50 memos so far
• Analyze using grounded theory
48
Michaela Greiler, Arie van Deursen, Margaret-Anne D. Storey: Test confessions: A study of testing practices for plug-in systems. ICSE 2012: 244-25
![Page 49: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/49.jpg)
Preliminary Findings (1)
[ Still collecting, analyzing, and digesting ]
• Metrics added require componentization / architectural view. – Documenting it is a useful process
– Low %IC values trigger discussion
• Important to see metric in full suite (avoid “one-track method”)
49
![Page 50: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/50.jpg)
Preliminary Findings (2)
Different “architectures” exist:
1. In the minds of the developers
2. As-is on the file system
3. As used to compute the metrics
• These should be the same!
• Should [3] capture [1] or [2]?
50
![Page 51: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/51.jpg)
Preliminary Findings (3)
• Is low %IC fixable?
• Requires (substantial) refactoring:
– Smaller components and interfaces,
– Fewer dependencies.
• Consultant’s wish list: “what-if analysis”:
– See effects of changing architecture
• Major refactoring for one of the systems monitored planned based on %IC insights
51
![Page 52: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/52.jpg)
Encapsulation Can be Measured!
Module types:
1. Internal
2. Inbound
3. Outbound
4. Transit
52
And doing so, leads to meaningful discussions.
![Page 53: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/53.jpg)
Software: Good, Bad, or Just a Matter of Taste?
Ultimately a Matter of Engineering Tradeoffs
Yet there is a difference between right and wrong
53
![Page 54: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/54.jpg)
Software Metrics Matter
• A key tool for informed decision making
• Benchmarking, monitoring, and interpretation are key
• Pitfalls are plentiful, in research as in practice
54
![Page 55: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/55.jpg)
(Open Source) Repositories: A Research Gold Mine
• Today: Metrics validation.
• GitHub! GitHub! GitHub!
• Relentless identification of threats to validity esstential: “Ameaças à validade”!
55
![Page 56: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/56.jpg)
Paradigm: Pragmatism
• Evaluate based upon the possibilities of action
• Calls for rigorous qualitative studies capturing reality in rich narratives
• Rich palette of empirical methods needed
56
![Page 57: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/57.jpg)
The Final: July 13, 2014
57
![Page 58: Software: Good, Bad, or Just a Matter of Taste?](https://reader034.vdocument.in/reader034/viewer/2022052505/555e579ed8b42a8e4c8b50d9/html5/thumbnails/58.jpg)
The Final: July 13, 2014
58
Brazil vs Netherlands