sandeep krishnan, isu robyn r. lutz, isu & jpl katerina goseva-popstojanova, wvu dept. of...
TRANSCRIPT
1
Empirical Evaluation of Reliability Improvement in an Evolving Software
Product Line
Sandeep Krishnan, ISURobyn R. Lutz, ISU & JPL
Katerina Goseva-Popstojanova, WVU
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
This research is supported by NSF grants 0916275 and 0916284
2
BackgroundProduct line – “A family of products designed to take advantage of
their common aspects and predicted variabilities” [Weiss and Lai 1999]
e.g. Nokia cellphones, HP printers, etc.
Products -
Commonalities – Shared by all products. e.g. Platform
Variabilities – Differentiate the productsHigh-reuse variation – JDT, PDE, Mylyn, Webtools, etc. Low-reuse variation – CDT, Datatools, Java EE tools.
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
3
MotivationDoes reliability improve as a product line evolves?
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
PL Evolution
Reliability
Reliability – Continuity of correct service; measured in terms of post-deployment failures [Avizienis, et al., 2001]
4
Research QuestionsFailure trends: Do serious failures decrease over time
as the common/high-reuse variation/ low-reuse variation components evolve over releases?
Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components?
Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components?
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
5
FindingsFailure trends: Do serious failures decrease over time as the
common/high-reuse variation/ low-reuse variation components evolve over releases?
Common High-reuse Low-reuse
Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components?
Common High-reuse Low-reuse
Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components?
Common High-reuse Low-reuse Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
6
Approach
Source of failure reports-
Failure cateogries –
Raw number of severe failures and percentage.Source of change reports – CVS repository of Eclipse.We look at change to existing files measured by KChanges
and change via new files [Mockus et al., 2000].Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
Blocker
Critical Major
Normal Minor
2007 2008 2009 2010 2011
Europa
Ganymede
Galileo
Helios
Blocker
Critical Major
7
Observations
Number of severe failures decreases over time as expected
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
1. A. Failure trends – Common components
8
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
Percentage of severe failures tends to stabilize and even increases gradually
1. A. Failure trends – Common components
9
Observations
Number of severe failures show mixed behavior, contrary to our expectations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
1. B. Failure trends – High-reuse variation components
10
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
Percentage of severe failures tends to stabilize and even increases gradually, contrary to our expectations
1. B. Failure trends – High-reuse variation components
11
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
Number of severe failures do not monotonically decrease; rather show mixed behavior
1. C. Failure trends – Low-reuse variation components
12
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
Percentage of severe failures show mixed behavior and not a decreasing trend
1. C. Failure trends – Low-reuse variation components
13
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
2. Change trends
• Common components: Percentage of new files decreases.• High-reuse variation components: Percentage of new files
tends to remain stable.• Low-reuse variation components: Percentage of new files
shows decreasing trend.• All components show relatively high modification rate.
14
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
3. Failure/evolution relationship
• Failures/new-file o Common components: For two subcomponents, release 1 has
lower failures/new-file than release 4. o High-reuse variation components: Release 1 always has higher
failures/new-file than release 4. o Low-reuse variation components: No specific trend.
15
Observations
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
3. Failure/evolution relationship
• Failures/Kchangeso No stable behavior for all components; mixed trends are
observed.
16
ConclusionCommon components have fewer severe failures over time
and amount of change also decreases.
Variation components show no uniform pattern of decrease in failures.
Even high-reuse variation components continue to change fairly rapidly.
Study suggests there may be no simple answer as to whether reliability tends to improve as product line matures.
Further investigation is needed on the relationship between failure and change in software product lines.
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011
17
Thank You!
Dept. of Computer Science, Iowa State University, MSR, May 22, 2011