component-based software engineering: modern trends ... · • the sevo (software evolution)...

1

Anita Gupta 28/05/2009

Component-Based Software Engineering: Modern Trends, Evolution and Perceived Architectural Risks

PhD thesis

Odd Petter Nord Slyngstad

Trondheim, 1st April 2011

2

Overview

• Introduction• Research Questions RQ1-RQ4;

from Design to Discussion • Contributions• Lessons Learned and Future Work

Odd Petter N. Slyngstad, 1. April 2011

3

Introduction• The SEVO (Software EVOlution) project (supported by Research

Council of Norway under the ICT-2010 program).

• Focus on improvements in knowledge of, and methods to handle, software evolution; CBSE as overall umbrella.

• Research performed in the Norwegian and European IT-industry; focused study within Norwegian company StatoilHydro ASA.

• Prof. Reidar Conradi as project leader, 2 PhD students, 1 Post Doctoral position.


Presenter

Presentation Notes

4Motivation

• Increased use of Commercial Off-The-Shelf software (COTS) components and Open Source Software (OSS) (means changes in e.g. component selection, integration, testing, vendor control).

• Risk Management closely related to Software Evolution; – Awareness of architectural evolution risks since architectural

changes can permeate a software system. – Focus in earlier studies commonly only on structured architecture

evaluations, but methods used range quite widely [Babar 2007a].

• Need to improve integration of architectural activities, notations and artifacts into software processes and tools [Buchgeber 2008].


Presenter

Presentation Notes

Software evolution is inevitable; changes in society and technology is the driver for keeping software systems up to date [vanVliet 2008]. Increased use of Commercial Off-The-Shelf components and Open Source Software (OSS) in new development mean different characteristics and impact on development and maintenance processes have to be taken into account. Risk Management is closely related to Software Evolution issues. Since Software architecture constitutes a central part of any software system [Bass 2004]; Awareness of potential architectural evolution risks is important as architectural changes can permeate a software system. Earlier studies on software architecture evolution focused on structured architecture evaluations, but actual methods used in industry can range quite widely [Babar 2007a], and context-specific factors may also have an influence [Babar 2007b]. Improvements needed with respect to the integration of architectural activities, notations and artifacts into software processes and tools [Buchgeber 2008].

5 CBSE, COTS/OSS, and Risk• Component-Based Software Engineering (CBSE): The process to

define/implement/integrate/compose components into software systems [Sommerville 2010]. Emphasizes:– Software reuse: reuse of e.g. components, design, knowledge [vanVliet 2008].

• Component: unit of composition, specified such that the interfaces are separate from the implementation [Crnkovic 2002].

• Commercial Off-The-Shelf software (COTS): Pre-existing, commercially ready-to- use software [Oberndorf 1997]. Vendor control of requirements, release schedule, and evolution [Vidger 1996][Vidger 1997].

• Open Source Software (OSS): free use, adaptation, and improvement of original source code [OSI 1998-2010] [FSF 1985-2010].

• Risk: any issue that can affect a project adversely if not handled correctly [Boehm 1991].


Presenter

Presentation Notes

Architectural environment/context (stakeholders, architect’s knowledge/experience, technical + organizational environment), Dev for vs. with reuse unit of composition with contractually specified interfaces and explicit context dependencies only and can be composed according to a component model by third party without modification [Szyperski 2002]

6Maintenance, Evolution and Architecture

• Software maintenance: work to keep already released software running and up- to-date, categorized as corrective, perfective, adaptive and preventive changes. Consumes 50%++ of the total software costs [Sommerville 2010].

• Software Evolution: under maintenance “umbrella” (non-corrective) [Sommerville 2010], or separate lifecycle step [Bennett 2000].

• Software Architecture: the structure(s) of the system, which comprise(s) software elements, the externally visible properties of those elements, and the relationships among them [Bass 2004].


7 Research Design• Three main types of empirical studies: quantitative, qualitative, and

mixed-method [Wohlin 2000][Creswell 2003] approaches;

– Quantitative: Using one or more study objects in combination for e.g. theory testing, while attempting to minimize contextual effects (i.e. “noise”).

– Qualitative: drawing information from a natural environment or social context [Denzin 1994] (i.e. context is essential and not “noise”).

– Mixed-method: combination of quantitative and qualitative methods, which complement each other with respect to individual scopes, strengths, limitations and biases [Seaman 1999].

• 4 surveys, 2 case studies in this thesis; using the mixed-method approach towards gaining deeper insights, and triangulating data/methods.


Presenter

Presentation Notes

Quantitative E.g. data mining for numerical data. Qualitative E.g. exploring textual data for inferred or combined meaning.

8

RW

Studies in StatoilHydro ASA Studies in European IT industry

Phase 3: Risk ManagementImpact of one risk management strategy (Test Driven Development)

Phase 1: State-of-practice of SPI in CBSE

P1 P2C12004

2008

Phase 2: Actual evolution;Software defects & changes

2005

2006

Perceived architectural risks and employed strategies

SP1 - SP7, SP10

P3 C2

2006

2008P4 P5 P6 C4bC4a

SP8, SP9

C3 RQ4, G2RQ3, G2

RQ2, G1

RQ1, G1

Research Overview

Figure labels:

Paper/Study

Contribution

Study Phase

- P1-P6 = Selected Articles, G1-G2 = SEVO project goals

- C1-C4 = Contributions,

- SP1-10 = Secondary Articles

- RQ1-4 = Research Questions

Presenter

Presentation Notes

Qualitative studies – investigating research questions based on e.g. textual data. Typically through surveys, oral interviews and discussions with developers. Quantitative studies – investigating research questions based on numerical data, typically through “data mining” of company databases. 4 surveys, 2 case studies. The studies in this thesis combine quantitative and qualitative methods towards gaining deeper insights. Studies run with some overlap due to time constraints, data source and resource availability (Phases started consecutively but finished concurrently)

9

SEVO goals vs. Research Questions (1)

• G1. Better understanding of software evolution, focusing on CBSE; investigating:

– Increased usage of COTS/OSS; exploring e.g. vendor control, component selection and integration issues [Li 2004]. COTS/OSS development presents new risks, e.g. effort estimation [SP1].

– RQ1: State of practices in CBSE for reusable components (articles P1, P2).

– Software reuse in CBSE; enables e.g. quality, effort (cost) and time to market, and standards improvements [Sommerville 2010]. Reusable components have lower defect densities than non-reusable components across releases [Mohagheghi & Conradi 2004b].

– RQ2: software defects and changes for individual, in-house reusable components (article P3).


Presenter

Presentation Notes

10

SEVO goals vs. Research Questions (2)

• G2. Better methods to predict the risks, costs, and profile of software evolution in CBSE; investigating:

– Test Driven Development (TDD); improving e.g. code comprehension, efficiency in defect detection. Prior research did not investigate TDD‘s effect on non-corrective changes and on reusable components [Janzen 2005].

– RQ3: impacts of TDD vs. test-last development for reusable components; defect and change density (article P4).

– Perceived risks and risk management issues in software architecture evolution; little prior effort has been made.

– RQ4: Perceived architectural risks and mitigation strategies from the viewpoint of software architects in industry (articles P5, P6).


Presenter

Presentation Notes

Test Driven Development (TDD); improving development and use of reusable assets (e.g. code comprehension, efficiency in defect detection). Prior research did not investigate TDD‘s effect on non-corrective changes and on reusable components [Janzen 2005].

11

Overview


from Design to Discussion• Contributions• Lessons Learned and Future Work


12 RQ1 – Design and Context

• RQ1: State of practices in CBSE for reusable components (articles P1, P2)

• Design: Survey at StatoilHydro ASA (P1) and survey in the European IT-industry (P2)

• Context: In-house: convenience sample at StatoilHydro ASA; all developers working with reuse. COTS/OSS: Convenience sample for pilot survey, expanded to stratified-random for larger survey (e.g. Census Bureau).


2006 2008

Phase 22004

Phase 1, State-of-practice of SPI in CBSE

P1 P2C1 SP1 - SP7, SP10RQ1, G1

Phase 3

Presenter

Presentation Notes

RQ1: What is the state of practices and issues with respect to software process improvement in CBSE for COTS/OSS and in-house reusable software? (articles P1, P2)

13RQ1 – Findings and Discussion (P1, P2)Findings: •In-house components: defined/standardized architecture and knowledge sharing organization are important for reuse success.

•COTS/OSS components: cost factors and convenience/stability influence development processes to be enriched with COTS/OSS specific activities (e.g. for selection, integration).

Discussion: •Main positive impacts agree with literature; costs, development time, quality, standardized architecture [Lim 1994]; training [Frakes 1995]; component understanding [Li 2004].

•Findings support literature on importance of component-provider relationship [Hauge 2010], and influence on component selection by project- specific property constraints [Hauge 2009a].


Presenter

Presentation Notes

Impact for in-house reusable components: We interviewed the developers involved in the reuse program at StatoilHydro ASA to identify and analyze the possible impacts on the development process. - A defined / standardized architecture is seen as key: It is likely, however, that the benefits of such standardization may be short-lived, as the architecture will probably also need to be evolved in order to accommodate future changes. This also provides a starting point for investigating architectural risks in CBSE-driven software evolution. - Organization of knowledge sharing remains important: Improved organization of knowledge sharing is needed; senior personnel in the company indicated that some external consultants were added to the project after regular training activities were carried out. These consultants therefore experienced a lack of knowledge with regards to the reuse practices of the company. This could be helped through improvement of documentation of the reusable components (as the results show that a managed collection of information would be beneficial). Qualitative data from developers on related benefits includes improved information sharing and learning, higher documentation quality, better overview of functionality, and access to FAQ (frequently asked questions) answers. Improved organization of training and knowledge sharing would also aid in a better understanding of the quality specifications of the reusable components – indicated as a problem potentially caused by rapid changes in requirements, resources and personnel. Nevertheless, the developers appear to have a good understanding of the reusable components themselves, obtained through informal knowledge sharing. Impact for external COTS/OSS components: When it comes to external COTS/OSS (both called OTS – Off-The-Shelf) components, impacts in several dimensions can be seen in that traditional processes, enriched with OTS-specific activities, are being used to select and integrate OTS components as follows: - Selection: Selection of OTS components is done informally, without specific focus on a particular lifecycle phase. More systematic management of OTS components knowledge beyond functional features is, however, necessary. - Integration and testing: It becomes costly to debug defects in the borderland between in-house and external components, where the latter are mainly treated as ―black boxes‖ (closed source). Moreover, since a multitude of test criteria for both functional (―black box‖) and structural now exist, the focus is more on determining the right combination of criteria towards efficient testing [Bertolino 2007]. - Effort Estimation: Such estimation is performed using personal experience, commonly inaccurate, and largely affected by stakeholders. - Traditional Quality: Negative quality effects are rare, and the trust in external components high. - Management complexity: There is a complex relationship between an application developer and an OTS provider, while customers (new software owners) are commonly not involved in decisions about use of certain OTS components. (Re)negotiation of requirements is therefore nonexistent, as stated in P2. In article P2, we investigated the state-of-the-practice in the European IT-industry concerning development based on COTS and OSS components, using a large multi-national survey in Norway, Germany and Italy. Here, we also investigated COTS/OSS in the IT-industry. The results from this study are presented as a set of findings regarding industrial practices with development based on Off-The-Shelf (OTS, i.e. COTS and OSS) components. The findings are taken verbatim from P2 and numbered as Fact 1 – 10, with corresponding complementary summaries: “Fact 1 – Development process: Companies use traditional processes enriched with OTS-specific activities to integrate OTS components.” Familiarity with OTS candidate components is an important factor to consider in customizing the entire development process. Sufficient knowledge of OTS candidate components may make the use of already adapted development processes (e.g. adapted evolutionary) unnecessary. “Fact 2 – Component selection: Integrators select OTS components informally. They rarely use formal selection procedures.” Benefits of and pre-conditions for using a formal component selection process are unclear due to the lack of clear empirical evidence. Lacking such evidence leaves integrators reluctant towards using such a formal process since it is also presumed complex and time-consuming. “Fact 3 – Component selection: There is no specific phase of the development process in which integrators select OTS components. Selecting components in early phases has both benefits and challenges.” We have identified possible problems that component integrators must consider when selecting OTS components in the early phases of a software development project. Results 51 “Fact 4 – Component integration: Estimators use personal experience when they estimate the effort required to integrate components and most of the time they do not estimate accurately. Stakeholder-related factors will affect dramatically the accuracy of estimates.” Some estimation tools, e.g. COCOTS [Abts 2000], consider both the technical nature of the components, and e.g. component understandability and vendor response time. Estimation tools should also take into account possible requirement changes and the evolution of components, especially for large, long-lived projects. “Fact 5 – Quality of the integrated system: Negative effects of OTS components on the quality of the overall system are rare.” For reasons such as low costs, component integrators sometimes select OTS components of a lower quality. It is thus the quality assurance efforts during selection and integration that ensure the quality of the OTS component in the finished system. “Fact 6 – OSS and COTS components: Integrators usually used OSS components in the same way as commercial components, i.e. without modification.” Alterations to the source code of an OSS component may be infeasible, in particular for long-lived commercial systems with many evolutionary iterations over their lifetimes, due to the possible internal support required. The context of the application must therefore be considered when deciding whether to use OTS components. “Fact 7 – Locating defects is difficult: Although problems with OTS components are rare, the cost of locating (i.e. within or outside OTS components) and debugging defects in OTS-based systems is substantial.” The deployment environments and configurations of OTS components come in a wide variety. This variety represents an obstacle towards reproducing reported errors, and irreproducible errors will commonly not be prioritized to be fixed by the component provider. “Fact 8 – Relationship with the provider: The relationship with the OTS component provider involves much more than defect fixing during the maintenance phase.” Different persons within an OSS community may be involved in separate tasks supporting the use (e.g. evaluation, selection, integration) of a component. “Fact 9 – Relationship with the client: Involving clients in OTS component decisions is rare and sometimes unfeasible.” It is often the case that the application client has no interest in implementation technicalities, due to lack of relevant competencies. It is thus important that clients‘ interests and technical competences are clarified at the start of a project to determine possible strategies for requirements (re)negotiation. “Fact 10 – Knowledge management: Knowledge that goes beyond the functional features of OTS components must be managed.” External channels for sharing knowledge and experience on the use of COTS/OSS are few and uncommon. This kind of information is spread across e.g. portals and bulletin boards, or managed internally by a ‗component responsible‘. It is therefore essential to manage knowledge beyond mere component functionality. Furthermore, we have found that there is a mismatch between academic theory (which is often based on incorrect assumptions) and industrial practice when it comes to components usage, due to the lack of empirical evidence, the lack of studies involving industry, and the lack of industrial adoption of research results. Examples include: While academia has deemed traditional development models unsuitable for COTS/OSS development and calls for adapted models, companies simply enrich these with COTS/OSS-relevant activities since they have sufficient knowledge of relevant COTS/OSS components. Researchers suggest selecting COTS/OSS components early in the development cycle, but there is no specific phase for this activity in industry. Moreover, selecting components early has both benefits and challenges.


• RQ2: software defects and changes for individual, in-house reusable components (article P3)

• Design: “Data mining” case study at StatoilHydro ASA (ClearCase/ClearQuest tools). Defect density (#trouble reports/NSLOC) and change density (#change requests/NSLOC) as metrics.

• Specific Context: JEF framework of reusable components, mainly based on the Java and J2EE technologies; six of seven individual components (~20 KNSLOC) studied.


2006 2008

Phase 3

Phase 1

2004

Phase 2, Actual Evolution: Software defects & changes

P3 C2SP8, SP9RQ2, G1

20052004

15RQ2 – Findings and Discussion (P3)Findings:•Decreasing defect density over consecutive releases, for five of six components

•Decreasing change density for the reusable components; – Increased maturity as they are being adapted to accommodate

new and altered requirements.– Improved reliability due to accumulated operational testing

through usage.

Discussion: •Prior research: reusable components were more stable (i.e. have lower densities of code modification) than non-reusable components over several releases [Mohagheghi & Conradi 2004b]. •Our results: larger components had higher defect and change densities in the first release, but this trend did not continue over consecutive releases.


Presenter

Presentation Notes

we verified findings on defect and change density for individual reusable components that were part of software development at a large industrial company in Norway (StatoilHydro), while they experienced new and changed requirements through software evolution. the individual reusable components investigated had lower defect densities over several releases. although the larger components had higher defect densities in the first release, this trend did not continue over several releases. Five of the six reusable components had a higher change density in the first release than in subsequent releases. However, in subsequent releases the larger components no longer had the higher change densities. In summary, These components have a decreasing defect density over several releases, but for change density the results remain inconclusive. - Evolution impact from defect density: A decreasing defect density indicates that fewer corrections are gradually needed (as the code matures), and thereby a higher quality level is achieved for these reusable components individually. The findings support the results from literature. Evaluation and Discussion 62 - Evolution impact from change density: A decreasing change density for the reusable components could indicate that they become more mature as they are being adapted to accommodate new and altered requirements in relation to new and existing ―client‖ software components. It could also be a natural result of improved reliability due to operational testing through usage (which is inherently higher for the reusable components).


• RQ3: impacts of TDD vs. test-last development for reusable components (article P4)

• Design: “Data mining” case study at StatoilHydro ASA (ClearCase/ClearQuest tools). Defect (#trouble reports/NSLOC) and change (#change requests/NSLOC) density as metrics.

• Specific Context: JEF-framework at StatoilHydro ASA; seven components (~20 KNSLOC) refactored to five components (~10KNSLOC) in the two latter releases developed with TDD.


Phase 3, Risk Management:Impact of one risk management strategy (Test Driven Development)


2006 2008

P4 P5 P6 C4bC4aC3 RQ4, G2RQ3, G2

Phase 2

Phase 1

2004

Presenter

Presentation Notes

Since 2004, this has become based on an in-house customized framework of reusable components, called the ―JEF framework‖ (the name ‗JEF‘ signifies that it is based on Java Enterprise Framework components). The aim of StatoilHydro was to explore the potential benefits of reusing software in a systematic manner. The framework is based on J2EE, and is a Java technical framework for developing Enterprise Applications [JEF 2006]. This initiative was started, as a response to changing business and market trends, by providing a shared platform for further in-house development and integration. The strategy is also being propagated throughout the company. Upper management has mandated that the JEF components are to be reused in all new development where applicable. There is also a training program in place for developers to gain knowledge of these components, their functionalities and their interfaces. The framework itself, and the corresponding applications that are using the framework, have been further developed in several releases over four years (2004 – 2008), with the latter releases being developed using Test Driven Development. Our research group was allowed to participate in the process to potentially verify some of the benefits of reuse sought by the company. The majority of the quantitative data comes from the ClearCase/ClearQuest software change management system, which is commonly in use at StatoilHydro. This system allows reporting of both Change Requests (for non-corrective changes) and Trouble Reports (for corrective changes or defects).

17

RQ3 – Findings and Discussion (P4)Findings: influencing factors include: •Reduction in mean defect density, higher mean change density.•Higher inherent maturity/reliability growth due to operational testing.•Process impact in terms of tests developed and level of adaptation. •Context factors should be explicitly considered when writing new test cases.

Discussion: •Reduction in mean defect density (36%) similar to non-reusable components [George 2004] [Maximilien 2003] [Janzen 2005].

•[George 1994] indicate the importance of context. Our results: specific characteristics of reusable components as part of this context.

•Refactoring overhead seen as disadvantage [George 1994]. Our results: worthwhile towards future reuse and adaptability.


Presenter

Presentation Notes

RQ3: What are the impacts of Test Driven Development versus test-last development on reusable components? (article P4) Impact on defect density: reduction in mean defect density for TDD compared to traditional development methodology. Tests developed in TDD remain a valuable asset towards reuse. Impact on change density: higher mean change density for TDD compared to traditional development methods, indicating that the reusable components are more adaptable (rather than having lower quality). an underlying well-defined architecture allows a high level of modification. introduction of new requirements from other systems and e.g. prior knowledge of and experience with TDD also play a role here. Additional factors: The following context factors: potentially higher change density, increasing abstraction level, higher number of effective users and middleware-like position, stable application domain and APIs. should be considered as part of the context for writing new test cases. The associated refactoring overhead is a worthwhile investment towards future reuse and adaptability. Shortcomings with respect to design for reusable components can be handled by implementing and using additional design and documentation practices.

18RQ4 – Design and Context

• RQ4: Perceived architectural risks and mitigation strategies from the viewpoint of software architects in industry (articles P5, P6)

• Design: Pilot survey, followed up by full-scale survey.

• Context: Norwegian IT-industry: convenience sample for pilot, expanded to snowball sampling variant.


Phase 3, Risk Management:Impact of one risk management strategy (Test Driven Development)


2006 2008

P4 P5 P6 C4bC4aC3 RQ4, G2RQ3, G2

Phase 2

Phase 1

2004

Presenter

Presentation Notes

Snowball sampling; key practitioners serve as contact points towards potential respondents

19RQ4 – Findings and Discussion (P5, P6)

Findings:

•Larger number of risks identified during project planning (than evolution). •Risks: indicate architects discover e.g. design errors from evaluations.•Mitigation strategies express recovery from missing evaluation output. •Risks and mitigation strategies organized in three-part operational matrix (technical, process, organization).

Discussion:

•The 15 of 21 (pilot survey) and 16 of 19 (main survey) most influential risks fit risk categories identified in [Bass 2007], [Ropponen 2000].

•Corresponding mitigation strategies identified appear not to have been investigated earlier.


Presenter

Presentation Notes

To investigate issues directly related to risk management in software architecture evolution, we first elicited actually experienced risks and employed mitigation strategies from software architects in industry. This was performed through a pilot survey entailing a series of semi-structured interviews, and the results are presented in article P5. The risks and management strategies we discovered in our pilot study were summarized and used as input towards the questionnaire-based full-scale survey. We investigated risks and strategies on the technical, process and organizational levels. Technical issues (e.g. in terms of existing or new technologies) can affect the architecture of a system to a large extent. Also, organizational and process issues are important because they are central to the success of operative reuse programs in industry. Our results from this first software architecture study show that the most influential risk was that “lack of stakeholder communication influenced the implementation of new and changed architectural requirements in a negative way”. This was also the most frequent risk encountered by the respondents. Secondly, “poor functionality clustering causing disadvantages towards performance” was also seen among the most frequent risks. Additionally, we found that there is little effort among software architects to evaluate and document the architecture, as they attempt to meet challenges as they are encountered during development. Identification of the most influential architectural risks: We have explored planning and development risks in projects, on the technical, process and organizational levels. These risks indicate that while some efforts towards proper risk management have already been made, further improvements are warranted in terms of learning and reflection. Furthermore, the identified risks span many different issues, showing that architectural risk management in software evolution must consider a wide range of factors. - Identification of the most successful risk mitigation strategies: The risk mitigation strategies were also identified according to technical, process and organizational levels, as matched to the identified risks. The low complexity level of some of these strategies indicates that risk management adoption does not necessarily require a large investment up front.

20


Operational Matrix, part two:

Process level risks and strategies

Presenter

Presentation Notes

We further developed a three-part operational matrix of risks and corresponding mitigation strategies as a tool to support risk management in software architecture evolution, based on the identified risks and strategies under contribution C4a. This matrix is based on the levels of technical, process and organizational risks, and includes an aggregated rating of outcome towards successful risk mitigation for each strategy. The matrix is presented here in three parts for technical (Table 3), process (Table 4) and organizational risks (Table 5), over the following pages. This matrix shows identified risks based on indicated level of influence, and corresponding management strategies along with a relative ranking of outcome towards successful mitigation, and can be expanded as needed. Risk influence is indicated as the number of respondents who replied that the corresponding risk had a ―Very High‖ (VH) or ―High‖ (H) influence on the architecture. - An operational matrix tool for risk management in software architecture evolution: This tool is intended for use in risk management within software projects, where evolution of the software architecture is an issue in one or more forms. Existing perceived risks are matched with corresponding strategies, which can be used ―as is‖ or as basis for further elaboration.

21Validity Summary – Surveys (RQ1, RQ4)

• Construct: Well-connected research literature, RQs, and questionnaire questions. Pre-tested questionnaire.All terminology defined.

• External: P1, P5: Convenience sample, Norwegian IT-industry. P2: stratified random sample, Norway/Germany/Italy. P6: Constrained snowball sample.

• Internal: All respondents had required background and knowledge. Any ambiguities were directly clarified. Minor issues: linguistic errors and translation to national languages (P2).

• Conclusion: Relatively small sample size in P1, P5. Main survey P6 depends on background from pilot survey P5.


Presenter

Presentation Notes


22Validity Summary – Case studies (RQ2, RQ3)

• Construct: defect density and change density are well founded concepts. Data based on complete and stable releases.

• External: individual industrial systems at one single company; results relevant and valid for other releases of same components (others on a case-by-case basis).

• Internal: Records with missing data were excluded. Analysis was cross-checked to ensure compliance, consensus and correctness. Potential for classification and assignment uncertainties at StatoilHydro ASA.

• Conclusion: Relatively small, but complete sets of data to draw conclusions. Confounding factors (e.g. developer experience differences between teams) were not considered a threat; all development within same organizational unit.


Presenter

Presentation Notes


23

Overview

• Introduction • Research Questions RQ1-RQ4;



24 Contributions, RQs and articles


Contribution ArticlesC1: Improved knowledge of modern trends in CBSE and their impacts on software development processes (RQ1)

P1, P2

C2: Improved understanding of evolution impact on individual reusable components in terms of defect and change densities (RQ2)

P3

C3: Improved understanding of the impact and effectiveness of TDD (RQ3)

P4

C4a: Identification of perceived risks and related mitigation strategies specifically for the evolution of software architecture (RQ4)

P5, P6

C4b: An adapted operational matrix for risk management in software architecture evolution (RQ4)

P6

25

Contributions to SEVO project goals

• G1. Better understanding of software evolution, focusing on CBSE: on process and component aspects towards a profiling of software evolution; contributions C1 and C2.

• G2. Better methods to predict the risks, costs and profile of software evolution in CBSE: Effectiveness of TDD – defects and changes; contribution C3. Risk and strategies in software architecture evolution; contribution C4a, and an adapted operational matrix tool; contribution C4b.

• G3. Contributing to a national competence based around these themes and G4. Disseminating and exchanging the knowledge gained: Publications in refereed international conferences and journals, reported to the FRIDA national database of research results. SEVO results integrated in NTNU courses and arranged workshops. All contributions C1-C4b.


Presenter

Presentation Notes

G1. Better understanding of software evolution, focusing on CBSE. We claim that this thesis advances the state-of-the-art within the field of software engineering, specifically in the context of the contributions C1 and C2. These contributions address process and component aspects towards a profiling of software evolution. G2. Better methods to predict the risks, costs and profile of software evolution in CBSE. Contribution C3 addresses the effectiveness of Test Driven Development as a strategy to manage software evolution impact in terms of defects and changes. Contributions C4a and C4b specifically address the risk aspect of software evolution, investigating perceived risks and corresponding risk management strategies. Through these contributions, we have achieved a better understanding of impact in terms of perceived risks and management strategies related to software architecture evolution. G3. Contributing to a national competence based around these themes. The work reported in this thesis has been published in refereed international conferences and journals (totaling 40 articles). All publications and related results have been reported to the FRIDA national database of research results. SEVO results have also been integrated in NTNU courses. G4. Disseminating and exchanging the knowledge gained. We have established regular contact, and have several joint future publications, with other researchers with similar research interests. We have also presented our work at international conferences (e.g. ISESE, ICSEA), and arranged workshops (e.g. at Simula Labs in Oslo) to disseminate and further build on the knowledge we have gained through our investigations.

26

Contributions to general practitioners and to StatoilHydro ASA

Improvements of components and processes for handling of software evolution:

•Integration of management tools needed for proper data collection and analysis. •Improved commitment required to properly report data on e.g. defects, changes.•Reuse training and knowledge sharing: Organization, scheduling and active participation should be improved.

Improvements in risk management of software architecture:

•Software architecture evaluation: implemented as a complete end-to-end process. •Need for improvements in implementing risk management strategies.


Presenter

Presentation Notes

In terms of recommendations for general practitioners of software engineering, including those at StatoilHydro ASA, we would like to highlight the following: - Improvements of components and processes for handling of software evolution: o Standardized architecture: In our results, a defined / standardized software architecture is seen as beneficial. Such an architecture should be evolved to accommodate future changes. An explicitly defined / standardized architecture could provide a proper basis for making such wide-reaching changes. o Integration of management tools: It is imperative that the project management tools, used for internal reporting in a company, are properly integrated with each other to enable proper data collection and analysis. In our case, Rational ClearCase (for SCM) and Rational ClearQuest (for handling TRs and CRs) tools were used for version control and defect/change reporting, respectively. Even though these two tools came from the same manufacturer, there was a lack of tool integration that made it difficult to obtain direct information on defects and changes (e.g. their ―size‖) incurred on specific versions. Rather, the company had to rely on complementary sources to provide this information. Evaluation and Discussion 69 o Reporting efforts: There has to be commitment in the development organization to properly report data (defects, changes, effort, etc.) being collected for later analysis in the company. Our experience is that unless such data are directly related to their everyday work tasks, many developers see their reporting as unnecessary and even a target for cheating. Even the name of an affected component or effort usage (simplified just as small, medium, large) is frequently missing. The same ―state-of-affairs‖ has been seen in dozens of companies according to [Mohagheghi 2006], thus enabling large changes may prove very difficult. Nevertheless, developers‘ input is crucial in providing relevant data for proper analysis towards product and process improvement. o Reuse training programs: Our results indicate that although company reuse programs do exist, knowledge of these programs may be variable among developers. Organization and scheduling of such programs and related activities should hence be improved, especially to account for the needs of external personnel that may be added to a given project during its lifetime. o Knowledge sharing: Rapid changes of personnel, requirements and resources can affect the quality specifications and related knowledge of reusable components negatively, as indicated in our results. It is therefore important that practitioners maintain knowledge sharing, regarding tacit and explicit knowledge, with their peers, and participate in the use and implementation of relevant collaboration tools (concrete examples include Microsoft SharePoint [Microsoft 2010] and open source suites such as Trac [Trac 2010]). - Improvements in risk management of software architecture: o Software architecture evaluation: Software architecture evaluation should be implemented more explicitly as a complete end-to-end process. As aforementioned, the current focus is merely on recovering artifacts, rather than hindsight reflection and learning. o Risk management strategies: The median outcome rating for the strategies from our results for all three risk categories (technical, process, and organizational) was ―Medium‖. So, there is still need for improvements in implementing risk management. o Risk mitigation and training: The focus of system architects‘ mitigation efforts appears currently to be on recovering needed architecture details and improving communication, while producing the system according to specification. Effort should therefore be made towards improving regular documentation and evaluation of the architecture, integrated with the maintenance / evolution process. Proper training of both architects and organizational management are means to achieve these improvements.

27

Overview




Presenter

Presentation Notes

Introduction – Intro, Motivation, Research Goals, Research Challenges, Research Questions, State of the Art (most important/central to thesis) Overview slide (?) Context – Company Backgrounds, systems studied, etc. Overview slide (?) Research Design – Research design details, use figure to show how the studies relate Overview slide (?) Results and Contributions – per research question, summarize the findings, overall discussion of results, relation between e.g. Contributions, RQs and papers, mapping to overall SEVO goals Overview slide (?) Recommendations and future work – recommendations, future work (bullet but talking points), acknowledgements (SEVO, supervisor/colleagues, StatoilHydro for access to data)

28

Lessons Learned for Researchers

• The proposed operational matrix tool can serve as a base to incorporate future insight.

• Defect density and change density give only a limited view of software evolution, e.g. component complexity is not considered.


Presenter

Presentation Notes

We have provided an updated definition of software evolution [P5], building on the one found in [Mohagheghi & Conradi 2004a]. This definition will aid in facilitating further studies of software evolution, as it specifically includes both code and other software artifacts, as well as quality aspects of software architecture evolution. The table-driven tool we have proposed for effective handling of architectural risks in software evolution (Tables 3, 4, and 5) can serve as a base to incorporate future insight on architectural risks in software evolution. The two metrics used, defect density and change density, are well-known in the literature as measures of software evolution [Mohagheghi 2004a] [Mohagheghi & Conradi 2004b]. However, these metrics give only a limited view of software evolution, without taking aspects such as component complexity into account. We would therefore encourage researchers to work towards improved and more detailed metrics for assessing component reliability and modifiability in software evolution.

29

Lessons Learned for Practitioners

• Standardized architecture – Keep architectural design and assets as “live” artifacts for dynamic updates – Ensure long-lasting commitment to empirical studies

• Encourage knowledge sharing to counter frequent changes of personnel, requirements, and resources; training as important precursor.

• TDD can promote reuse; higher inherent focus on testing and refactoring.

• Risk management: need to improve architectural documentation artifacts, and evaluate the architecture on a regular basis.


Presenter

Presentation Notes

A standardized architecture in the company should function as a platform for adapting the JEF framework of reusable components towards new requirements and further evolution at StatoilHydro ASA. To follow up such a goal, the company should: o Keep the architectural design and related assets as ‗live‘ artifacts, allowing updates dynamically as the architecture is evolved. Conclusion 77 o Ensure and maintain long-lasting commitment to empirical studies at all levels of the developing organization. Developers in our context are inherently skeptical towards tasks (read: ―extra work‖) that could not explicitly be related to their daily work. Knowledge sharing to counter frequent changes of o personnel, o requirements, and o resources can affect the combined collective knowledge of a team or even an entire software development organization. This means that maintaining and encouraging continuous knowledge sharing and exchange is important, in order to allow the organization to retain relevant knowledge once a developer leaves. A related recommendation appears in [Chen 2007], where the authors urged organizations to appoint a ―component uncle‖ to follow up the evolution of each adopted OSS component, or a component in a certain area or from (a) certain provider(s). Training can, depending on an individual‘s prior knowledge and experience, be an important precursor to effective knowledge sharing. Our results show that proper knowledge of a reuse training program was not present among all developers and hired-in consultants at StatoilHydro ASA. It is therefore important that such programs be properly promoted and carried out. Test Driven Development can help promote reuse due to the higher inherent focus on testing and refactoring. While our results show that TDD can help reduce the number of defects in comparison to test-last development for reusable components, effort measurements should also be a part of introducing TDD. This is because lower productivity has been shown in other industrial studies [Janzen 2005], albeit for non-reusable components. Risk management: system architects are currently focusing on obtaining and recovering evaluation artifacts and improving communication. At the same time, our results show that the median of the outcome ratings for the strategies was ―Medium‖. We thus need to improve architectural documentation artifacts, and evaluate the architecture on a regular basis. Furthermore, these aspects should be integrated seamlessly into the evolution process. Implementing a training program that covers both architects and organizational management seems to be a sensible way to achieve this.

30Future WorkImpact of defects and changes in software evolution (G1): •Costs/benefits of using COTS/OSS software•Improving metrics for code level complexity

Impact of modern trends in CBSE on the development process (G2):•Knowledge management and sharing•Efficiency of COTS/OSS and communities; benefits vs. disadvantages •Proposed improvements and their effects, e.g. at StatoilHydro ASA•Divergent perspectives in software architecture and agile development•Approach to TDD at StatoilHydro ASA

Architectural risks in software evolution (G2): •Code-level and other artifact data •Expanding the operational matrix tool further•Architecture evaluation methods•Architectural versus non-architectural changes


Presenter

Presentation Notes

With respect to the impact of defects and changes in software evolution (and related to SEVO goal G1): - A study of cost estimation related to OTS software: StatoilHydro uses a large amount of OTS software, requiring a certain amount of integration and tailoring for such components. Personal experience as the basis for estimating integration effort appears to be common in industry, and although it often leads to inaccurate estimates [P2], expert estimates are usually better than formal-model ones. The question remains how to define who is an expert. StatoilHydro aims to determine the total costs associated with integration and tailoring, as well as their distribution over change types for OTS components. Additionally, to investigate possible differences with respect to reuse of in-house components is of interest. - A study on complexity: Data on defect density and change density [Mohagheghi 2004a] [Mohagheghi & Conradi 2004b] [Mohagheghi & Conradi 2008] is commonly used towards investigating reliability and maintainability of software components. Nevertheless, these metrics do not include any weighting of complexity of individual components. Function points is a method that has been proposed to include software complexity in the size measure [Umholtz 1994], but as noted, no standard method for counting function points that includes algorithmic complexity is currently supported by ISO [ISO 2010]. This would be a study towards improved metrics for providing a more detailed view of component quality and reliability in software evolution. With respect to the impact of modern trends in CBSE on the development process (and related to SEVO goal G2): - A study of knowledge sharing: Certain teams at StatoilHydro ASA (indeed in most ICT companies) undergo fast changes with respect to personnel, requirements and resources [Gupta 2009a]. The company is concerned with how to best maintain knowledge which would otherwise be lost. There is some knowledge sharing technology in place, but this may not fulfill the needs of the company. Further studies are necessary to investigate the potential role of collaboration tools (e.g. Wikis, networks, ecosystem) in the company. - COTS/OSS and communities: Earlier investigations of OSS and communities have focused on the current state-of-practice within industry with respect to component selection and integration [Ayala 2007] [Scacchi 2006b]. Future research on current best practices within OSS selection should have a more detailed focus on evaluating benefits versus disadvantages to get the full picture about efficiency, and allow researchers to propose relevant improvements where needed. - Improvements to a company’s software processes: We have made several recommendations for process improvements towards general practitioners and at Conclusion 79 StatoilHydro ASA (see Chapter 5). Additional investigations are needed to study the implementation and effects of these recommendations. - Agile Architecture: As noted in Chapter 2.4, the long-term perspectives involved in software architecture and short-term foci of agile software development should be combined to allow the benefits of both to prevail. Additional studies are necessary to explore how these two paradigms can best be combined. - Further study of the approach to TDD at StatoilHydro: The company aims to investigate the effort used towards handling defects and changes, and to investigate other potential facets of benefits related to TDD. With respect to architectural risks in software evolution (and related to SEVO goal G2): - Study of code-level and other artifact data for software architecture evolution: We want to couple the risks and corresponding risk management strategies we have identified with an investigation of empirical data related to architecture evolution. This can, in the future, facilitate a method framework for better handling of these issues, also on the code-level. - Expand our knowledge on architecture risk management in software evolution: Though several possible concepts and related activities towards effective risk management in CBSE have been proposed, there is a lack of actual empirical studies in the area [Glass 2001]. This means that the actual value and effectiveness of proposed activities and tools remain largely unknown. Our studies represent a start to investigate certain software architecture risk issues in connection with software evolution. We would nevertheless like to expand our survey base to possibly confirm and/or add to our findings thus far. Performing hands-on investigations (e.g. case studies) of actual evolution of software architecture also remains a priority issue. - Thorough investigation of software architecture evaluation methods: Earlier investigations on software architecture analysis [Bass 2007][O‘Connell 2006] have focused on structured analysis outputs as a basis for determining risks. However, the actual methods used for analyzing the software architecture can vary quite a lot [Babar 2007a]. Investigating a wider range of analysis methods will help to discover risk issues possibly missed by earlier studies. - A study of architectural versus non-architectural changes: Failure of the software architecture can cause failure of an entire development project. Better knowledge and understanding about architectural evolution can potentially improve handling of actual software architecture changes, which can cause subsequent changes in many components of a software system [Bass 2004]. Such an investigation should study architectural versus non-architectural changes, with respect to their distribution and possible handling.

31

Acknowledgements

• Thanks go to SEVO (RCN contract number 159916/V30), and NTNU for financial support.

• Thanks to my supervisor, Prof. Reidar Conradi, and colleagues on the SEVO project and at IDI, especially Dr. Jingyue Li, Dr. Anita Gupta, Prof. Letizia Jaccheri and Prof. Tor Stålhane.

• Thanks to international colleagues Dr. Bunse, Dr. Morisio, Dr. Torchiano, Dr. Babar, and Dr. van Vliet for collaboration on the surveys.

• Thanks to StatoilHydro ASAs Harald Rønneberg, Einar Landre and Harald Wesenberg for encouragement during the studies and permission to publish the case study results.


Presenter

Presentation Notes

This thesis comprises a part of the SEVO project (Software EVOlution in Component-Based Software Engineering). The data collection was performed partly at StatoilHydro ASA at Trondheim/Stavanger, and partly in the European IT-industry. Cooperation with other persons is a key part of being able to complete work on a PhD. The environment I‘ve been allowed to be a part of has fostered independent thought along with constructive cooperation. I‘d like to thank my main surpervisor, Dr. Reidar Conradi, for continuous constructive advice and support during my doctoral work. I‘d also like to thank Dr. Jingyue Li, Dr. Letizia Jaccheri, Dr. Tor Stålhane, and all the other members of the Software Engineering group at IDI/NTNU. I‘d also like to recognize the importance of international collaboration on publications I was allowed to be a part of through my colleagues from Germany (Dr. Bunse), Italy (Dr. Morisio, Dr. Torchiano), China (Dr. Jingyue Li), Ireland (Dr. Babar) and the Netherlands (Dr. van Vliet). Thank you also to Harald Rønneberg, Einar Landre, Harald Wesenberg and everyone else I‘ve been involved with at StatoilHydro ASA. I‘d also like to thank all my colleagues with whom I‘ve had the pleasure to share and exchange knowledge and experiences. Finally, I want to thank Sari for her love and continuing support throughout my PhD. I‘d also like to thank my friends and family for their support and encouragement.

component-based software engineering: modern trends ... · • the sevo (software evolution)...

Documents