introduction - macos serversunset.usc.edu/events/2001/feb6-9-01/p61-dean.pdf · source code level,...

ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 61

COTS Workshop: Continuing Collaborations for Successful COTS Development John Dean, National Research of Council Canada

[email protected]} Patricia Oberndorf, Software Engineering Institute

{po(~_sei.cmu.edu} Mark Vigder, National Research Council Canada

fMark.Viader~_nrc.ca~ Chris Abts, University of Southern California

{cabts~sunset.usc.edu} Hakan Erdogmus, National Research Council of Canada

{Hakan.Erdogmus~nrc.ca} Nell Maiden, City University, UK

fN.A.M.Maiden(~citv.ac.uk} Michael Looney, DERA, UK

[email protected]} George Heineman, Worcester Polytechnic Institute

{heineman(~cs.wpi.edu} Michael Guntersdorfer, University of California, Irvine

fmQunters~uci.edu}

Introduction In early June of 2000 a COTS Workshop entitled "Continuing Collaborations for Successful COTS Development" was held in Limerick, Ireland in conjunction with ICSE 2000. The purpose of the workshop was to collect experience reports re- garding the use of commercial off-the-shelf (COTS) software to build systems, identify best-practices for the use of COTS software, and to establish a research agenda for those researchers interested in COTS-based software systems. This one and a half day workshop was an extension of the work begun during the workshop entitled "Ensuring Successful COTS Development" held in conjunction with ICSE '99. Re- sults from that workshop demonstrated that there were a number of common research areas, including acquisition, planning and management, architecture and implementation, and evaluation and testing, for which researchers saw the possibil- ity of collaboration. These areas included specific topics such as estimating the effort required to implement COTS-based systems, classification of architectural styles, and certification of COTS products for reliability and safety. The group will reconvene at ICSE'01 (w_~_ww.c_s_r..u.w_'c;.ca/icse200D to discuss further the results achieved.

The ICSE 2000 Workshop had about 26 participants and was formatted as a combination of plenary sessions and small breakout groups that worked on specific issues related to COTS-based systems. The breakout groups investigated the impact of COTS software usage in the following areas:

• Economic and financial issues. • Requirements definition_ . Software engineering process. • Integration, maintenance and system management. • Business models.

Each breakout group tried to identify the current state of the art in COTS software usage as well as open questions that could provide the basis for fiLrther research in the coming years. Each group was responsible for producing a written

summary of their discussions which are included, without major edit- ing, below. A more complete description of the workshop, as well as all the participants' position papers, can be found at:

http://seg.iit.nrc, ca/projects/cots/icse2000wkshp/index.htnd

Workshop Results The Workshop brought together a diverse group of researchers and practitioners experienced in many areas of software engineering. This allowed the Workshop to identify a number of best practices when building COTS-based systems, as well as directing the participants to problems requiring further research.

A complete summary of the breakout sessions is provided below, but some of the key findings that emerged during the plenary included:

* The Economic and Financial Issues breakout agreed that improved models need to be developed for cost-estimation associated with the entire lffecycle of COTS-based systems. This includes estimating cost, schedule, and effort for each stage of development and maintenance. Significant data gathering is still required for calibration of these models.

As well, managers need models and tools to support making decisions based on economic models associated with the technical aspects of the system. This allows managers to decide, for example, whether to buy or build a component, or whether to upgrade a component now or wait for a future release.

• The Requirements Definition breakout focussed on the relationship between requirements and COTS selection and usage. Given that much of the functionality of a system is being bought, not developed, product selection becomes critically intertwined with requirements definition_ The group used two case studies to ground their discussions and used them as a basis for setting a research agenda for the next three to five years. The most important issue was determined to be appropriate Product Description Languages for COTS products by which integrators can determine operational characteristics of a product within different contexts. The product descriptions would ideally be developed by independent third par-


ties and cover properties of a product well beyond its operational characteristics.

• The Software Engineering Process breakout identified how the development process changes when a system is COTS-based. The number of development activities required and the constraints that exist between them di- rected the discussion. The activities of system architecture and infrastructure design, product selection, and requirements definition all have much more complex relationships among them for COTS-based systems than custom- built systems. There are also new project management functions relating to software licencing and warranty management and COTS product evaluation and upgrade decisions. Many of these issues were discussed as well in the Economic and Financial Issues and the Requirements Definition Breakout.

• The Integration, Maintenance and System Management breakout agreed that the integration costs for COTS-based systems are often higher than for custom-built systems. The group felt that there must be ways that products can be described to allow system builders to determine the integration issues and costs before building the systems. Similar to the Requirements Definition group, they felt that there was a need for independent evaluators to provide profiles of COTS products. Maintenance and management become significant issues as the system is being maintained at the COTS product level rather than at the source code level, and many of the maintenance and management decisions are being driven by the COTS vendor rather than the system builder.

• The Business Models breakout identified all the stakeholders involved in developing a COTS-based system and the various risks faced by each of these stakeholders. They then proposed a set of risk mitigation strategies that can be practiced by each of these stakeholders. Although stir requiring a significant amount of research to complete, they have provided a direction that allows those involved to manage the risk.

Breakout Summaries Each Breakout Coordinator prepared a summary of the results of their individual sessions. These summaries are presented in this section.

Breakout 1: Economic and Financial Issues Chris Abts, (University of Southern California, USA) Hakan Erdogmus, (National Research Council Canada) Patricia Obemdoff (Software Engineering Institute, CMU, USA)

The purpose of the breakout session was to explore possible areas of collaboration that pertain to the economic and financial aspects of COTS-based software development. The discussions were centered on two topics:

1) Application of financial vahmt__ion models to evaluate COTS-based development decisions.

2) The COCOTS cost estimation model.

Application of Financial Valuation Models The COTS approach must provide clear fiuancial benefits for it to be a viable alternative to traditional development Fin.q_ ncial valuation techniques such as Net Present Value and Real Options Analysis aid in this kind of assessment by linking development decisions to their economic impact. Within this framework, the problems being addressed include:

• Comparison of a COTS-based development strategy with a custom development strategy for a given system.

• Comparison of two different COTS-based development strategies for a given system.

• Assessment of the value of flexibility inherent in a COTS-based system.

• Assessment of migration strategies, both from a COTS-based system to a custom system and from a custom system to a COTS- based system.

• Timing and value of COTS product upgrade and system release decisions.

• Management of COTS product license costs to mitigate risk.

CO 7"5 vs. Custom Decisions The viability of COTS-based development with respect to custom development can be assessed using a simple financial model based on the concept of Net Present Value (NPV). In corporate fin_ance, NPV is the gold standard for making capital budgeting decisions. The NPV of a project is given by the sum of its discounted future cash flows. Dis- counting takes into account time value of money, i.e., the notion that money that is yet to be expended or received is worth less today than it is in the future. The NPV rule states that only projects with positive NPV are worth undertaking, and a project with a higher NPV is prefer- able to a project with a lower NPV.

An NPV-based model of software development can compare two development strategies using a metric that captures the economic incen- tive to choose one strategy over the other. The five parameters that a~ect this metric are:

• the ratio of the development cost of the two strategies; • the ratio of the time-to-market of the two strategies; • the ratio of the operation cost of the two strategies; • the ratio of the asset value (a measure of post-development bene-

fits or market value) of the two strategies; and • the market risk of the end product, as captured by the discount

rate.

With the estimates of these pmmneters, it is possible to compare different development strategies. These include not only different types of pure COTS-based and pure custom development sWategies, but also hybrid strategies, and with the use of Real Options Analysis, strategies that start with one type of system and gradually migrate to another type of system. In the latter case, the incremental migration of system components is viewed as a series of options that are exercised provided that the long-tea-a~ benefit of each migration phase exceeds the immediate cost of that phase.

Two rules of thumb that emerge from an NPV-based analysis are:

• When the COTS approach has both siotmificanfly higher asset value and operation cost, increasing product risk increases the in- centive for the COTS approach only if that approach also has sufficient time-to-market advantage over a custom system.


If time-to-market is important and the custom system has significantly higher market value, COTS-to-custom migration strategy is usually most valuable.

Value of Replacement Flexibility One advantage of the COTS approach is flexibility. Flexibility arises from the abitity to replace the COTS components used in the systems at a fraction of the cost normally required to develop those components from scratch. However, to achieve such flexibility, the system must be designed accordingly. This implies an up-front investment in a robust COTS architecture. Such an investment buys the developer the option to replace the COTS products at a reasonable cost in the future. This early investment can be thought of as a risk mitigation strategy to hedge against replacement cost uncertainty. The cost of the extra investment in system architecture is not jusli- fled if it surpasses the value of the replacement flexibility, or the option value of future replacement opportunities. There- fore, it is important to identify the circumstances under which the benefits of flexibility exceed the cost of achieving it.

A state-of-the-art financial valuation technique, Real Options Analysis, can address this issue by mapping the replacement scenario to an option pricing problem. By investing in a robust COTS architecture, the developer is acquiring a future option to upgrade the technologies underlying the COTS products used. When sufficient information is available, the value of this option can be estimated and compared to the amount of architectural investment needed.

Upgrade Decisions In a COTS-based system, upgrading a COTS product should be a discretionary activity. Some upgrades will be minor and incur a relatively small cost, while others will be major and require substantial rework. When the cost of a product upgrade is superior to its long-term benefits, it does not make good economic sense to proceed with the upgrade. This thinking gives upgrade decisions an option-like flavor. How- ever, product obsolescence must also be accounted for when technology evolves at a rapid pace. Delaying an upgrade in- definitely may have serious long-term economic consequences that are not immediately apparent.

In addition, in the presence of multiple COTS products, syn- chrouization of a system's upgrade cycles with the release cycles of the COTS products used may be problematic.

We feel that integer programming combined with Real Op- tions Analysis together can tackle the timing and economic feasibility of upgrade decisions.

Management of COTS Product Licenses Any multi-user system that is based on COTS products will incur license costs. When license fees are volatile, the total cost may be substantial, and their proper management is thus instrumental for mitigating the resulting risk. Special option- like contracts may be negotiated with the COTS vendors, for example to give the developer the future right to acquire a yet unknown number of licenses of a certain COTS product with a cap on the single-license fee. A contract such as this is an asset

for the developer, but a liability for the vendor when license fees ex- hibit a great deal of fluctuation. Therefore the vendor is justified to ask for an up-front premium from the developer in return. The fair value of this premium, again, can be determined by Real Options Analysis.

COCOTS (COCOMO for COTS) Update Cost estimation is a vital input for any economic analysis, including financial valuation. The COCOTS cost model is being developed at USC Center for Software Engineering to address the needs of COTS- based systems. It is one of a suite of specialized models being developed as extensions of the COCOMO (Constructive Cost Model) II cost estimation model. The COCOTS model has features that make it suit- able for effort and schedule estimation in cases where parts of a system are implemented by COTS products.

The two defining characteristics of COTS software, and thus which drive the whole COTS usage process, are: 1) the COTS product source code is (typically) not available to the application developer, and 2) the future evolution of the COTS product is not under the control of the application developer. (The market decides when and how a given COTS product evolves.) These characteristics are the underlying con- stmints that shaped the current form of the COCOTS model.

(Note: COCOTS as described here deals only with initial COTS integration efforts. Subsequent to the COTS workshop at ICSE 2000, work has been done to expand the model to provide estimates of long-term COTS-based system operation and maintenance effort through system retirement. That work is on-going, however, and remains outside the scope of the discussion here.)

COCOTS is actually a collection of four related submodels, each addressing individually what has been identified by USC-CSE as the four primary sources of COTS softwar¢ integration costs (from a system construction activity point of view). These sources include the effort needed to perform (1) candidate COTS component assessment, (2) COTS component tailoring, (3) the development and testing of any integration or "glue" code needed to plug a COTS component into a larger system (or to make it function with another COTS component), and (4) increased system level progranmung due to volatility in incorporated COTS components.

Assessment is the process by which COTS components are selected for use in the larger system being developed. Tailoring refers to those activities that would have to be performed to prepare a particular COTS program for use, regardless of the system into which it is being incor- poratecL or even if operating as a stand-alone item. These are things such as initializing parameter values, specifying input/output screens or report formats, setting up security protocols, etc. Glue code development and testing refers to the new code external to the COTS com- ponem itself that must be written in order to plug the component into the larger system. This code by nature is unique to the particular context in which the COTS component is being used and must not be con- fused with tailoring activity as defined above. Volatility in this context refers to the frequency with which new versions or updates of the COTS software being used in a larger system are released by the vendors over the course of the system's development and subsequent de- ployment.

Schedule Estimator As well as effort, COCOTS will attempt to estimate the associated calendar schedule that wig be required to construct a system using


COTS components. The schedule estimator should define the relationship between development time and effort for a COTS- based system. It is also through the schedule estimator that the connection between COCOTS and COCOMO II is most di- rectly evident. COCOMO II can be used to determine effort associated with those parts of a system that are to be created from scratch or by reusing components for which the source code is available. COCOTS is to be used to estimate the effort associated with those parts of a system that are to be implemented using COTS components. The required schedule needed to complete the entire system (including new, reused, and COTS elements) must be a function of the effort required for all those elements combined. Thus, the use of the CO- COMO II schedule estimator equation is proposed for CO- COTS, but the input effort parameter (defined in person- months) includes both the traditional COCOMO and the additional COCOTS derived effort (as well as a possible recalibra- tion of some of the constants in the schedule model).

Effort and Schedule Distribution Percentages Compared to COCOMO II, the initial COCOTS calibration data set suggests a slightly different distribution of total development effort and schedule across development activities. For our current set of 20 industrial data points, and within our defined CoTS-related activities, preliminary analysis indicates that approximately 8% of the total COTS related effort is dedicated to assessment, 28% to tailoring, 45% to development of glue code, and 19% to activities that pertain to COTS volatility. See the position paper and the accompanying slides for further details.

COCO TS Future Directions As stated previously, currently the COCOTS model covers only the development phase. The initial model is actively being extended to address operation and maintenance activities as well.

The next round of data collection for calibration purposes is expected to begin in fall 2000. A revised version of the model is expected soon afterwards. The revised model will account for operational and maintenance costs in addition to the initial development costs.

As new data becomes available, the initial development model as well as the opemtious and maintenance model parameters will be recalibrated.

Conclusions Constructing a sound business case for or against COTS product usage requires economic models that can deal with the peculiarities of COTS-based system development. Specialized cost estimation models and state-of-the-art financial valuation techniques are expected to help in this endeavor.

Cost estimation and financial valuation are highly comple- mentary in that cost data is a vital input of financial valuation. Whereas cost estimates are traditionally of more direct interest to project managers, financial valuation is of more direct interest to executives and investors who are concerned not only with costs, but also with the overall long-term value.

This breakout session aimed at bridging the gap between these com- plementary approaches by exposing two promising threads of work to each other. We hope that it will lead to fruitful collaborations among the parties involved.

Breakout 2: Requirements Definition & COTS: Setting a Research Agenda for COTS-Based Development 3-5 Years Down the Line

Nell Maiden (City University, UK); Ljerka Beus-Dukic (University of Northumbria, UK); Colin Paul Glasier (City University Dublin, Ireland); Gillian Mallalieu (University of Sunderland, UK); Cornelius Ncube (City University London, UK); Thuy Nguyen (EDF, France); Joan Pastor (Universitat Polictecnica de Catalunya, Spain); William M~ Thomas (The MITRE Corporation, USA)

Structure Of The Breakout Session The work carried out in this breakout session was divided into 3 basic sessions, each addressing a basic problem or issue concerned with defining requirements and evaluating soflavare packages for their f it with these requirements. Firstly, we elicited 2 real-world COTS selection and configuration case studies, to ground the academic discussions and provide a focus for research later in the session. The first of the 2 case studies looked at COTS package selection at EDF, the French electricity supplier, and the second reported problems in adaptation of a COTS software package to include missing funetion_~lity. Secondly, in order to enable us to explore a research agenda for requirements engineering and COTS package selection we brainstormed a vision of the COTS software marketplace 3-5 years down the line. This vision was to act as a baseline for the research agenda. Thirdly we developed a comprehen- sive research agenda for COTS software package selection 3-5 years into the future, with particnlar emphasis on determining what consti- tutes a COTS software package to enable new decision-making capabilities. The following describes each of these 3 sessions in more detail.

Two Cots Product Selection Case Studies The breakout session investigated two case studies reported by its participants.

Selecting COTS Products for Electn'city Power Plants This first case study was reported by Thuy Nguyen from EDF in France. Selecting COTS software packages for electricity power plants is a large, long, and complicated process. Thuy gave us some stark facts. In a typical process of selecting embedded COTS software packages for power plant conlrol, there are 3000 pages of user requirements, up to 85% of the requirements are non-technical requirements, there are 200-300 COTS candidates to choose from, and the selection process can take several years to complete. Furthermore the selection of the fight COTS software package is business critical: EDF lose $1M per day ff the plant is not working. If this problem were not complex enough, there is a need to consider multiple power plant systems: ideally EDF would like to reuse software packages and package evaluation information for design of new and future power plants, however the time-scale of each selection process currently makes this infeasible.

The current COTS software package selection process in EDF is a 4- stage process:


i. Analyse eqm'pment facilities: EDF investigate complex non-functional requirements (e.g., performance requirements) with simple formal models, which also enables them to demonstrate these requirements to other stakeholders. At the same time EDF place contracts on all software pacl~ge vendors to provide them ALL required information for subsequent decision-making phases,

i i . Write the user requirements document: This process can often take 1-2 years. The requirements are often written by domain experts rather than software experts, with implications for the style and content of the requirements. Most user requirements are written as untestable natural language statements. EDF must also take care to keep the requirements' generic', to avoid bias towards one package vendor. For critical requirements, EDF critique these requirements with formal models similar to the use of formal models applied in stage 1. The result of this stage is an invitation-to-tender (ITF) document,

iii. Collect vendor responses and generate system specifications: Vendor responses to the 1Tr provide the basis for writing the system specification, a process that can take a further one year or more. This specification is a specification of the wider socio-technical system, and will include, for example, human accident handling procedures as well as software procedures. Each system requirement is defined as either negotiable or non-negotiable. Responses from vendors are often in two parts: (i) the 'standard' vendor specification for their package, (ii) responses to critical customer requirements;

iv. Select the soRware package: The package selection process is a highly mechanistic one, which is essential to avoid being sued for unfavourable selection decisions. As such, due to this mechanistic nature, EDF see package selection as something distinct fIom requirements engineering.

Thuy reported that EDF find the current process too long, ex- pensive and not 'reusable' to meet new market opportunities such as new power plants to be developed in the developing world. He identified the following requirement that EDF might have to overcome these problems:

Third-party assessors to do the difficult and time-consuming package selection work, guarantee requirements compliance, and offer confidence in this guarantee; furthermore the package vendors should pay for this work.

Tailoring an Enterprise Resource Planning (ERP) Package (The Back-Flushing Problem) This case study came from Gill Mallalieu of the University of Sunderland. She reported back on a UK organisation that had implemented a complex COTS (ERP) product without a certain 'backflush' function. This omission was surprising given the good processes that were followed, inch~ding: effective requirements elicitation, product evaluation, and vendor in- volvement in the process. However, several problems emerged, including:

• lack of top-level management commitment; • lack of ownership of decision-making;

• failure to recognise incremental development and future software versions.

Again, non-technical issues were identified as more important than technical issues in the success of this COTS customisation.

Brainstorming 3-5 Year Market Vision To Baseline Our Research Agenda The next stage of the session established a baseline for long-term research in requirements engineering and package selection in a COTS- driven development process. However, the session faced a problem: setting a research agenda based on current problems and processes might invalidate the agenda if these problems and processes change sig~aificantly over the next 5 years. Hence to establish the baseline the session undertook a brainstorming session to discover future trends in COTS software packages and their use of software development. Ideas were collected on a whiteboard, then structured into themes. The re- suiting market vision was written as the following scenario:

"Both customers and suppliers of COTS software products will use third-party product evaluation agencies to deliver added-value to their selection, customisation and integration processes. These agencies will use in-depth knowledge of customer requirements, vendor products, customers and suppliers, and how to undertake decision-making processes to broker deals and advise their sponsors. These product evalua- tions will be 'lighter' than current product logs, profiles and certifications, in that the agencies will recegnise the diverse range of different situations in which COTS software is used and tailor their advice accordingly. These agencies will emerge bottom-up to seize market share when the opportunities arise, rather than through top- down government or other intervention. R will be just as likely that the agencies will recommend to purchase COTS software as to use Appli- cation Service Providers (ASPs), renting products and ~ n g open source software. Due to the demand and scale of the problem, agencies will maximise the use of semi-automated computer-based techniques that rely on standard requirements and product description languages, that is an agreed lingua-ftanca for deciding about COTS products."

Other identified trends in this market vision not incorporated into the scenario included:

• using COTS products as 'free' prototypes to drive the requirements elicitation process;

• tendering for software services as you tender for other commodi- ties such as financial services;

• greater componentisation of COTS products, leading to components-on-demand;

• the rise of architectures, middle-ware and integration frameworks; • the rise of multi-disciplinary evaluation and decision-making

teams to handle the increasing importance of non-technical factors;

• the emergence of more COTS product case studies and disaster stories, leading to guidelines and success stories for software managers to use;

• understanding that COTS products will evolve constantly; • agencies use novel techniques such as intelligent agents; • who will be responsible in the decision-making process when

third-party agencies will be around?

These trends provided the basis for the positing of the following re. search agenda.

ACM SIGSOFT Software Engineering Notes vol 26 no I January 2001 6

Software Product Description Languages There was strong agreement in the breakout session that the single most important issue related to the market vision was the emergence of standard product description languages. The need to understand the capabilities of a COTS software package under specific operational environments has led to the concept of COTS package profiling. This involves developing over time a description of the known capabilities and other characteristics of the package under consideration. We envis- age that this profiling service will normally be performed by a

third party who will make this information available to COTS-based systems integrators.

The critical question is: what are the discriminating characteristics of a software package? Our attempt to determine these characteristics is shown in the graphic below. Participants played the roles of software product suppliers, customers and brokers, and identified characteristics that were important to their roles in package selection. These characteristics were then grouped according to broader categories.

f V E N D O R : ~ / S O F T W A R E ~ location issues ~ ~ n 6 d o n s , actions and se~ces

basic vendor details; ('~software package architecture; devt process quality (CMM); =jtability and version frequencie= plans, credentials & viability; Y,,fit to inter-oparability standards~ ,,~ future product trends /

batch(es) features

effective architectures; "~ wative/novel configurations ~t to non-functional requir(

& integrations; \1 target environment cult inter-operable s e f f w a m ~

amhitectuml styles

AGENCIES r reqt- product compliance; )

interchangeable products

wider system architecture; : to non-functional requirements

project references; consultant CVs; impl. strategies;

:/STOMER REQUIREMEN expressed reqts for system

DEVT:

relevant user expertise;

, work-arounds, fixes, use of best-of-breed etc /

Discriminating Characteristics of a Software Package

The most important finding is that most important characteristics are not specific to the COTS software. This has profound implications for software product description languages - these languages need to describe more than the software products; for example, they need to describe the wider system architecture, the system environment, previous product implementations and the customer development process related to the software product. Furthermore, the session ranked the characteristics according to the level of difficulty to obtain these characteristics reliably, where l=simple and 3--difficult. The results of this ranking are also shown on the graphic above - most information was perceived to be difficult to elicit and use.

Finally, the session discussed issues relating to software prod- uet description languages. Important issues included:

• recognise the strong inter-dependencies between software product characteristics that provide a basis for multi- criteria decision making (MCDM).

• q~_mntitative and qualitative decision-making; • make qualitative (e.g., requirements) and quantitative

(e.g., through measurable fit criteria)

• multiple techniques needed, but need to know how to use which technique when sensitivity analysis is too complicated.

We look forward to advancing this work on standard product descrip, tion languages.

Breakout 3: C O T S . B a s e d Sof tware Eng ineer ing Process Michael Looney (DERA, UK) Shane Lunga (DERA, UK) Mark Mosco (University of California, Santa Cruz, USA) John Dean (National Research Council Canada)

The group started off by discussing why there was a need to identify development process and identifying the scope of what the gron 1 might address within that subject.

It was accepted that the process for COTS component based systen development was different from that which had been applied for thq development of bespoke systems and that there was a need to formali~ that process. It was also agreed that the development process uncle consideration would only be applicable at the 'system level' and wouh not apply to the actual development of the COTS components them selves. It was felt that this was a task carried out by component ven dors, and at that level there are already well-established soflwar, development processes in place.


It was agreed that, although component selection was a sig- niiieant part in the development process, it was also the subject of one of the other groups and so would not be addressed in any detail.

From the experiences of the group members it was clear that there was a range of systems which needed to be considered, and it was accepted that what might be appropriate at one end of the spectrum might be overkill at the other. This resulted in the conclusion that it was necessary to identify a basic process that could be specified as a set of activities or stages that might be elaborated to a greater level of detail when dealing with the more complex systems.

At this point there was also some discussion on the relationship between the initial development and the support/maintenance cycle. The consensus was that this should be looked at once some form of initial process had been identified. There was also an extended discussion on the role of system framework or architecture to provide for the independ- ence and cooperation of the components and on where this came into the development process. The feeling was that it might be of considerable importance in a larger more complex system where it might be necessary to extend the system use over time and to replace components within that framework as

Specify

Project Management, Configuration Control

Implementation/ Acceptance/

Fielding 4 -

required due to obsolescence. If the system was relatively small and was not expected to evolve or be supported over any extended period, it was felt the framework would be of less importance than the selection of the components. This subject was taken further during the later, more detailed discussions on the identification of possible solutions to meet the system requirements and con_straints.

The discussion on the basic stages for the process started with requirements and the need to identify the appropriate stakeholders for the system and to get their 'buy in' to what was being undertaken as the first stage. This was then followed by the identification of possible/feasible candidate solutions using the requirements and any con- swaints either from the first stage, as part of the requirement, or as a result of looking at the various options. An example of this might be that the selection of one possible component as a part of the solution might imply the use of a second compatible product and prohibit the use of other alternatives. The main objective of this stage would be the consideration of the technical options and how best the system re- qnirements might be achieved from that point of view. This would then be followed by some form of prototyping to achieve actual measures of acceptability agnin~ the overall requirements. If there were a 'perfect match' with a selected set of components, which meet all the criteria, then it would be a simple step to implement the system by integrating the parts and carrying out the necessary system acceptance activity.

Requirements/ 1 Constraints q I

Produce set of feasible System Designs

t l Analyse System Negotiate Trade off

~ - ~ a t Technical System level Prototype selution

Presentation of Alternative / Options for Decision Maldng 4 - - . J

Process at Business level

Basic Development Process Representation

If there were two or three possible technical solutions or con- sideratiorts of a non-technical nature, then the next stage would be some form of business level assessment of the op- tious. This would look at factors outside the main technical areas such as the 'Wade off' of cost against functionality or the need to be 'in line' with the company strategy. If no acceptable solution was achieved for whatever reason at this point, then the initial requirements will need to be reconsidered and the process repeated. If there is an acceptable solution, then the f i l l stage for the development would be the implementation and acceptance followed by the fielding of the system.

The discussion then identified the common threads, which would be needed throughout all of the activities, including those associated with the project management and control of such a development approach. While it was accepted that the appropriate levels of configuration management and version control would be needed, there were some additional factors that had never been present in the development of bespoke systems, those of licenses and warranties. It was agreed that the concerns in the areas configuration and version control should be addressed at the system level with the collection of information on the component relationships done within a system context rather than at the component level. This subject was also covered during the support/maintenance discussions. There was some discussion on the rela-


tionship between the vendors and the developers during the whole process, and it was agreed that it was a significant factor in how well some of the component assessment could be carried out. But, as the selection process discussion was with another group, this aspect was not taken further.

Following this identification of the basic form of the process there was some discussion on the ways in which it might differ for the range of possible systems and the levels of elaboration that might be necessary. In the task of identifying the set of possible technical solutions there are several possible activities that need to be carried out, the most obvious being the selection of the appropriate components. Whether that is the first activity is not at all clear and could, to a large extent, be dependent on the type, size and complexity of the system under

development. It is under these circumstances that the need for elabora tion arises. If the system is a simple one, with no requirement for lon~ term support, then the selection of the components could well be con. sidered the main task of this stage, with little or no effort being spent h identifying the architecture/framework and the quickest most simplistic approach being adopted for the integration of the components. How, ever, if the system is large and complex then it may be necessary t( look at a wider set of issues before attempting to select the compo nents. It might be appropriate to consider a system architeo ture/framework as well as the interface sta.d~rds to be preferred. Als~ it might be appropriate to give some consideration of the approach tc be adopted for the glue code software as shown in the figure below which might comprise legacy parts as well as COTS products ant some new custom items.

VuUy Wrapped

Legacy com

New Development

Item

~ Newly developed component

Architecture/framework

mq A d d e d / M o d i f i e d

Interface

Glue code COTS component

Illustration of the Use of the Glue Code with the Different Components

It might also be appropriate to consider how system complexity will vary depending on how well the various parts of the system fit together and the requirements imposed by the possible need to support the system over some extended period.

It was agreed that the aspects covered by the choice of architecture/framework, the interface standards, and the glue code were the keys to the long term support of any system and that through life costs would be very dependent on making the

i ........... Derived Constraints S/W, H/W, Socio/economic Human Factors

I

right decisions during the initial development. It was also fel~ that during this stage other socio/economic factors associate¢ with the technical options would be considered. These mighl relate to the selection of the human factors related issues or t( previous system compatibility, but whatever the source the3 would act as constraints on the possible solution space.

Selection of A rch./Framework Identification of Glue Code Standard Identification of Interface Standards 4 - - - -

I Im pac t Analys is , C o m b i n a t o r y issues

t

Componen t Set Selection ..................... ]

Elaborated Version of the Produce Set of Feasible System Designs Activity

The final activity considered under this stage of the process was the consideration of impact and combinatory issues.

Again the level of effort needed to address this might be de. pendent .on the type of system under development but for the


larger more complex ones this would be an essential part of the support process if not the initial development. Any new or

components, the ripple effect. Is the new component larger, slower, faster? These are all characteristics that might impact the final system capability and must be checked in the system context to demonstrate the 'usability' of the new component and its relationship with existing components. Many products satisfactorily pass stand-alone tests but fail the performance requirements when integrated as part of a larger system. Worse still, unknown side effects caused by interdependencies could bring the system to a stop.

Two specific requirements were identified that might have a significant impact on the amount of elaboration required in certain areas of the process if they were stated as being necessary. These were safety and security, but due to the limited time it was not felt appropriate to explore these further during this workshop.

of production and loss of revenue that would need to be mini- mised. This raised the question of whether it was necessary to have a 'back up' system on which to carry out the testing required during the upgrading of the system. While this was considered important, it was not taken further during the discussions.

The next point made was that as there was already a system, the second stage was much reduced in that it was basically the identification of the component or components to be changed

replacement component considered for integration nmst be assessed for its impact on the other existing

The group then went back to the question of whether the process for development and that for support for a system were the same. This went over a lot of the ground again in terms of the various activities that were needed as part of the support process. The general conclusion was that while there were basically the same stages, in the support process the constraints were different. The most obvious one is that there was a system in existence with the various versions of the component, which had been originally selected in place but which was being 'up- graded' for some reason or other. Therefore there were components and a framework that were functioning and for which any new changes made should not degrade current capability or performance. This was considered to be of significance because, for any production line system, there was a problem of down time caused by the necessity for testing or acceptance. This would entail some loss

and that if the framework had been selected satisfactorily then integrating the new component would not be too onerous a task. However, it was felt that with the possible divergence of a product being used from the needs of the system over time, there might be a more complex set of considerations to undertake as part of the upgrade; more akin to the original development process but with many more constraints giving a much reduced solution space.

Selection/Qualif ication Criteria

System Requirements /Constraints

Illustration of the Reduced Solution Space as the Selection Criteria and Constraints Become More Precise

The conclusion reached on the support process was that while the basic stages for support were very similar to those for development there were differences and that it might be possible

to consider these as changes to the level of elaboration or detail at each stage. But it was felt that the area still warranted further study and in- vestigation.


The final area of discussion arose from the input of some of the latest SEI thoughts on how the process would function and the way in which it might be used 'for real' by any large project. If the system was large and complex then it was essential to identify interface standards and retain all of the relationships between the components in order to provide the information required to allow through life support to take place. This leads to the need for some sort of project database to hold all of this information. Consideration was given to the sorts of information that could be held, why it would be needed and how it might be used. Due to the limited time available it was accepted that this was not something that could be addressed to any level during this workshop but that there were several aspects of it that might be worth following up at some other opportunity, While it was obvious that such things as the requirements and constraints should be captured it would also be

essential to keep details of interfaces and component dependencies so that during the maintenance some form of software agent could be used to identify the impact of selecting any new component. There would be other similar occasions when it might be possible to provide automated checking or high lighting of conflicts thereby reducing the chances of in~oducing errors. Another factor considered significant was to identify and capture cost information at as early a stage as possible, perhaps even 3~inst the requirement but certainly against the component such as the various license fees etc.

As a final conclusion the group thought that progress had been made and that while no final definitive process model had been generated there was now a much better understanding of what was required and how some of it fitted together. It was hoped that the activity now started would continue and that people would follow up on what had been achieved in this very important area.

; Specify /

. . . . . . . . . "J~ Req~drements/ : - - ' - - , ' l , Constraints ~t %/ ,

Group Inputs / .

: !

Produce set of feasilte Project Management, ~ I [ ~ . ~ System Desigm Configm'ation C o n t r o i ~ O j e c t I - ~ - - - -

I , i

Maintenance I l~OOeSs cycle . . . . l~

I I I

! I I I

Architecture/ Framework/ In ter face t Glue Code

Componmt Selection Task

Socio/economlc issues ImpacffComHnatory issues

. . _ . - . - - "

I Implementation/ - Acceptancd

Fielding

4 • m ¢ m

i

V i Analyse System Negotiate Trade of f at Technical System level Prototype solution

I m

Presentation of Alternative I Options for Decision Making 4t , - Process at Business levd Process F low - ' - ~

Information Flow ~-

A Possible Full Process Model for the Development and Support of COTS Component System

Breakout 4: Integration, Maintenance and System Management

Leigh Davis (University of Tulsa, USA) Jerry Gao (San Jose State University, USA) George Heineman (WPI, USA) Alok Mehta (AFS, USA) Robert Seacord (Software Engineering Institute, CMU, USA) Mark Vigder (National Research Council Cavsda)

We started with a focus on non-functional properties of COTS software

a) security • confidentiality • identification/authentication • authorization

b) perform~n~ c) platform (e.g., which operating system, which chip) d) technologies (e.g., Java, CORBA, DCOM)

If COTS software was specified using these non-functional properties (provided by the vendor, then a third party could independently verify these properties.

The trouble is that non-functional properties are often dependent on fimctional properties. Industry lacks a performance model for COTS

ACM SIGSOFT Software Engineering Notes vol 26 no i January 2001 Page 71

software, and such a model would be helpful to predict performance of COTS software.

When rating the properties for COTS software, an application builder can make tmdeoffs. How is this to be reflected in the properties for a product?

1. How can we detect in advance a conflict with using a particular COTS software system? It should be possible to identify candidates that can be disqualified. If there were a standard mechanism for specifications (and this would be a big cultural change)...

a) How detailed would these specifications be? (shallow vs. deep)

b) CoUld we develop algorithms to search potential COTS software being considered for use and detect known problems? • The collection of specifications does not need to

be centralized in a database. • It canbe incrementally updated. • The specifications can be in heterogeneous lan-

guages if we have sophisticated searching algorithms.

2. How can we determine IN ADVANCE integration costs?

a) What are the problems with integration? • Mismatch in interfaces (detected by compiler) • Inserting COTS software that can break other

parts of the system (i.e., the COTS software is not at fault)

• Architectural mismatch (i.e., Ocker- bloom/Ccarlan)

• Tremendous glue code needed (no reusable wrappers)

• Costs that depend on the system into which COTS software is being injected.

b) This problem defies simplification. • No standard means of composing COTS soft-

w a r e .

c) Does the use of COTS lower integration costs? • We agreed that using COTS will increase inte-

gration costs, and this goes against the idea of using COTS to reduce overall lifecycle costs.

d) Configurable COTS software will increase integration costs.

e) Since COTS vendors focus on feature differentiation, often COTS soRware is chosen because of a particular nifty feature. Instead, an organization should select a COTS software feature based on "the risk to the system if the feature is not available."

3. How can we lower the integration costs?

a) Organizations should build expertise in particular black-box integration methods. This more than anything will help lower the costs of integration by man- dating the use of tried methods.

b) Since the integration of COTS software is often a failure point, the evaluation process for selecting COTS software should include a prototype integration effort into a toy system.

c) Rapid prototyping integration points, therefore, are suggested as a means to (1) verify the ability to integrate the COTS software at all; and (2) attempt to measure what the overall costs will be.

4. How can we mitigate the risk of maintenance costs?

a) You must never forget that upgrading is driven by the COTS vendor.

b) Could the industry select a consistent monitor- ing/instmmentation standard to make it possible to track the use of COTS software within a system and determine where defects are occurring? • The only way this would work is if the "value added" of

the standard mechanism were accepted by huge numbers of vendors.

5. How do we do system management of COTS systems?

a) Configuration Management is addressed at the source code level.

b) Version checking at the COTS soRware level (i.e., this version of COTS system A works with any version number 2.5 and higher for COTS product B)

c) There is a constant threat that an upgrade of COTS software will alter its interface.

(I) There is a lack of utilities to manage such COTS systems. How can multiple in~allafions be maintained? How can we mange/control the system without direct user intervention (i.e., automatic scripts)? How do we deal with license expira- tion and re-installations?

e) There will always be a tmdeoff between upgrading to include the new version of COTS software or keeping and dealing with an older version that is known to work. Try to develop business model for selecting technology - only upgrade COTS soRware if benefits exceed costs.

6. Does Open Source COTS Software change anything?

a) There are options available to the integrators, and it's easier to find defects.

b) We must avoid the temptation to modify the code and thereby create a long-term internal maimenance cost.

7. Are there success stories7

a) Flight Simulation software for F18 aircraft b) Domain-Specific Software Architectures

8. Recommendations

a) Databases and compilers have benchmarks. Can one be developed to help COTS?

b) Work with Underwriters Laboratories (UL, Inc.) to find how to develop data sheets for COTS software

Breakout 5:Business Models Aria Alvarez (European Software Institute, Spain) Michael Guntersdoffer (University of California, Irvine, USA) Rouald Kohl (AverStar, Inc., USA) Maurizio Morisio (University of Maryland, USA) Jeffrey Voas (Reliable Software Technologies, USA)

The purpose of the breakout session on Business Models was to:

• identify today's problems of COTS software components and how they relate to their weak business model;

ACM SIGSOFT Software Engineering Notes vol 26 no 1 JanuarY 2001 Page 72

• discuss how those problems are currently mitigated; • identify shortcomings in those mitigation strategies in

order to identify the areas with need for improvement; • determine where improvement could be offered by the

sofi~are engineering community; and • identify the corresponding research areas and topics. The breakout was carried out in two sessions. The first session was used to identify both essential and accidental problems of COTS components and their business model, as defined by Brooks. Groups of people (parties) that are affected by these problems (including the linkage of individual problems to individual parties), and how mitigation is carried out today in each case were identified. The second session was used to identify the problems where mitigation is insufficient and where software engineers could come to the rescue. Finally, corresponding research topics were identified.

Session I The following typical problems/risks of COTS components and their application were identified:

1) A component reveals too much functionality, i.e., the component exceeds the specification and offers additional functionality in addition to the one specified or needed by the purchaser.

2) A component reveals too little functionality, i.e., the component cannot provide all the functionality specified or needed by the purchaser.

3) A component's documentation is incomplete, i.e., functionality, features, or any other behavior of the component is not described in its documentation.

4) A component's documentation is incorrect, i.e., the documentation is indeed flawed and error prone, containing incorrect statements about the component's behavior.

5) The component's vendor is not viable, i.e., the seller of the component is driven out of business and cannot further maintain the product or support the customer. The buyer is therefore left on its own.

6) The component's functions are unreliable, i.e., the buyer of the component cannot depend on the component to carry out its tasks correctly.

7) An operational assumption mismatch occurs, i.e., the creator and the user (both parties are software developers here) have different opinions on how the component functions. This is the famous architectural mismatch coined by Garlan.

8) A component vendor decides to limit or stop supporting a component, i.e., even though the vendor is still in business and capable of supporting the customers, he decides to dump a product, e.g., due to lack of profitability.

9) .A component turns out to be volatile, i.e., the component has frequent downtimes where it doesn't function properly (e.g., crashes) and cannot be used at all.

10) The system needs diverges from the component's capabilities, i.e., over time the vendor's ongoing development of the component results in versions with different functionality from the needs of the purchaser, whose system is also evolving.

11) The composer won't integrate, i.e., the purchaser's system integrators are unable to integrate the component into

their system even though the component is technically sound. 12) In case of the occurrence of problems with a component, often

another problem occurs: Who is to blame? For example, consider a system that is composed of components from more than one vendor. In the case of interaction problems between two components bought from different vendors, it is likely that both vendors will deny responsibility and point fingers at each other.

The different types of parties affected by these problems were identified as

v. The procurement personnel of the purchaser (buyer side). vi. The user of the system, which integrates the component

(buyer side). vii. The system integrators (buyer side). viii. The publisher (seller side). ix. The (system) maintainers (buyer side). x. The COTS analyst (independent, third party). The problems (or risks) were assigned as relevant to each party as fol- lows:

c) 2,5,8,10,12. d) 1-10,12. e) 1-12. f) 2 , 4 , 6 , 8 , 9 , 1 1 , 1 2 . g) 1 -6, 8 - 10, 12. h) 4&5.

The following mitigation strategies of each party were identified as being current "state-of-the-art" for each case:

A:

B:

2. Lawsuit (breach of contract), return to vendor, use alternative component.

5. Understand vendor options, lawsuit (bankruptcy law).

8. Understand support options, lawsuit (breach of contract, if applicable).

10. Purchase alternative product, purchase additional functionality, hire personnel to write additional functionality.

12. Threat of lawsuit (to all parties), contact third party to decide (arbitration), use internal resources to decide.

1. Write user procedures (guidelines for users, what not to use and how).

2. Use another product, request more (missing) functionality.

3. Document yourself, request better (complete) documentation.

4. Document yourself, request better (correct) documentation.

5. N/A (nothing the user can do about that).

6. Modify usage procedures, report failure.

7. Report failure, change hardware/operating system profile, if applicable.

8. Complain to vendor.

9. Complain to vendor.

10. Complain to system integrator.


12. Provide more data to determine culprit. C,E:

1. Wrappers, firewalls, procedures for users, request product modificatiorL

2. Request product modification, extend product via wrappexs to add functionality or via other products.

3. Same asB3.

4. Same as B4.

5. Find competing alternatives, plan to build in-house, contract to put source code into escrow.

6. Find competing alternatives, plan to build in-house, request fixes.

7. (C only) same as C6 + B7.

8. Same as B8 + subcontract support to 3rd party.

9. Same as B9 + refuse to upgrade.

10. Same as C7 + stand down on the requirements.

11. (C only) Return, find alternatives, request modification from vendor, wrappers.

12. Hire third party expert, collect data via instnnnenta- tion.

D:

2. Add functionality, deny, better requirements.

4. Fix, deny.

6. Fix, deny, improve V&V, better requirements.

8. Improve vendor support, subcontract (outsource) support, make explicit to customers that support is limited,

F:

spin-off product.

9. Listen to customers, modify release schedules.

11. Use standards, redesign, test for additional configurations, offer on-site integration support.

12. Surveillance instrumentation by vendor, hire third party, complex licenses that protect publishers, lawsuit.

4. Hire a third party expert, ask around to validate documentation claims, in-house test lab.

5. Do independent analysis of company or product.

Session Ih The following software engineering research topics were derived from the results of session I:

I) Detecting hidden functionality and hidden interfaces (see also: dormant code, malicious code). Targets risks C3, C4, C11, C12, E3, E4, E12, and F4.

2) How to determine that a COTS component is unreliable (creation of oracles, creation of test infrastructures). Targets risks C6, C9, C11, C12, E6, E9, and E12.

3) Waystocrcatewrappcrs. Targctsrisks CI, C2, Cll, El, andE2. 4) Definition of volatility metrics. Targets risks C9, C12, E9, and

El2. 5) Instrumentation of binaries to collect information about existing

behavior. Targets risks C3, C4, C6, C7, C9, CI0, C11, C12, E3, E4, E6, E9, El0, El2, and F4.

6) Definition of metrics for similarity of competing products. Targets risks At0, B2, C5, C7, C9, C10, Cll, E5, E9, and El0.

7) Definition/development of metrics/gauges for intcroperability. Targets risks C5, C7, C9, CI0, Cli, C12, D7, DI I, E5, E9, El0, and E12.

introduction - macos serversunset.usc.edu/events/2001/feb6-9-01/p61-dean.pdf · source code level,...

Documents