A Systematic Review on Strategic Release Planning Models

Mikael Svahnberg∗, Tony Gorschek, Robert Feldt, Richard TorkarSaad Bin Saleem, Muhammad Usman Shafique

Blekinge Institute of TechnologyPO Box 520, S-372 25 Ronneby SWEDEN

{Mikael.Svahnberg | Tony.Gorschek | Robert.Feldt | Richard Torkar}@bth.seAveeator@gmail.com, Ushafique786@yahoo.com

November 13, 2009

Abstract

Context. Strategic release planning (some-times referred to as road-mapping) is animportant phase of the requirements engineer-ing process performed at product level. It isconcerned with selection and assignment ofrequirements in sequences of releases such thatimportant technical and resource constraintsare fulfilled.Objectives. In this study we investigatewhich strategic release planning models havebeen proposed, their degree of empirical vali-dation, their factors for requirements selection,and whether they are intended for a bespokeor market-driven requirements engineeringcontext.Methods. In this systematic review a numberof article sources are used, including Com-pendex, Inspec, IEEE Xplore, ACM DigitalLibrary, and Springer Link. Studies areselected after reading titles and abstracts todecide whether the articles are peer reviewed,and relevant to the subject.Results. 24 strategic release planning modelsare found and mapped in relation to eachother, and a taxonomy of requirements selec-tion factors is constructed.Conclusions. We conclude that many modelsare related to each other and use similartechniques to address the release planningproblem. We also conclude that several

∗Contact Author

requirement selection factors are covered inthe different models, but that many methodsfail to address factors such as stakeholdervalue or internal value. Moreover, we concludethat there is a need for further empirical vali-dation of the models in full scale industry trials.

Keywords. Strategic Release PlanningModels, Systematic Review, Road-mapping,Requirements selection factors.

1 Introduction

The idea of selecting an optimum set of fea-tures or requirements to deliver in a releasewithin given constraints is called Strategic Re-lease Planning or road-mapping [1, 30]. Thepurpose of strategic release planning is to bal-ance between competing stakeholders’ demandsand benefits for the developing organisation ac-cording to available resources [30]. Strategic re-lease planning is a complex problem, as appro-priate understanding of planning objectives andother technical and non-technical constraintsare required for a good release plan [30, 32].

A strategic release plan is refined and re-planned after execution of a release as a con-sequence of updates and feedback from cus-tomers, defects in the previous release, marketfactors, new customer demands and other tech-nical and non-technical requirement selectionconstraints [1, 30]. Strategic release planning is

1

considered important for both bespoke as wellas market-driven software products [32, 7]. Inthe context of bespoke products (i.e., where thecustomers are known and actively involved inthe requirements engineering process), strate-gic release planning is useful for selecting themost valuable requirements of a customer in thefirst release and of diminishing importance infuture releases [4, 1]. In the context of market-driven products (i.e., where there is not a di-rect contact with customers) the importance ofstrategic planning is vital, as it helps in decid-ing which customer (among many competingcustomers) will get what features or require-ments and in which release. Thus, strategic re-lease planning relies extensively on selecting theright requirements to guide a product’s evolu-tion and to keep the product aligned with com-pany strategies [3, 8].

There are different approaches to developa strategic release plan and update this planthrough post release analysis [37, 34, 1]. Ad-hoc planning and systematic planning are twobasic approaches used for strategic release plan-ning. Some models are developed by com-bining traditional ad-hoc and systematic ap-proaches named as hybrid approaches [32], butmost models discuss release planning from dif-ferent perspectives and consider different tech-nical and non-technical factors of requirementsselection [34, 1, 13, 20]. Various models use asystematic (e.g., Cost-Value Approach for Pri-oritising) and some use a hybrid (e.g., Evolve*)approach for release planning [20]. A few mod-els are appropriate for strategic release plan-ning with a limited planning scope (one or tworeleases in advance) and others are useful with-out any planning scope limitation [32]. Somemodels have appropriate tool support and theseare considered useful in industrial settings, butthere are also several models that have no toolsupport and those that are not validated in in-dustry [34]. Among the validated models a feware partially validated in industry and some arebeing used in industry, such as e.g. Evolve (im-plemented in the form of the ReleasePlannertool) (see e.g. [13, 20]).

Each of the available strategic release plan-ning models is based on different technical

and non-technical factors of requirements se-lection [33]. Technical factors includes develop-ment tools, existing system architecture, tech-nical precedence among requirements, featuresto include in a release, quality requirements(like security, performance, maintainability),requirements volatility, reusability and inter-dependencies (functionality, value and imple-mentation oriented interdependency) betweenrequirements [19, 20]. Non-technical factorsincludes product strategy, business strategy,company strategy, product value, stakeholdervalue, priority of requirements set by stake-holders, maturity of the product, market place,required and available effort to implement re-quirements, delivery time of release, develop-ment cost estimation [4, 9, 7]. Producing a scal-able strategic release planning model that dealswith a few of these factors at a time is challeng-ing. Considering technical together with non-technical factors in a holistic manner [7] is evenmore challenging.

A comparative analysis of existing models/approaches proved that most of the organi-sations are still using ad-hoc approaches forstrategic release planning even for their largeproducts [32, 16], and thus the models proposedfor release planning are not commonly adoptedin industry. Saliu & Ruhe [34], tried to sum-marise these facts about release planning mod-els, but they only analysed seven models withrespect to a specific system and their scope waslimited to models presented by academia. Thecurrent research aim in the area of strategic re-lease planning models appears to be to improveand validate existing models or approaches [33].For example models such as Evolve+ is an im-proved version of Evolve*, where more require-ments selection factors are included, and ap-propriate tool support is also included in thisversion [1, 19, 13]. In terms of validity, modelsare being validated in different industrial casesto analyse the appropriateness of models in dif-ferent situations [32, 13, 20].

In order to assist product managers in theirchoice of which method to use for strategic re-lease planning, and what to consider in their de-cision, there is a need to know which models areavailable and their contribution towards strate-

2

gic release planning. This is the contributionof this article: A systematic review of availablestrategic release planning models, their stateof validation, and what requirements selectionfactors they propose. From an industry prac-titioner standpoint the results can be used toassess what factors the models use as input,but also to what level the model has been eval-uated in industry. From an academic stand-point, the results can be used to map currentstate of the art and to contemplate what modelinput factors that are currently supported ormissing, which may be valuable input for fu-ture improvement work.

The research questions are thus as follows:

RQ1. What strategic release planningmodels have been presented?RQ2. What technical and non-technical requirements selection fac-tors are discussed in models foundthrough RQ1?RQ3. To what extent have the strate-gic release planning models in RQ1been validated?RQ4. Are the models from RQ1 in-tended to be used in a market-drivenor a bespoke context?

In this study, we use an inclusive definitionof “strategic release planning”. Most releaseplanning methods are developed with the nextrelease (or project) in mind. However, used ap-propriately they can become a product manage-ment tool for long term release planning. Giventhat few release planning methods have a moreholistic view when it comes to product plan-ning (combining company, product, and projectviews [7])[32] we have chosen to include ratherthan exclude methods. We thus adopt the def-inition used by Al-Emran and Pfahl [1]:

Strategic release planning aims at as-signing features to subsequent releasessuch that technical, resource, risk andbudget constraints are met. Once astrategic release plan has been gener-ated, i.e., a decision has been madeon which features are to be developedin which release, operational release

Table 1: Search Terms Construction Process

Step1 Major terms are formed from the research ques-

tions by identifying the population, intervention,outcome, context and comparison

2 By altering the spellings, identifying alternativeterms and synonyms of major search terms

3 By checking the keywords in some papers we al-ready have

4 Boolean OR is used for incorporating searchterms of alternative spellings and synonyms

5 Boolean AND is used to link the major terms withother terms and for combing different terms

planning focuses on the developmentof the identified features in a singlesoftware release.

The remainder of this article is organised asfollows. In Section 2 we describe the planningphase of this systematic review. Section 3 ex-plains how this research was executed, and theresults of the systematic review. This is anal-ysed in relation to the research questions in Sec-tion 4. The results are briefly discussed in Sec-tion 5, and the paper is concluded in Section6.

2 Planning

In this section we describe the planning of thesystematic review. We discuss the search strat-egy, the data sources used, the inclusion andexclusion criteria, and the overall methodologyused to obtain the results.

2.1 Search Strategy

All search results were documented to make thesearch process transparent and replicable [14].For this purpose a systematic review search logwas maintained. Similarly we kept track of se-lected studies and rejected studies.

Search terms were formulated in collabora-tion with a librarian. For constructing thesearch terms the steps in Table 1 was followedas suggested in [10], resulting in the set ofsearch terms presented in Table 2.

In this study, we used the databases listed inTable 3. In addition, we also scanned the jour-

3

Table 2: Search Terms

Search Term1 Release plan2 Release planning3 Planning release4 Software release plan5 Software release planning6 Planning software release7 Strategic software release plan8 Strategic software RP9 planning strategic software release

10 retrospective / post release analysis11 Requirements selection12 Selecting requirements13 Analysing software release defects14 Managing software release15 1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8 OR 916 5 OR 1117 5 OR 6 OR 1118 7 OR 8 OR 9 OR 1119 5 OR 6 OR 7 OR 8 OR 9 OR 1220 11 OR 1221 4 OR 5 OR 6 OR 7 OR 8 OR 9 OR 13 OR 1422 {1,2,4,5,7} AND 1223 {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} AND {Models,

framework, Methods prototype, criteria, Tech-niques, Approaches}

24 {4, 5, 6, 7, 8, 9} AND Industry25 {1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14} AND

Market-driven26 {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14} AND

Decisions

Table 3: Databases Used

Database NameIEEE XploreACM Digital LibrarySpringer LinkScience Direct (Elsevier)Engineering Village (Compendex, Inspec)Wiley Inter ScienceBusiness source premier

nal “International Journal of Hybrid IntelligentSystems”.

2.2 Study Selection Criteria andProcedures

Basic and detailed inclusion/exclusion criteriawere defined for including studies and then se-lecting the most related studies for the purposeof data extraction. The basic inclusion crite-rion is to identify studies related to strategicsoftware release planning models, a frameworkor a study with relevance to a strategic releaseplanning model, a framework of post release

Table 4: Detailed Inclusion and Exclusion Cri-teria

Study Inclusion Criteria1 The article is peer reviewed.2 The article is available in full text.3 The article can be a literature review, system-

atic review, case study, an experiment, industrialexperience report, survey, action research or com-parative study.

4 The article discuss a model / framework of strate-gic release planning or post release analysis ofstrategic release planning.

5 The article will be included if it gives an overviewof models / frameworks of strategic release plan-ning or post release analysis of strategic releaseplanning.

6 The article will be included if it compares twoor more models / frameworks of strategic releaseplanning or post release analysis of strategic re-lease planning with each other.

7 The article will be included if it evaluates or anal-yse an existing model of strategic release planningor post release analysis.

8 The article will be included if it discuss a vali-dation of existing model of strategic release plan-ning or post release analysis.

Study Exclusion Criteria1 Articles that do not match the inclusion criteria

will be excluded.2 Articles related to only operational release plan-

ning will be excluded.3 Articles related to re-planning of a release on op-

erational level will be excluded.

analysis of strategic release planning, or anystudy related to a model framework of strate-gic release planning or post release analysis ofa strategic release plan.

Detailed inclusion/exclusion criteria are pre-sented in Table 4. These were applied to thestudies identified using the basic inclusion cri-terion. From question 3, it can be discernedthat literature reviews and systematic reviewswere also to be included. Our strategy for deal-ing with these later was to use them to findthe original studies that, in turn, should fit thebasic and detailed inclusion/exclusion criteria,and not to include them in any further analysis.During the study, however, it became clear thatwe did not need to apply this strategy since wedid not find any literature reviews or system-atic reviews.

Studies were selected individually by the re-searchers by applying the basic and detailed in-clusion/exclusion criteria. The included stud-ies were double-checked through discussions

4

Table 5: Quality Criteria

Quality Criteria1 Is an appropriate introduction of strategic release

planning or post release analysis of strategic re-lease planning provided?

2 Is the research methodology clearly defined andappropriate for the problem under consideration?

3 Is the design of the study clearly stated and doesit have proper conceptual argumentation basedon references?

4 Does the research methodology map to study de-sign, the study design to research questions, andthe research questions to conclusions?

5 Are validity threats related to study results re-ported?

6 Are negative findings related to the model re-ported?

7 Are any restrictions or limitations on results ofthe study reported?

among the researchers. The basic and de-tailed inclusion/exclusion criteria were appliedas follows. First, the basic inclusion criterionwas applied by reading the titles, keywordsand abstracts of all studies. If a study satis-fied the conditions of the basic inclusion cri-terion then the study was included, and oth-erwise excluded. Second, the detailed inclu-sion/exclusion criteria were applied on the thusfar included studies’ abstracts, conclusions, in-troductions and sources of publication.

2.3 Quality Assessment

Along with inclusion/exclusion criteria, it isalso important to assess the quality of the in-cluded studies [14]. The purpose of quality as-sessment in this research is to further under-stand the limitations of each individual studyduring data synthesis. The criteria listed inTable 5 were used to evaluate the quality of se-lected studies, as recommended in other studies[14, 15, 10].

The quality criteria were used as a checklistwhile extracting data from the selected studies,and each question was answered with Yes orNo. The quality assessment result of a partic-ular study is summarised in Section 4.3. More-over, since this was mostly used as an internalquality measure to fully understand and beingable to compare the state of validation betweendifferent studies (i.e., aid us in answering RQ4),

these assessments are further explained in thedata extraction form for each study.

2.4 Data Extraction

We used a standardised data collection form toextract relevant information to answer the re-search questions. In Table 11 we present theData Extraction Form. Extracted data wasdouble-checked by two of the authors to elim-inate uncertainties. A pilot study was per-formed on the data extraction form to ensurethat it worked before conducting the full scalesystematic review. Some difficulties were foundand resolved through discussion among the au-thors. In case of multiple publications of thesame data, the most recent results were usedfor data extraction and synthesis.

3 Execution and Results

The process of searching for studies thatmatch the basic inclusion/exclusion criteriawas performed individually, although the inclu-sion/exclusion decisions were double-checkedand discussed at each stage of execution.

Two literature resources (electronicdatabases and one journal) were scannedin this systematic review, and this was done intwo separate phases. In the first phase eachelectronic database was scanned by applyingthe search terms. The basic inclusion/exclusioncriterion was applied on the found results andrelated studies were selected. The informationabout the total number of results found fromeach electronic database against each searchterm, selected articles and rejected articles ateach stage (by reading the title only and byreading the title and abstract) were logged inthe systematic review search log. All searchterms were applied on the specified electronicdatabases and a total of 12541 results wereretrieved.

From the 12541 studies 3804 studies were ex-cluded by just reading the title. The titles andabstracts of the remaining 8737 studies wereread and the basic inclusion and exclusion cri-teria were applied, leaving 904 studies in theinclusion set. Finally, duplicates were removed,

5

Table 6: Results Found per Database

DB Name TotalFound

TotalSelected

1 Engineering Village(Compendex, Inspec)

3678 369

2 IEEE Xplore 636 1343 ACM Digital Library 2711 1264 Springer Link 2123 1645 Science Direct 1370 346 Wiley-Inter Science 421 357 Business Source premier 1602 42

Sum: 12541 904

resulting in 124 remaining studies. In the nextstage, the detailed inclusion/exclusion criteriawere applied on the selected 124 studies. Thisresulted in the selection of 27 relevant studies.The other 97 studies were excluded.

In the second phase, one journal was manu-ally scanned and one additional relevant study[31] was found.

In summary, 28 relevant studies were foundthrough systematic review. Table 6 presentsthe number of results retrieved per database(total found and total selected), and Figure 1illustrates the systematic review process. Table7 lists the selected articles.

4 Analysis

4.1 Available Strategic ReleasePlanning Models

RQ1. What strategic release planning modelshave been presented?

28 studies related to strategic RP modelswere found through the systematic review. Inthese studies, 24 models of strategic releaseplanning are presented and the rest of the stud-ies are related to validation of some of the pre-sented models. Specifically, study number 15 isa validation of study 6, study 17 a validationof study 19, study 21 a validation of study 22.Moreover, study 25 presents the same model asstudy 28. Thus, 24 models are presented.

Of the 24 models found, 10 models are ex-tensions of other models and 14 are originalmodels, as can be seen in Figure 2. It should,however, be noted that the original models are

also often based on existing ideas and tech-niques. 22 of the models are used for strate-gic release planning or road-mapping and onemodel (PARSEQ [12]) is related to post releaseanalysis of a strategic release plan. Anothermodel (QIP [2]) is used for strategic releaseplanning process improvement. It is also no-table that most of the research on strategic re-lease planning models has been done within thelast ten years.

A further analysis shows that there are threemain groups of strategic release planning mod-els; those that are related to the EVOLVE-family and the ReleasePlanner tool [20], thosethat are created by the SERG research group atLund University, and those that are related tothe Centre for Organization and Information atUtrecht University. In addition, there are a fewother publications that are not related to anyof these three main categories. Of the threecategories, the EVOLVE-family is by far thelargest, including 16 of the presented models.

Figure 2 presents a map of the strategic re-lease planning models. In this figure, we use thename of the model, where it has been named,and the title of the article where the model isnot named. The models from Utrecht Univer-sity are based on a model not related to strate-gic release planning, which we have includedwithin parentheses in the figure. Each model isalso identifiable via a number that correspondsto the id number in Table 7.

4.2 Requirement Selection Fac-tors

RQ2. What technical and non-technical re-quirements selection factors are discussed inmodels found through RQ1?

All models provide different solutions ofstrategic release planning and discuss differentrequirements selection factors. Some of themodels categorise requirements selection fac-tors into groups, but most of the models donot discuss any categorisation of factors, butrather give a description and use of factors inthe model. There are many common require-ments selections factors among the majority ofidentified models.

6

Figure 1: Systematic Review Process

In order to assist the analysis, we have cre-ated a taxonomy of the factors used by the dif-ferent strategic release planning models, as pre-sented in Figure 3. This taxonomy is mainlycreated using terms and collections of termsused in the included studies. For example, thedivision into Soft Factors and Hard Constraintsis influenced by the requirements selection fac-tors used in Evolutionary Evolve+ [20].

In Figure 3 we include information (in per-cent) about how many of the models that ad-dress each factor type, and in Table 8 wepresent which requirements selection factorsthat are addressed by each study. In thistable, we list the factors used in the origi-nal study, together with a translation to theconstructed taxonomy of requirements selec-tion factors. Below, we briefly describe eachof the factors further, and also introduce theacronyms used in Table 8.

Hard Constraints Hard Constraints includethose factors that may restrict the order

and time when certain features or require-ments can be implemented. The hardconstraints include Technical Constraints,Budget and Cost Constraints, ResourceConstraints, Effort Constraints, and TimeConstraints.

Technical Constraints (TeC) These con-straints deal with constraints in therequirements themselves and the abil-ity to implement them. For examplerequirements dependencies, which isa sub-category to this category, andthe extent to which an existing systemneeds to be modified to accomodate therequirement.

Requirements Dependencies (RD)Involves all constraints regarding the or-der in which requirements or features canbe implemented, as well as dependenciesthat may influence the cost or value ofrequirements.

7

5. EVOLVE

2. EVOLVE+ 26. Evolutionary EVOLVE+

6. EVOLVE*

3. Quantitative Win-Win

24. Release Plan Simulator

23. Bi-Objective Release Planning for Evolving Systems

7. Quality Improvement Paradigm

18. Explain Dialogue

13. Art and Science of Release Planning

8. Release Planning under Fuzzy Effort Constraints

11. Fuzzy Structural Dependency Constraints in Software Release Planning

12. Measuring Dependency Constraint Satisfaction in Software Release Planning using Dissimilarity of Fuzzy Graphs

9. S-EVOLVE*

14. EVOLVEext

16. F-EVOLVE*

1. CVA28. Supporting Road-Mapping of Quality Requirements

4. An Analytical Model for Requirements Selection Quality Evaluation in Product Software Development

19. PARSEQ

(Optimizing Value and Cost in Requirements Analysis)

10. Determination of the Next Release of a Software Product: An Approach using Integer Linear Programming

27. Software Product Release Planning through Optimization and What-if Analysis

22. A Decision Modeling Approach for Analyzing Requirements Configuration Trade-offs in Time-Constrained Web Application Development

20. A Risk-Driven Method for Extreme Programming Release Planning

Figure 2: Map of Strategic Release Planning Models

Requirements Dependencies (75%)

Quality Constraints (8.3%)

Stakeholders' Influence Factors (29.2%)

Value Factors (37.5%)

Risk Factors (12.5%)

Resource Consumption Factors (20.8%)

Hard Constraints (100%)

Soft Factors (58.3%)

Technical Constraints (87.5%)

Budget and Cost Constraints (29.1%)

Resource Constraints (33.3%)

Effort Constraints (50%)

Time Constraints (16.7%)

Figure 3: Taxonomy of Requirements Selection Factors

8

Table 7: Articles Selected from the Systematic Review

Id Ref Year Study Name1 [11] 1997 A cost-value approach for prioritising requirements2 [29] 2003 Quantitative studies in software release planning under risk and resource constraints3 [28] 2003 Trade-off analysis for requirements selection4 [27] 2003 An analytical model for requirements selection quality evaluation in product software development5 [9] 2004 Software release planning: an evolutionary and iterative approach6 [31] 2004 Hybrid intelligence in software release planning7 [2] 2004 Intelligent support for software release planning8 [21] 2004 Release planning under fuzzy effort constraints9 [34] 2005 Supporting software release planning decisions for evolving systems10 [35] 2005 Determination of the next release of a software product: an approach using integer linear program-

ming11 [22] 2005 Fuzzy structural dependency constraints in software release planning12 [23] 2005 Measuring dependency constraint satisfaction in software release planning using dissimilarity of

fuzzy graphs13 [32] 2005 The art and science of software release planning14 [30] 2005 Strategic release planning and evaluation of operational feasibility15 [19] 2006 Release planning process improvement – an industrial case study16 [17] 2006 Decision support for value-based software release planning17 [13] 2006 Case studies in process improvement through retrospective analysis of release planning decisions18 [6] 2006 An explanation oriented dialogue approach and its application to wicked planning problems19 [12] 2006 Introducing tool support for retrospective analysis of release planning decisions20 [18] 2006 A risk-driven method for eXtreme programming release planning21 [38] 2007 An experiment with a release planning method for web application development22 [38] 2007 A decision modelling approach for analyzing requirements configuration trade-offs in time-

constrained web application development23 [33] 2007 Bi-objective release planning for evolving software systems24 [24] 2007 A system dynamics simulation model for analyzing the stability of software release plans25 [26] 2007 A quality performance model for cost-benefit analysis of non-functional requirements applied to

the mobile handset domain26 [20] 2008 A systematic approach for solving the wicked problem of software release planning27 [36] 2008 Software product release planning through optimization and what-if analysis28 [25] 2008 Supporting road mapping of quality requirements

Quality Constraints (QC) Constraints onquality requirements (or non functional re-quirements), legal requirements, etc.

Budget and Cost Constraints (B&CC)All constraints that restrict the budget.Typically, these are expressed as costconstraints.

Resource Constraints (RC) Constraintson the amount of resources that may beused during development.

Effort Constraints (EC) Constraints on theamount of effort that can be spent duringa development instance.

Time Constraints (TiC) Constraints thatmandate that certain requirements are re-leased, resources used, or costs inflicted atcertain times.

Soft Factors Soft Factors include those fac-tors that are more difficult to estimate and

provide exact numbers on, but may causecertain features or requirements being pri-oritised higher than others. The soft fac-tors include Stakeholders’ Influence Fac-tors, Value Factors, Risk Factors, and Re-source Consumption Factors.

Stakeholders’ Influence Factors (SIF)All factors that deals with stakeholders’ability to influence the release planning.

Value Factors (VF) Factors that help in as-sessing or maximising the value of a re-lease.

Risk Factors (RF) Factors that help in as-sessing the risk of requirements or features.

Resource Consumption Factors (RCF)This factor includes the estimated amountof resources that a requirement will con-sume. This can then be matched against

9

several of the constraints such as budget &cost, effort, as well as resource constraints.

In summary, we can observe that all modelsdeal with at least some of the hard constraints,whereas 40% do not deal with any of the softfactors. In addition, we can see that most mod-els (87.5%) address technical constraints, and50% of the models address effort constraints.

Since the EVOLVE family is the largestgroup of strategic release planning models, itis interesting to study the requirements selec-tion factors addressed by these models. Ananalysis reveals that although there is a largeremphasis of the hard constraints, members ofthe EVOLVE family also address all of the softfactors. What is interesting, however, is thatEVOLVE itself addresses most of the soft fac-tors, whereas its direct derivatives (EVOLVE+and EVOLVE*) do not address any of them.Only its “grandchildren” reappear among thesoft factors, and only four of its “cousins” (i.e.[28, 32, 33, 24]) address any of the soft factors.

4.3 State of Validation

RQ3. To what extent have the strategic releaseplanning models in RQ1 been validated?

In Figure 4 and Table 9 (using the id’s fromTable 7 as index) we present details of themodel validations. Please note that studies 15,17, 21, and 25 are not included separately inthis list, but are instead combined with theirrespective original model papers. The qualityof each study is further indicated in Table 10,where the quality criteria from Table 5 are ap-plied.

We can thus see that 23 models (96%) arevalidated and only one model (4%) is not val-idated. 56% are validated in industry, and40% are validated in academia. Almost ev-ery model validated in industry is validatedthrough case studies, except one model thatis validated through an experiment. The casestudies are carried out by conducting interviewswith practitioners and by testing models in in-dustrial contexts on a set of real requirementsfrom industry.

This means that 44% of the models, i.e. thosethat are not validated or validated in academia,

are never validated in any industry setting, noteven pilot projects. More alarmingly, most ofthe models (including those that are validatedin industry) are validated in a limited scale, i.e.only a few case studies are performed, or thevalidation is performed on only a small set ofrequirements. Reports of actual full scale in-dustrial use (even in pilot projects) are onlyavailable for four of the 24 models.

For some models validation details are miss-ing or not provided at all. It is thus difficult forreaders to understand and trust the results ofthe model validations. This may also preventindustrial organisations to adopt a model, asresults of model validations can not be gener-alised and the models are not validated in anindustrial setting. Furthermore, only a few ofthe validation studies report negative findingsas well as positive findings.

4.4 Intended Context

RQ4. Are the models from RQ1 intended to beused in a market-driven or a bespoke context?

Most of the presented models provide deci-sion support regarding requirements’ selectionfor market-driven software development. Thereason for this is likely to be that delivering aproduct in releases and developing road-maps(strategic release planning) is an important andcommon phenomena for market-driven softwaredevelopment. The results show that 83% of themodels are considered to be useful for both be-spoke and market-driven software developmentand the remaining 17% are appropriate only formarket-driven software development.

5 Discussion

Despite the importance of release planning andthat the complexity of the task requires sup-port of methods, models, and tools, we findthat there are in fact few real choices for prac-titioners that wish to adapt a release planningmodel. There are few models to begin with, andthere are many family ties between the modelsthat are proposed. Moreover, one of the keydrivers for release planning in a market-drivencontext is to consider value for the customer,

10

Table 8: Requirements Selection Factors per Article

Hard Constraints Soft FactorsTechnical Constraints

id Factors in Article RD QC TeC B&CCRC EC TiC SIF VF RF RCF Sum1 Cost

ValueStakeholders’ Satisfaction

1 1 1 3

2 Requirement DependencyRequirement Effort EstimationRisk FactorsResource Constraints

1 1 1 1 4

3 Stakeholder PreferencesEffort ConstraintsTime ConstraintsQuality Constraints

1 1 1 1 4

4 Requirement DependencyBudget RestrictionsRequirements Decomposition

1 1 2

5 Required Effort EstimationRequirement DependencyStakeholder EvaluationMinimum Release PenaltyMaximum Release Benefit

1 1 1 1 1 5

6 Requirement DependencyRequired Effort EstimatesResource ConstraintsBudget Constraints

1 1 1 1 4

7 Requirement DependencyRequired Effort EstimatesResource ConstraintsBottleneck Resource Constraints

1 1 1 3

8 Fuzzy ConstraintsRequirement DependenciesEffort Constraints

1 1 2

9 Stakeholders’ ValueStakeholders’ SatisfactionTechnological ConstraintsResource ConsumtionsCapacity Bounds on ResourcesSystem’s Constraints

1 1 1 1 1 1 6

10 Requirement DependencyRequirements’ Projected ValueRequirements Resource Claim per De-velopment Team

1 1 1 3

11 Structural ConstraintsEffort Constraints

1 1 2

12 Requirement DependencyRequired Effort ConstraintsResource Constraints

1 1 1 3

13 Feature DependencyStakeholders’ InterestsAvailable ResourcesFeature Prioritisations

1 1 1 1 4

14 Requirement DependencyStakeholder ValueTime to MarketRequirement Volatility

1 1 1 1 4

16 Resource Capacity ConstraintsTime ConstraintsFeature Dependency ConstraintsImplementation CostAnnual Revenue per Requirement

1 1 1 1 1 5

18 Requirements Precedence ConstraintsRequirements Coupling ConstraintsResource ConstraintsPre-assignment ConstraintsEffort Estimation

1 1 1 3

19 CostValue

1 1 2

20 Requirement DependencyValue in terms of Cost or RevenueCost of ImplementationEffort per IterationBusiness Value

1 1 1 1 4

22 Time EstimatesRequirement DependencyUrgency of Implementing a Requirement

1 1 1 3

23 Value from Business PerspectiveRisk of Implementing a FeatureFeature Dependency

1 1 1 3

24 Availability of ResourcesRequired Effort

1 1 1 3

26 Soft ConstraintsHard Constraints

0a

27 Development by one pool of developersDevelopment teamsTeam transfersExternal resource or dead-line extensionRequirements Dependency

1 1 1 3

28 Quality of non-functional requirementsCost of non-functional requirements

1 1 2

Sum 17 2 1 7 10 12 4 6 10 3 5a In fact, we interpret this method to be generic enough to cover all categories, especially given that thefactors used in the article are the influence for the two major categories in our taxonomy.

11

Figure 4: Model Validation

Table 9: Model Validation Details

id Validation Details1 Two Industry Case Studies (Interviews with Project Manager)2 Industry Case Study (20 requirements, 5 stakeholders), Academic Experiment3 Industry Simulation (10 requirements)4 Industry Case Study (Survey, 33 respondents)5 Industry Experiment (20 requirements)6 Two Industry Case Studies (First on 30 requirements and 3 stakeholders, Second is deployment in a

company)7 Industry Case Study (25 requirements, 5 stakeholders)8 Academic Case Study (30 requirements, 5 stakeholders)9 Industry Case Study (49 requirements, 6 stakeholders)10 Academic Simulation(9, 24, 99 requirements, 3, 17, 17 teams respectively)11 Academic Simulation (25 requirements)12 Academic Simulation (10 requirements)13 Academic Case Study (15 features, 2 stakeholders)14 Industry Case Study (Interviews)16 Industry Case Study (30 requirements, 2 stakeholders)18 Not Validated19 Two Industry Case Studies (deployment in companies)20 Industry Case Study (deployment on a project)22 Academic Experiment (63 participants)23 Academic Case Study (33 requirements, 3 stakeholders)24 Academic Simulation (8 features, 6 stakeholders)26 Industry Case Study (50 requirements, 6 stakeholders)27 Academic Experiment (2 software packages)28 Industry Case Study (Interviews, deployed in company)

value for the company, and value at a givenpoint in time, and decide upon a roadmap orrelease schedule according to this in order tomaximise the profit of the company. However,our study finds that as many as 40% of themodels do not focus on these soft factors, butonly emphasise the hard constraints.

Furthermore, it is still a challenge to find

models that are thoroughly validated; few mod-els are tested in full scale industry trials. Onecritical issue for strategic relase planning mod-els is that of scaleability; a model may workwell with a handful of requirements but may beimpractical to use when the number of require-ments and decisions grow. Moreover, strate-gic release planning models may be challenged

12

when the requirements are less than perfect,and when stakeholder availability can not be as-sumed. In addition, studies in requirements en-gineering (e.g. [5]) indicate that the accuracy ofe.g. requirements prioritisation may depend onwhether the system in question matters to thesubjects. Thus, while carefully planned aca-demic evaluations with large sets of industrygrade requirements may test certain aspects ofstrategic release planning models, the only realtest would be to apply the models in real largescale industry cases. As shown in Section 4.3,these kinds of evaluations are rare.

The consequence of this is that it may bedifficult to find a release planning model thatsuits a company’s needs and addresses the de-sired requirements selection factors, and wherethere is also adequate empirical evidence thatthe method works as intended in full scale in-dustry trials.

5.1 Strengths and Weaknesses

A large number of articles have been covered inthis systematic review in order to extract thearticles listed in Table 7. We can thus be fairlysure that the systematic review actually coversthe strategic release planning models that havebeen published to date. A possible weaknessis the search terms (Table 2) that can, at leastfrom a logical perspective, be simplified furtherand also be made more general. To make themas generic as possible, and in order for them tofit the databases used, they were developed incollaboration with a librarian. English is notthe native language of any of the involved re-searchers in this study, so there is – as always– a risk that some of the papers have been mis-interpreted during any of the involved stages.To counter this, all decisions and results weredouble checked by at least one other person.

6 Conclusions

With the increase of market-driven software de-velopment, there is also an increasing need forconducting strategic release planning. Severalwell known models for strategic release plan-ning exist, as well as several not so well known

models. When deciding on how to conductstrategic release planning, it is important to beable to make an informed decision, weighingthe strengths and weaknesses of several mod-els against each other. To this effect, the sys-tematic review presented in this article identi-fies the existing models and studies them fromthree aspects: (1) Their coverage of require-ments selection factors, (2) Their degree of val-idation, and (3) their applicability for market-driven or bespoke software development. Thesethree aspects are selected since they togethergive a good understanding of whether a modelwould be applicable in a given situation or not.

We have analysed the models in the aggre-gate in order to see overall trends. To this end,we present the found strategic release planningmodels in an overview map (Figure 2). More-over, we have created a taxonomy of require-ments selection factors used (Figure 3).

The principal findings from the systematicreview on strategic release planning are:

• 24 strategic release planning models havebeen presented in academic papers. 16 ofthese belong to the EVOLVE family of re-lease planning models.

• Most methods focus on a limited set of re-quirements selection factors, with an em-phasis on hard constraints. Approximately58% of the models also include soft factors.

• Most of the presented models are vali-dated, approximately half in industry andhalf in academia, and a large majority withthe help of case studies (80%). Validationon full scale industry projects is scarce.

• All models are intended for market-drivensoftware development. All but two mod-els can also be used for bespoke softwaredevelopment.

Future work includes studying the use ofstrategic release planning models in industry.This will serve as an additional source of vali-dation for the models found in this systematicreview, and may also reveal other factors thatneed to be addressed. A more in-depth anal-ysis of the identified strategic release planning

13

Table 10: Quality of Selected Studies accordingto Quality Assessment Criteria

Id 1.

Isin

troducti

on

pro

vid

ed?

2.

Isth

ere

searc

hm

eth

odolo

gy

defined?

3.

Isth

edesi

gn

ofth

est

udy

state

d?

4.

Isth

est

udy

desi

gn

cohesi

ve?

5.

Are

validity

thre

ats

report

ed?

6.

Are

negati

ve

findin

gs

report

ed?

7.

Isth

ere

any

rest

ricti

ons

or

lim

itati

ons

report

ed?

1 N N N N N Y Y2 Y Y Y Y Y N Y3 N N N N Y Y Y4 Y Y Y Y Y N Y5 Y N N Y N N N6 Y N N N N Y Y7 Y Y Y N N N N8 Y N N N Y N Y9 Y N N N N Y N10 Y N N N N N Y11 Y N Y N Y N Y12 Y N Y N Y N Y13 Y Y N N Y N Y14 Y Y Y N Y N Y15 Y Y Y N Y Y N16 Y N N N N N N17 Y Y Y Y Y Y Y18 Y N N N N N Y19 Y Y N Y Y Y Y20 Y N N N Y Y Y21 Y Y Y Y Y Y N22 Y N N N N N N23 Y Y Y Y Y Y Y24 Y Y Y N Y Y Y25 Y Y Y N Y Y Y26 N Y Y Y Y N Y27 Y N N N Y Y Y28 N N N N N N YY 24 13 13 8 18 13 21N 4 15 15 20 10 15 7

models, and the process used in each of themis also part of future work.

Our recommendations can be divided intotwo parts; recommendations for model mak-ers, and recommendations for model users. Formodel makers, we suggest an increased atten-tion on soft factors and an increased attentionto empirically validate the models. For model

users, the level of empirical validation of modelsshould be carefully analysed before deciding touse a specific model. Moreover, we suggest thatmodel users define which requirements selectionfactors are needed, and to use this as a guidinginput when selecting a strategic release plan-ning model, since it cannot be expected thatany model support all types of requirements se-lection factors.

References

[1] A. Al-Emran and D. Pfahl. Operationalplanning, re-planning and risk analysis forsoftware releases. In Proceedings of the8th International Conference on Product-Focused Software Process Improvements(PROFES 2007), Lecture Notes in Com-puter Science LNCS 4589, pages 315–29,Berlin Germany, 2007. Springer Verlag.

[2] A. Amandeep, G. Ruhe, and M. Stanford.Intelligent support for software releaseplanning. In Proceedings of the 5th Inter-national Conference on Product FocusedSoftware Process Improvement (PROFES2004), Lecture Notes in Computer ScienceLNCS 1840, pages 248–262, Berlin Ger-many, 2004. Springer Verlag.

[3] A. Aurum and C. Wohlin, editors. En-gineering and Managing Software Require-ments. Springer Verlag, Berlin Germany,2005.

[4] S. Barney, A. Aurum, and C. Wohlin. Aproduct management challenge: Creatingsoftware product value through require-ments selection. Journal of Systems Ar-chitecture, 54(6):576–593, 2008.

[5] P. Berander. Using students as subjectsin requirements prioritization. In Proceed-ings of the 2004 International Symposiumon Empirical Software Engineering (IS-ESE’04), pages 167–176, Washington DC,2004. IEEE Computer Society.

[6] G. Du, J. McElroy, and G. Ruhe. Ad-hoc versus systematic planning of softwarereleases: A three-staged experiment. In

14

Proceedings of the 7th International Con-ference on Product-Focused Software Pro-cess Improvement (PROFES2006), Lec-ture Notes in Computer Science LNCS4034, pages 335–340, Berlin Germany,2006. Springer Verlag.

[7] T. Gorschek and A. M. Davis. Require-ments engineering: In search of the depen-dent variables. Information and SoftwareTechnology, 50(1-2):67–75, 2008.

[8] T. Gorschek and C. Wohlin. Requirementsabstraction model. Requirements Engi-neering Journal, 11:79–101, 2006.

[9] D. Greer and G. Ruhe. Software releaseplanning: an evolutionary and iterativeapproach. Information and Software Tech-nology, 46(4):243–253, 2004.

[10] J. Hannay, D. Sjøberg, and T. Dyba. Asystematic review of theory use in softwareengineering experiments. IEEE Transac-tions on Software Engineering, pages 87–107, 2007.

[11] J. Karlsson and K. Ryan. A cost-value ap-proach for prioritizing requirements. IEEESoftware, 14(5):67–74, 1997.

[12] L. Karlsson and B. Regnell. Introduc-ing tool support for retrospective analy-sis of release planning decisions. In Pro-ceedings of the 7th International Confer-ence on Product-Focused Software Pro-cess Improvement (PROFES 2006), Lec-ture Notes in Computer Science LNCS4034, pages 19–33, Berlin Germany, 2006.Springer Verlag.

[13] L. Karlsson, B. Regnell, and T. The-lin. Case studies in process improvementthrough retrospective analysis of releaseplanning decisions. International Jour-nal of Software Engineering and Knowl-edge Engineering, 16(6):885–915, 2006.

[14] B. Kitchenham. Procedures for perform-ing systematic reviews. Technical ReportTR/SE-0401, Keele University, 2004.

[15] B. Kitchenham, E. Mendes, andG. Travassos. Cross versus within-company cost estimation studies: Asystematic review. IEEE Transactionson Software Engineering, pages 316–329,2007.

[16] M. Lindgren, R. Land, C. Norstrom, andA. Wall. Key aspects of software re-lease planning in industry. In In Pro-ceedings of the 19th Australian Conferenceon Software Engineering, 2008 (ASWEC2008), pages 320–329, Washington DC,2008. IEEE Computer Society Press.

[17] S. Maurice, G. Ruhe, O. Saliu, and A. Ngo-The. Decision Support for Value-BasedSoftware Release Planning, pages 247–261.Springer Verlag, Berlin Germany, 2006.

[18] L. Mingshu, H. Meng, S. Fengdi, andL. Juan. A risk-driven method for ex-treme programming release planning. InProceedings of the 28th International Con-ference on Software Engineering (ICSE2006), pages 423–430, Los Alamitos CA,2006. IEEE Computer Society Press.

[19] J. Momoh and G. Ruhe. Release planningprocess improvement – an industrial casestudy. Software Process: Improvement andPractice, 11(3):295–307, 2006.

[20] A. Ngo-The and G. Ruhe. A systematicapproach for solving the wicked problem ofsoftware release planning. Soft Computing- A Fusion of Foundations, Methodologiesand Applications, 12(1):95–108, 2008.

[21] A. Ngo-The, G. Ruhe, and S. Wei. Releaseplanning under fuzzy effort constraints. InProceedings of the Third IEEE Interna-tional Conference on Cognitive Informat-ics, pages 168–75, Los Alamitos CA, 2004.IEEE Computing Society.

[22] A. Ngo-The and M. O. Saliu. Fuzzy struc-tural dependency constraints in softwarerelease planning. In Proceedings of the 14hIEEE International Conference on FuzzySystems, pages 442–447, Los Alamitos CA,2005. IEEE Computer Society Press.

15

[23] A. Ngo-The and M. O. Saliu. Measur-ing dependency constraint satisfaction insoftware release planning using dissimilar-ity of fuzzy graphs. In Proceedings of theFourth IEEE Conference on Cognitive In-formatics (ICCI 2005), pages 301–307, LosAlamitos CA, 2005. IEEE Computer Soci-ety Press.

[24] D. Pfahl, A. Al-Emran, and G. Ruhe. Asystem dynamics simulation model for an-alyzing the stability of software releaseplans. Software Process Improvement andPractice, 12(5):475–490, 2007.

[25] B. Regnell, R. Berntsson-Svensson, andT. Olsson. Supporting road-mapping ofquality requirements. IEEE Software,25(2):42–47, 2008.

[26] B. Regnell, M. Host, and R. Berntsson-Svensson. A quality performance modelfor cost-benefit analysis of non-functionalrequirements applied to the mobile hand-set domain. In Proceedings of the 13thInternational Working Conference on Re-quirements Engineering: Foundation forSoftware Quality (REFSQ 2007), Lec-ture Notes in Computer Sience LNCS4542, pages 277–91, Berlin Germany, 2007.Springer Verlag.

[27] B. Regnell, L. Karlsson, and M. Host. Ananalytical model for requirements selec-tion quality evaluation in product softwaredevelopment. In Proceedings of the 11thInternational Conference on RequirementsEngineering, pages 254–263, Los AlamitosCA, 2003. IEEE Computer Society Press.

[28] G. Ruhe, A. Eberlein, and D. Pfahl.Trade-off analysis for requirements selec-tion. International Journal of SoftwareEngineering and Knowledge Engineering,13(4):345–366, 2003.

[29] G. Ruhe and D. Greer. Quantitative stud-ies in software release planning under riskand resource constraints. In Proceedingsof the 2003 International Symposium onEmpirical Software Engineering (ISESE

2003), pages 262–270, Los Alamitos CA,2003. IEEE Computing Society.

[30] G. Ruhe and J. Momoh. Strategic releaseplanning and evaluation of operational fea-sibility. In Proceedings of the 38th AnnualHawaii International Conference on Sys-tem Sciences (HICSS 2005), page 313b,Washington DC, 2005. IEEE ComputerSociety Press.

[31] G. Ruhe and A. Ngo-The. Hybrid intelli-gence in software release planning. Inter-national Journal of Hybrid Intelligent Sys-tems, 1(1-2):99–110, 2004.

[32] G. Ruhe and M. O. Saliu. The art andscience of software release planning. IEEESoftware, 22(6):47–53, 2005.

[33] M. O. Saliu and G. Ruhe. Bi-objective re-lease planning for evolving software sys-tems. In In Proceedings of the 6th JointMeeting of the European Software Engi-neering Conference and the ACM SIG-SOFT Symposium on the Foundations ofSoftware Engineering (ESEC/FSE 2007),pages 105–114, New York, NY, 2007. ACMPress.

[34] O. Saliu and G. Ruhe. Supporting soft-ware release planning decisions for evolv-ing systems. In Proceedings of 29th AnnualIEEE/NASA Software Engineering Work-shop, 2005., pages 14–26, Washington DC,2005. IEEE Computer Society Press.

[35] J. M. V. van den Akker, S. Brinkkemper,G. Diepen, and J. Versendaal. Determina-tion of the next release of a software prod-uct: an approach using integer linear pro-gramming. In Proceeding of the 11th Inter-national Workshop on Requirements Engi-neering: Foundation for Software Quality(REFSQ 2005), pages 119–24, 2005.

[36] M. van den Akker, S. Brinkkemper,G. Diepen, and J. Versendaal. Softwareproduct release planning through opti-mization and what-if analysis. Informationand Software Technology, 50(1-2):101–111,2008.

16

[37] C. Wohlin and A. Aurum. What is im-portant when deciding to include a soft-ware requirement into a project or release.In Proceedings of the International Sympo-sium on Empirical Software Engineering,Noosa Heads, Australia, 2005.

[38] S. Ziemer and I. C. Calori. An experi-

ment with a release planning method forweb application development. In Proceed-ings of the 14th European Conference onSoftware Process Improvement (EuroSPI2007), Lecture Notes in Computer ScienceLNCS 4764, pages 106–17, Berlin Ger-many, 2007. Springer Verlag.

17

Table 11: Data Extraction Form

Fundamental Information1 Data Extractor2 Data Checker3 Date of Data Extraction4 Article Title5 Authors? Name6 Application Domain7 Journal/Conference/Conference proceedings8 Retrieval Search Query9 Date of publication

Specific Information10 Study Context Academia

Industry11 Research Methodology Literature Review

Systematic ReviewCase StudyExperimentSurveyAction Research

12 Study Subjects ProfessionalStudents

13 Validity Threats Conclusion ValidityConstruct ValidityInternal ValidityExternal Validity

RQ 1 What strategic release planning modelshave been presented?

14 Name of presented model / framework15 Model / Framework proposed in Literature or in

industry16 Newly presented model / framework or extension

of already developed model/ framework17 Means of representation (table, diagrammati-

cally, mathematical means, logically)18 Description of presented model19 On what grounds the model / framework is con-

structed20 Model or framework use in Industry21 Any requirement selection technique used in the

model22 Any limitation of the model / framework23 Practical application of model / framework in the

form of tool24 Discussion about any other RP model / frame-

work

RQ 2 What requirements selection factors are dis-cussed?

25 What technical and non-technical requirement se-lection factors are discussed

26 Any other name of technical and non-technicalrequirement selection factors

27 Common requirements selection factors discussedin two or more than two models / framework.

RQ 3 To what extent have the models been vali-dated?

28 Evidence of validation of the proposed Model /framework (static validation or dynamic valida-tion)

29 Model / framework is validated in academia30 Model / framework is validated in industry31 Model / framework is validated in both academia

and industry32 Model / framework is validity threats33 Model / framework is statically validated or im-

plemented in industry

RQ 4 Which models have been used for bespokeand market-driven software development?

34 Model / Framework is proposed for bespoke only35 Model / Framework is proposed for market-driven

only36 Model / Framework is proposed for both kinds of

products37 Model / Framework is adopted (in use) for be-

spoke product(s)38 Model / Framework is adopted (in use) for

market-driven product(s)

18