estimating, planning and managing agile web …gibson/teaching/teaching...estimating, planning and...

Information and Software Technology 61 (2015) 124–144

Contents lists available at ScienceDirect

Information and Software Technology

journal homepage: www.elsevier .com/locate / infsof

Estimating, planning and managing Agile Web development projectsunder a value-based perspective q

http://dx.doi.org/10.1016/j.infsof.2015.01.0060950-5849/� 2015 Elsevier B.V. All rights reserved.

q The views presented on this paper are those of their authors, and do notnecessarily reflect those of their employers.⇑ Corresponding author.

E-mail addresses: [email protected] (C.J. Torrecilla-Salinas), [email protected] (J. Sedeño), [email protected] (M.J. Escalona), [email protected] (M. Mejías).

C.J. Torrecilla-Salinas a,⇑, J. Sedeño a,b, M.J. Escalona a, M. Mejías a

a Department of Computer Languages and Systems, University of Seville, Spainb Agencia Andaluza de Instituciones Culturales, Junta de Andalucía, Spain

a r t i c l e i n f o a b s t r a c t

Article history:Received 11 May 2014Received in revised form 10 January 2015Accepted 12 January 2015Available online 29 January 2015

Keywords:ManagementMethodologiesAgileScrumWeb Engineeringe-Government

Context: The processes of estimating, planning and managing are crucial for software development pro-jects, since the results must be related to several business strategies. The broad expansion of the Internetand the global and interconnected economy make Web development projects be often characterized byexpressions like delivering as soon as possible, reducing time to market and adapting to undefinedrequirements. In this kind of environment, traditional methodologies based on predictive techniquessometimes do not offer very satisfactory results. The rise of Agile methodologies and practices hasprovided some useful tools that, combined with Web Engineering techniques, can help to establish aframework to estimate, manage and plan Web development projects.Objective: This paper presents a proposal for estimating, planning and managing Web projects, bycombining some existing Agile techniques with Web Engineering principles, presenting them as anunified framework which uses the business value to guide the delivery of features.Method: The proposal is analyzed by means of a case study, including a real-life project, in order to obtainrelevant conclusions.Results: The results achieved after using the framework in a development project are presented, includ-ing interesting results on project planning and estimation, as well as on team productivity throughout theproject.Conclusion: It is concluded that the framework can be useful in order to better manage Web-basedprojects, through a continuous value-based estimation and management process.

� 2015 Elsevier B.V. All rights reserved.

1. Introduction

Starting a professional software development project soonraises some critical questions such as: How much will the projectcost? When will it finish? How much effort must be invested init? Will the investment be returned soon? What are the featuresour customers really need?

Being able to answer these questions and some others related tothem is crucial for designing business strategies (e.g. financial orcommercial, among others) from project results. The responses tothe aforementioned questions must condition all decisions likestarting one project or another, the type of product to developand the money that must be invested in it.

It is well known that estimating and planning a develop-ment project is a compulsory and complex process [7,18,45].To face this challenge, traditional estimation techniques focuson a predictive approach [1,12,13,37], which requires a stableand familiar environment. Essentially, these techniques beginwith a strong initial requirements gathering phase to freezeuser needs [53]. This approach makes these methods espe-cially sensitive to uncertainties and changes of customerneeds.

Nowadays, the rise of the Internet and the actual global andinterconnected economy has increased the needs for quicklyadaptation to changing customer needs. These events haveemerged in parallel with the acceptance of Web Engineeringas a discipline in Software Engineering [24]. Web Engineeringcan be defined as a set of methods, techniques and tools inSoftware Engineering that helps a development team build upsystems on the Web. There are several characteristics that dif-ferentiate Web projects from the rest of software developmentprojects [24,52]:

http://crossmark.crossref.org/dialog/?doi=10.1016/j.infsof.2015.01.006&domain=pdf

http://dx.doi.org/10.1016/j.infsof.2015.01.006

mailto:[email protected]





http://dx.doi.org/10.1016/j.infsof.2015.01.006

http://www.sciencedirect.com/science/journal/09505849

http://www.elsevier.com/locate/infsof

C.J. Torrecilla-Salinas et al. / Information and Software Technology 61 (2015) 124–144 125

� Complex navigational structure.� Critical interface requirements (such as unknown users or avail-

ability, among others).� Security aspects.� Increase on maintenance efficiency, avoiding downtimes.� Delivery as soon as possible.� Reduction of ‘‘time-to-market’’.� Adaptation to quick-changing requirements.

It is important to highlight that some of the aforesaid character-istics are not exclusive of Web development projects and can alsoappear in non-Web projects. Nevertheless, the concurrence of all ofthem together at the same time can be identified as a Web projectspecificity.

In such environments, Agile software development methodolo-gies, with constant monitoring and measurement, and frequentintervention mainly based on the use of empirical processes [61],are turning into a solid alternative for organizations developingsoftware to plan and estimate Web projects [4]. These methodolo-gies offer a suitable framework for the exposed Web developmentcharacteristics [55], like quick response to changes, adaptabilityand reduction of development time [31,50]. In addition, as it hasbeen mentioned, the classical approach regarding up-front require-ments gathering demands a stable environment, not being the caseof Web projects, where requirements change quicker. The incre-mental and iterative way of processing Agile methods require-ments [18,27] may better fit this particular case.

In contrast, the project management classical approach statesthat a project succeeds when it combines achieving the goalsestablished on variables such as cost, schedule and scope [53]. Fol-lowing these criteria, the Standish Group conducts the well-knownCHAOS surveys to test the projects success [72]. They define a suc-cessful project as the one that is carried out on time, on budget andincludes the originally specified features. Fig. 1 shows the projectssuccess level depending on the type of methodology used [73].

As it can be noticed, the percentage of successful projects usingAgile methods is significantly higher than that of projects usingtraditional approaches. These results can be associated with theimprovements that Agile techniques bring to project management,for example, ‘‘just-in-time’’ planning, iterative requirements gath-ering or frequent collaboration. However, it has to be added that, interms of the above definition concerning a project success, theclassical approach leaves behind crucial aspects such as qualityand delivered value to customers [33]. They are main issues toaddress on projects, since they are related to the functionalitydeveloped and the kind of process used. Thus, using techniquesthat allow us to better identify and measure the value delivered

Fig. 1. Results of projects dependi

to users will improve the results of our projects. For this purpose,we suggest some techniques to take into account these variables.

Lastly, it must be kept in mind that the number of unused func-tionality represents expended resources that rarely return to thedevelopment organization. A survey conducted by the StandishGroup [35], covering 2000 projects carried out by 1000 organiza-tions, showed that more than half of the functionality developedon a project is hardly ever or never used. Fig. 2 shows the resultsof that survey.

As before stated, the Agile iterative and incremental approachcan better fit the special needs of Web projects in order to partic-ularly identify what should be built and when it should be built.This approach will earlier identify these changes and will copewith them more properly to avoid designing unneeded features.It will allow a higher return on the projects investments. Basedon the foregoing, this work aims to cover the following objectives:

� Proposing a framework, based on existing Agile methods, toestimate, plan and manage Web projects that, guided by thebusiness value, will help to select what to build, estimating costand adapting plans to a changing environment.� Presenting the results obtained from a real experience dealing

with applying the proposed framework to a project developedfor a Spanish public administration office.� Taking out the main lessons learned after applying the proposed

framework, which will generalize successes and avoid failuresas well as will present future lines of work.

This paper is organized into the following sections. Followingthis introduction, Section 2 presents the research scope, includingalso the research questions and methodology. Section 3 presentsthe related work, describing different approaches to estimate, planand manage Web projects. Then, Section 4 provides an approach tothe suggested framework to estimate, plan and manage Web pro-jects based on Agile techniques, whereas Section 5 describes theexperience of applying the proposed framework to a real project.To conclude, Section 6 states the conclusions taken out, interpretsresults and consequences and advances possible future lines ofwork.

2. Research questions, scope and method

The main question we will try to answer in order to achieve theobjectives presented in the previous section is: ‘‘Is it possible todefine an Agile approach to estimate, plan and manage Web projectsguided by business value?’’ As it is very generic, we have tried todecompose it into the following research questions:

ng on the methodology used.

Fig. 2. Use of features and functions in a typical system.

Fig. 3. Research method.

Fig. 4. Case study definition.

126 C.J. Torrecilla-Salinas et al. / Information and Software Technology 61 (2015) 124–144

� RQ1: What are the suitable existing techniques for estimating,planning and managing Agile Web projects?� RQ2: Can business value help estimation, plan and management

of Agile Web projects?� RQ3: Can the identified techniques be integrated into an Agile

common framework, appropriate for Web projects needs?

The answer these questions will identify the research scope,that is, the definition of a framework suitable for:

� Agile Web development projects focused on estimation, plan-ning, and management activities.� Managing Web development projects guided by business value,

including continuous improvement throughout the project.

As described by Creswell [21], several factors must be takeninto account when selecting a research approach. Creswell alsostates that a qualitative approach might be appropriate when theresearch object still needs to be understood because of little exist-ing research. Case studies usually fall on the side of qualitativeresearch, in which an event or activity is presented in depth.Besides, Runeson and Höst [59] analyze the suitability of using casestudies in the field of Software Engineering. They put forward intheir work that case studies are suitable for exploratory research,in which new ideas to study are sought.

Based on this identified research scope, the used researchmethod includes the following steps:

� Step 1: Identify relevant related work in different contexts,focusing on the following research fields: estimation in Soft-ware Engineering, Web Engineering and Agile, Business ValueManagement in Agile projects and Earned Value Managementin Agile projects.� Step 2: Find out, from the identified related work, those existing

techniques that better suit estimating, planning and managingWeb projects and business value in an Agile way.� Step 3: Define a coherent Agile framework suitable for Web pro-

jects needs by means of the selected techniques.� Step 4: Validate the framework initially by means of a case

study.

Fig. 3 summarizes the research method.The case study will be defined and reported attending to Rune-

son and Höst’s proposal, including the following steps:

� Case study definition: The case study will consist in a project toassess the proposed theoretical framework and try to answerthe aforementioned research questions.

� Data collection: By means of project observations and projectmetrics retrieval to obtain meaningful data that will be furtheranalyzed.� Data analysis: It will include both quantitative and qualitative

analysis. The former is based on retrieved project metrics andthe latter focuses on project observations.� Reporting: All elements described will be included in the case

study report presented in Sections 5 and 6.

Fig. 4 presents the steps to design and report the case study.


3. Related work

As mentioned in the preceding sections, this work integratesseveral research areas that have to be analyzed in order to presenta right related work section. In fact, our approach is similar to otherapproaches regarding estimation models in Web Engineering par-adigm under an Agile perspective. As this general context is tooyoung, this section includes an overview of five contexts: estima-tion models in Software Engineering, solutions in Web Engineeringfor project estimations, solutions for project estimations in Agilecontexts, Business Value Management in Agile projects and EarnedValue Management in Agile. Besides, a subsection describing theScrum lifecycle has been included together with the related work,to better understand the terminology used along the paper.

3.1. Estimation models in Software Engineering

As previously introduced, estimating the effort and cost of adevelopment project is a complex and key process that should becarried out to assess whether a project is good value for money[7]. Many models have been proposed during the last 40 years toface this problem. They can be classified in two main categories:Non algorithm-based models and algorithm-based models. Thissection presents a high-level overview of some of the most popularmodels based on these approaches, without including details thatare out of the scope of this paper.

Algorithm-based models use mathematical approaches to cal-culate the project effort as a function of its major cost factors. Someof the most relevant algorithm-based models are listed anddescribed below:

� COCOMO (Constructive Cost Model): Proposed initially byBoehm [12] in 1981, its actual version, called COCOMO II [13],was published in 2000. This model uses a basic regression for-mula and its results are mainly based on code-size (given inthousand lines of code). A tool called Agile COCOMO II [3] hasbeen developed focused on this method, in order to allow thefine-tuning of the estimations according to the analogies amongprojects.� Function Points: This model, suggested by Albrecht and

Gaffney [1], provides a functionality-based measure of theprogram. Software Functional User Requirements are identi-fied and the total number of function points depending oneach one is categorized into one of these five types: outputs,inquiries, inputs, internal files and external interfaces. Oncethe function is identified and categorized into a type, it isthen assessed for complexity and assigned a function pointnumber. There are proposals associated with costs-modelfor Agile development projects combining story points withfunction points [68]. There are also case studies comparingfunction points estimation between Waterfall and Agile pro-jects [69].� Use Case Point technique: Proposed by Karner in 1993 [37],

this model is related to Use Case modeling. As Use Case offersthe functional scope of the application, the analysis can providevaluable insight on a development project size.

Below, some of the main non algorithm-based models are listedand described:

� Estimation by analogy [63]. This method consists in establish-ing analogies between the actual project and the costs of previ-ous and similar projects.� Delphi technique [42]. This method is centered on a panel of

experts view. Several rounds of consultations using question-naires are executed by providing participants with a consolidated

summary of the previous round results. Then, the experts reviewtheir opinion according to the previous results. There is a variantof this method, called Wideband Delphi [12], popularized byBoehm, which involves more participation on the expert’s part,as estimations are discussed together in a joint meeting.� Expert judgment method [36]. This method takes advantage of

a group of experts’ experience and understanding to figure outan estimated cost. This technique is used together with Delphitechnique in order to improve and systematize the consultedexperts opinion.

We have already stated that two of the main characteristics ofWeb projects are quick adaptation to changes and complex inter-faces. On the one hand, quick adaptation to changes implies severalmodifications on requirements. The project estimation of algo-rithm-based models focuses on the estimated absolute size (num-ber of code lines, functional requirements or Use Cases) of theproject, which is obtained at the beginning after a strong require-ments gathering phase. This could lead to the attempt of ‘‘freezing’’requirements at the end of this phase. This restriction may not per-fectly suit some Web projects, where, as previously introduced,requirements are not completely known at these early phasesand users needs could change quickly during the project. On theother hand, as having complex interfaces is another characteristicof Web systems, a collaborative and iterative process between userand development team guided by mockups, wireframes and pilotswill be necessary so as to design these interfaces. Due to this fact,the up-front estimation based on techniques as function points oruse case points can be difficult to use. Lastly non algorithm-basedmodels based on analogy, can better suit Web projects, since theydo not consider the absolute size of the project. However, this kindof estimation usually considers effort, and not the delivered value,which is crucial to select the features that should be built.

As it is known, Agile methodologies estimation models aremainly based on non-algorithm techniques, most of them focusedon Wideband Delphi methods (like the common Planning pokertechnique, which will be described later) [17]. We will also startour approach with this method, although it will be complementedwith the relevant techniques to ensure that the most value is deliv-ered early in the project.

To conclude, we can add that the special nature of Web devel-opment projects has not been regarded when facing the processof estimating effort and cost. On the contrary, they have beenaddressed as classical software development projects and theyhave been applied the same approaches. Nevertheless, asexplained, Web projects are different from classical developmentprojects [24,52] and have singular issues to face, both in technicalaspects (e.g. complex interfaces, navigation aspects or complexmaintenances, among others) and in managerial aspects (e.g. vola-tile requirements, short schedules, reduced time-to-market orshort feedback loops), which cannot be exclusively addressed bymeans of classical estimation approaches.

3.2. Project estimation in Web Engineering

Since 2002, Web Engineering becomes an accepted discipline inSoftware Engineering [24]. It can be defined as a set of methods,techniques and tools in Software Engineering that helps develop-ment teams build up systems on the Web.

In the last years, several methodologies in Web Engineeringarea have been proposed. Some like UWE (UML Web Engineering)[38], IFML (Interaction Flow Modeling Language) [47] or WebML(Web Modeling Languages) [14], among the newest, offer newsolutions and are widely accepted by the research community.UWE is a Model-Driven Engineering approach focusing on assistingthe Web engineer in the different phases of the development


lifecycle. In addition, we can find other proposals, such as the oneby Koch et al. [39], which presents a metric to measure the effortreduction resulting from applying this UWE approach. This metricis more centered on calculating the effort reduction than in fore-casting the needed effort for a particular project.

Despite the large number of existing approaches, they essen-tially concentrate on development phases and do not cover otherareas like project estimation, quality assurance or team manage-ment [28]. Nevertheless, HFPM (Hypermedia Flexible Process Mod-eling Strategy) [46] and NDT (Navigational DevelopmentTechniques) [29] represent two of the most relevant exceptionsthat can be found. Below, we will briefly describe HFPM andNDT, since defining all the existing models is out of the scope ofthis paper.

HFPM, proposed by Olsina in 1997, supports the project esti-mation phase as a task in the project lifecycle. However, it doesnot offer any specific technique or any technique adaptation tosupport this estimation, neither classic nor Agile. NDT is anotherexception. At the beginning, this approach did not support projectestimation as a task in its lifecycle. However, with the recent evo-lution of the approach, it adapts the Use Case point techniquesbased on the Model-Driven paradigm [7]. A new tool namedNDT-Counter was even included in NDT-Suite, the suite toolsfor NDT, to automate the application of this technique. This met-ric is based on an absolute magnitude (number of Use Cases),which will not perfectly cover continuous requirement changes.Therefore, this approach is hardly applicable in Agile contexts.Even though NDT supports Agile lifecycles in the developmentprocess, NDT-Suite is not appropriate for them, whereas NDT-Counter is completely designed for classical developmentprocesses.

Finally, it should be added that Mendes, in her work [45], hasanalyzed and compared several empirical cost estimation tech-niques proposed for Web development projects in the last years,most of them dealing with case studies analysis. In the paper byMendes, eleven Web cost estimation models are analyzed, mostof them, like these proposed by Mendes [44], Reifer [57] or Few-ster and Mendes [30], are based on estimating absolute magni-tudes (such as number of pages, number of links or COTS(Commercial Off-The-Shelf), for example), as they are derivationsof algorithm-based estimation models. Some of the problemsidentified in the study and stressed in the conclusions are thediversity of size measures proposed in the different works andthe absence of automated tools to simplify the data collectionprocess.

Mendes’s work points out the necessity and difficulty of defin-ing a Web estimation model, due to the special characteristics ofWeb projects (short schedules, fluidic scope, diversity of technolo-gies, short ‘‘time-to-market’’, diversity and knowledge of profilesinvolved in the development process and small teams).

As described before, Web development projects are character-ized, among other aspects, by the use of short feedback loops,which may cause frequent changes in requirements. Besides, thespecial characteristics of Web interfaces demand close collabora-tion and iterative design, which will cause difficulties, if the projectis estimated in terms of absolute magnitudes. In addition, it isworth mentioning that Mendes’s work detects the trend ofincreased usage of Agile methods in Web development projects[4]. These two facts enhance the benefits of defining an Agileframework to support the process of estimating effort in Webprojects.

Finally, it is important to highlight that none of these tech-niques focus on the delivery of business value, but only on a puremeasure of the projects’ size. This approach will not ensure thatthe most important features are delivered first, shortening ‘‘time-to-market’’, as Web development projects demand.

3.3. Agile estimation and plan

During the last years of the nineties, several methods and tech-niques based on an iterative and empirical approach emerged insoftware development projects [40]. The main goals of thesepractices were, on the one hand, to allow organizations to quicklyadapt to clients’ changing needs [50] and on the other hand, todeliver valuable results to customer as fast as possible. Some ofthese techniques, methods and methodologies are Scrum [70],eXtreme Programming (XP) [10], Crystal [15], Lean Software Devel-opment [51] or Feature Driven Development (FDD) [49].

The aforementioned methodologies propose a wide-ranging setof techniques to estimate and plan projects, mainly in terms ofnon-algorithm-based models [18,33]. Most of the Agile estimationtechniques concentrate on the use of a popular one called ‘‘userstories’’. It was firstly introduced by eXtreme Programming [10]and then popularized by Mike Cohn [17].

A user story can be defined as a short piece of functionality thatprovides a customer or a user of a system with a value. A user storyrepresents certain user needs, but not an exhaustive documenta-tion of them. It acts as a reminder and its details are discoveredduring the collaboration process that runs to develop it at a certainSprint. The details of the user story are usually recorded in a set oftests that is used to check that the story is finished. A good userstory is characterized by a set of attributes, known as the acronymINVEST [10]: Independent, Negotiable, Valuable, Estimable, Smalland Testable. Attending to user stories, the most relevant Agileestimation techniques are summarized below:

� The planning game: [64] This technique, proposed by eXtremeProgramming, assumes that customers have most of the infor-mation about what has to be developed and developers havemost of the information about how to implement those fea-tures. Developers estimate the cost of each feature and custom-ers prioritize the relative importance of each feature among therest of features. These two steps are repeated until all of the fea-tures are estimated and organized. During the process, develop-ers and customers interact by solving doubts about prioritiesand estimations. Initially, this technique was used to evaluatethe work in an iteration, but it can also be used to perform acomplete release planning [43].� Planning poker: [18,66] It was also initially used to plan the

work in an iteration. Cohn [18,19] and Highsmith [33] proposethis technique to perform estimation at the project level, com-paring user stories instead of technical tasks. The entire devel-opment team estimates a set of features in this technique.Each member of the team has a deck of cards with a discretesubset of values (for instance 1, 2, 3, 5, and 8), representingthe points to be assigned to each feature. One by one, the cus-tomer representative explains the features. Once a feature isdescribed, the team members can ask some questions to clarifyits scope. After that, each member, at the same time, shows acard from his/her deck with the estimation. If they are all thesame, the feature receives the estimation; if not, members pro-posing the highest and the lowest estimation explain theirpoints of view and new rounds are played until reaching a con-sensus. Then, the team forecasts velocity (meaning the numberof points that they can deliver in an iteration, by means of his-torical data, a ‘‘test’’ iteration or an educated guess) and estab-lishes the length of the iteration. The number of iterations isobtained by dividing the total number of points by velocity. Inthe same way, the duration of the project is calculated by mul-tiplying the number of iterations by length.� Blitz planning: [15] It is a variation of the Planning Game. In

this technique, representatives of all stakeholders meet in thesame room and brainstorm the relevant tasks to design the


desired system, by writing them on index cards. Cards arespread on a table and ordered in terms of their priorities andinterdependencies. Developers estimate the effort of each task,including any external dependency on them. Then, the teamidentifies an implementation of the system that performs asmall end-to-end feature (called Walking Skeleton), the earliestusable release (called First Delivery) and the first delivery thatproduces a revenue stream (called Point of Earliest Revenue).Finally, the team looks for excessive workload or a task thatcan either block the work or entail a high risk.

Although these techniques were initially proposed for generaldevelopment projects without taking into account the specialneeds of Web development project, as it has been stated, the char-acteristics of Agile methods fit some of the Web projects require-ments, such as short feedback loops, reduced time-to-market andquick changing users needs. To conclude, it has to be remarked thatall of these techniques only face up effort estimation without sys-tematically considering the business value of the delivered fea-tures. Both cases, Planning Game and Blitz planning, mentionthat customers will establish ‘‘priorities’’, without proposing a con-crete technique to do so. Particularly, Planning poker only consid-ers effort and not business value.

3.4. Business Value Management in Agile projects

As mentioned, a critical element on Web projects, in whichtime-to-market is a key success factor, is building the right featuresin the right moment. Business value and management in Agile pro-jects might become useful tools to better identify these features.We can find some interesting works studying the relation betweenAgile and Business Value Management, although as this is a rela-tively new field, and not many papers have been published on thistopic.

Concerning delivery of value, Highsmith [33] proposes using theAgile triangle. This triangle changes the vision of the main con-straints of a project: Value, Quality and Constraints (that in turnincludes cost, schedule and scope). He suggests the name of ‘‘Agiletriangle’’ to identify this set of variables. Fig. 5 shows the Agile pro-ject management triangle.

This author recommends estimating and tracking the valuedelivered by the project by means of the ‘‘value points technique’’.We will later describe that our framework recommends combiningboth size and value estimation techniques to better guide theselection of features, which is a crucial element in Web projectssuccess.

Yap’s work [78] presents an experience report describing how acompany tried to find a value-based feedback mechanism involv-ing shared responsibility between customer and team. Thecompany applied eXtreme Programming with teams from 6 to 10

Fig. 5. Agile triangle.

developers. The report explains how the company introduced the‘‘Value Based Investment Decisions’’ and ‘‘High confidence storiesfirst’’ techniques, together with other practices. The former wasbased on retrospectives to provide feedback on what projectdeserved investment and the latter suggested that the stories theteam thought they could finish should be first delivered in orderto maximize value. The work does not propose a simple quantita-tive technique that should take into account both customer anddevelopers’ points of view in order to prioritize work, althoughthe recommended techniques can improve the business valuedelivered both at portfolio and project level.

Register and Golding [58] suggest using Agile techniques,mainly Agile estimating and planning proposed by Cohn [18], inorder to maximize the delivered business value in build vs buydecisions. Again, this work specifically focuses on portfolio level,without proposing techniques to decide what work should be pri-oritized at project level.

Racheva et al. [54] present in their work the results of anexploratory cross-case study on Agile prioritization and businessvalue delivery processes in eight software organizations. One ofthe research questions that this paper tries to answer is whethercompanies are using value-based criteria to perform value-drivendecisions during Agile prioritization. One interesting conclusionobtained is that some of the companies are not only centered onthe value delivered by the feature, but also on the value detractedwhen the feature is not delivered. In addition, it analyzes the inter-relation between customer and developers during the value prior-itization. Once again, this work does not suggest any quantitativetechnique to perform prioritization taking into account both views.

Finally, Logue and McDaid [43] study the Agile release planningprocess, not only to embrace the viewpoint of developers, but alsothe business value proposed by customers. Their work, based onthe Planning Game of eXtreme Programming, puts forward theuse of a business value attribute for each of the stories. In this case,customers suggest three values (optimistic, pessimistic and mostlikely) for the business value. The development team does thesame with the story size. Once the distributions are established,a Monte Carlo simulation is carried out to distinguish how thecombined size and business value of the stories are distributed.This way all participants know the likelihood of each potentialrelease plan. This proposal, which mixes the different points ofview to define a good release plan, has a drawback; it increasesthe overhead of the planning, and that could discourage revisitingthe plan once the project starts.

To conclude, our framework offers a way of introducing busi-ness value in the process of elaborating and managing the releaseplan on an Agile way, without increasing significantly the overheadof the process to keep it as Agile as possible.

3.5. Earned Value Management in Agile projects

Earned Value Management [53] is a well-known tool that hasproved to be useful to manage classical projects as well as to directprojects on regulated environments. However, there is not muchliterature linking these techniques to Agile. This section presentsthe related work regarding the usage of EVM techniques in Agileprojects.

Alleman et al. [2] present a case study, in which they use theEVM techniques according to their client needs and combinethem with some Agile practices coming from eXtremeProgramming.

Cockburn, in his book presenting the Crystal methodology [15],includes some comparison between the EVM techniques and someAgile tools, like the burndown chart, but he proposes no modifica-tions in EVM techniques to be used in Agile projects.

Fig. 6. Elements conforming of the proposed framework.


Solomon and Young [65] recommend a variation of EVM, namedPerformance Based Earned Value (PBEV) that brings to EVM impor-tant elements, such as risk management or quality aspects, com-patible with Agile projects. This approach, being compatible withAgile methods, may require some extra overhead on the top ofScrum projects.

In the work by Sulaiman et al. [67] we find a very complete anddetailed proposal related to the usage of EVM techniques in Scrumprojects. They analyze EVM magnitudes and advise a way of calcu-lating EVM values at release level. Our proposal is based on theirs,with some variations: first, EVM values are obtained at Sprint levelto provide valuable management information more frequently inorder to best suit the quick adaptation required in Web develop-ment projects; second, we do not use the Agile EVM approach tocalculate the release date, but purely to monitor and control theproject to keep it under its constraints; and lastly, we maintainthe standard EVM terminology. Our variations to this proposalare described in subsequent sections.

3.6. Scrum

In 2001, after the appearance of several empirical softwaremethodologies, some of the most recognized practitioners, suchas Kent Beck, Alistair Cockburn, Martin Fowler, Ron Jeffries, RobertC. Martin, Ken Schwaber and Jeff Sutherland, promoted what wasknown as the ‘‘Agile manifesto’’ [8]. This manifesto included thevalues and principles supporting the Agile philosophy, which wereshared among all methodologies that were called Agile. Some ofthe basic principles summarized in this manifesto were: the con-tinuous delivery of valuable and potentially shippable software;the existence of small, self-organized teams; the use of short iter-ations; the inspection and adaptation of the development pro-cesses; and the ability to quickly respond to changes.

As it is known, in the last years, Agile approaches based on the‘‘Agile manifesto’’ values and principles are becoming more popu-lar in software development [11]. As Scrum is one of the most usedmethods [50] among them, this is the reason why we include it as abase of our proposal.

Jeff Sutherland and Ken Schwaber proposed Scrum [60], beinginfluenced by Takeuchi and Nonaka’s work [71]. It can be definedas a framework for product development [70] and it suggests aniterative and incremental approach for project management. Thedevelopment process in Scrum is divided into working cyclesnamed Sprints, with a constant size that can differ from 2 to4 weeks, which is repeated along the project. The main character-istic of these cycles is that they are time-boxed (they start andend with a fixed and expected date, even in the case that theplanned work may not be completely finished), and they areruled one after another, without interruptions until the end ofthe project.

Scrum projects start with the creation of a Product Backlog. Thisartifact consists of a list of all the features that can be developedduring the project and it is characterized by being ordered and pri-oritized. The Product Backlog is a living document, which canchange during the project. These features can be added, deleted,modified, re-prioritized and so on, in terms of any changes on usersneed.

As Scrum model states, at the beginning of each iteration, thedevelopment team runs a meeting called ‘‘Sprint Planning meeting’’,where all team members, with the help of users and customersrepresentatives, select the amount of work to perform during theSprint and they commit to carry it out. They use an artifact calledSprint Backlog to facilitate this process. It has to be pointed outthat the team commits to work and, at the same time, it is incharge as a group of fully developing the job by the end of the iter-ation, succeeding or failing as a whole. Everyday, the team checks

the progress during a short time-boxed meeting of fifteen minutesmaximum, called ‘‘Daily Scrum’’, to ensure the work is progressingsatisfactorily. All impediments and blocks are collected at thismeeting with the aim of addressing them.

Once the iteration ends, and in order to guarantee that theresultant product and the development process are supervisedand adapted, two meetings are arranged. The first one is called‘‘Sprint Review meeting’’, a time-boxed meeting where the teampresents the results of the iteration to the relevant stakeholders.During this meeting, several new features can be discovered andsome of the previous can be modified or discarded. All of theseaspects must be reflected in the Product Backlog. The second meet-ing is called ‘‘Sprint Retrospective meeting’’ and deals with review-ing, adjusting and improving the development process the teamhas executed.

4. An approach to Web Engineering estimation, plan andmanagement with Agile methods

As put forward in the previous section, there are severalapproaches to software development projects estimation andplan, coming from different fields (classical project management,Agile context or algorithm-based), but none of them individuallyconsiders all the special needs Web development projects entailtogether with a value-based perspective. In the introduction, weadvanced that there is a set of characteristics that differentiateWeb projects from classical software development projects. Someof these characteristics are: need of reducing the ‘‘time-to-mar-ket’’; value delivery as soon as possible; need of adaptation tocontinuously changing requirements; improvement of interfacerequirements criticality; and availability, maintenance and secu-rity. We have analyzed that Agile planning methods are appropri-ate to reduce ‘‘time-to-market’’ and adapt to changingrequirements as well as Web Engineering techniques address,among other issues, complex navigation, security and user-inter-face criticality, among but none of them by themselves can coverall Web projects specificities.

During the preceding sections, some valuable elements havebeen identified in order to configure a proposal to plan, manageand estimate Agile projects guided by value. Fig. 6 offers an over-view of these elements that will compose the proposed framework.They will be described in the following sections.

The framework to estimate, plan and manage Web develop-ment projects based on an Agile approach comprises the followingcharacteristics:


� The proposed framework is based on Scrum lifecycle, includingad-hoc modifications to fit the special needs of Web projects.� The proposed framework is based on a non-algorithm estima-

tion model, using Wideband Delphi techniques to initially esti-mate the project, as proposed by Agile estimation and planmethods. As stated before, these estimation techniques bettermeet the special characteristics of Web projects.� The proposed framework is based on an Agile approach, making

estimation and plan an iterative and incremental process duringthe whole project, not only a phase performed at the beginning.This approach matches well the short feedback loops that char-acterize Web development projects.� The proposed framework not only considers the classical pro-

ject management topics of scope, cost and schedule, but alsoelements like quality and business value, as proposed by theAgile triangle. As previously mentioned, this enables betteridentification of what should be built, a crucial element inWeb development projects.� The proposed framework uses a variant of the Earned Value

Management (EVM) technique to help team members managethe project during the lifecycle and monitor constraints.� The proposed framework also measures the team productivity

with the aim of improving it along the project.

As a first conclusion, it should be highlighted that the proposedframework can be presented as a continuous Plan and Estimate–Manage–Measure–Adapt cycle. Fig. 7 outlines this cycle.

Based on the cited elements, Fig. 8 summarizes the proposedframework to plan, estimate and manage Web development pro-jects with Agile techniques.

As mentioned before, none of the different identified techniquescan, on their own, provide an acceptable framework to estimate,plan and manage Web projects in an Agile way, guided by businessvalue. Thus, it can be pointed out that the main contribution of theframework is the combined use of all these techniques, which con-sequently will be able to cover all the special needs that Web pro-jects demand.

This section describes the framework presented in Fig. 8, ana-lyzing its different phases (planning, estimating, managing andmeasuring the productivity of the project) in different sections.

4.1. Project lifecycle

The project lifecycle will be divided in two phases, as shown inFig. 8:

� Project launching.� Project development.

The Project launching phase will cover the initial estimationand planning effort, described in Section 4.2. The Project develop-ment phase will be based on the standard Scrum lifecycle, as it isthe most popular Agile approach [50], including the techniquesdescribed in Sections 4.3 and 4.4.

Fig. 7. The proposed cycle.

Taking as a reference the standard Scrum process, our frame-work will involve the following elements:

� Definition of Product and Sprint Backlogs.� Sprint planning meeting.� Daily Scrum.� Sprint review meeting.� Sprint retrospective.

As an initial element, the process will include the so-called‘‘Sprint 0’’ [74] in order to establish the rules that the team willapply throughout the project. This element will be described lateron.

On the top of the standard Scrum practices, and in order to keepthe Product Backlog updated, 10% of the work-time will be spent inreviewing content [19]. This activity is usually named ‘‘ProductBacklog grooming’’. Sprint retrospectives will be organized follow-ing the principles of Agile retrospectives [25,32] including activi-ties and innovation games, such as satisfaction histogram, radar,color dot voting, Ishikawa diagrams or 5 whys, among others.Fig. 9 shows the proposed elements of Scrum to be used.

Together with these practices, it is recommended to establishteams composed of ‘‘generalizing-specialists’’ [5,6], that means, peo-ple that can efficiently develop different kind of tasks.

4.2. Planning and estimating Web projects

The proposed framework is particularly based on Scrum frame-work, which has been previously described, including some othertechniques to cover estimation, plan and management processescarried out in Web development projects. Fig. 10 presents the rec-ommended process to estimate and plan Web projects within theframework.

The steps below must be followed to estimate and plan the pro-ject, both at Project launching and Project development phases, asan iterative process:

� Populating Product Backlog.� Estimating Product Backlog size and value.� Calculating ROI and organizing Product Backlog.� Setting the iteration length and estimating team velocity.� Developing the initial Project Plan.

Scrum does not establish what the nature of pieces of func-tionality, which are part of the Product Backlog, must be. There-fore, in this proposed framework these features are ‘‘userstories’’, whose main characteristics have been described before.Users stories satisfactorily fit the characteristics of Web projectsand they encourage collaboration between users and the devel-opment team in processes such as interface and scopedefinition.

A set of workshops is conducted between users and develop-ment teams at the beginning of the project to populate the ProductBacklog with user stories. The initial Product Backlog is created atthese workshops, including all the identified user stories. Each ofthem should have the following attributes:

� Theme: It represents a category that links a relevant number ofuser stories.� Story ID: It offers a unique number to represent the story and

helps to find and referencing it in an easy way.� Description: It describes the functionality and/or value pro-

vided by the story. It should include a reference to the profilefor which the story provides a value.� Business value: It stores the business value the story offers.

Later in this section, we will describe how this value is assigned.

Fig. 8. The proposed framework.

Fig. 9. Used elements in Scrum.


� Size: It stores the relative size of the story, in comparison withthe rest of stories of the Product Backlog.� Return of investment (ROI): It stores the relation between cost

and value provided by each story.

Fig. 10. Agile estimating and plannin

� Proposed by: It identifies who proposes the user story.� Date: It specifies the date when the user story is included in the

Product Backlog.� Comments: It records any additional comment that can clarify

the scope of the user story such as restrictions, dependencies,limitations and special cases to take into account or examples,among others. This is a living attribute that has to be updatedduring the project as a result of the collaborative relationamong users, customers and development teams.� How to test it: It registers a description of any test that helps to

assert that the story is really executed. It is used as a basis toautomate tests, if appropriate.

The most popular Agile project management systems includemost of these properties. For instance, VersionOne [76] offersTheme, Story ID, Description, Size, Proposed by, Date andComments. Agile plugin for JIRA [34] recommends Theme, StoryID, Description, Business value, Size, Proposed by, Date andComments. Finally, Redmine [56] gives the possibility of definingcustom fields to apply to user stories. It has to be mentioned thatnone of the systems offers ROI property.

g process within the framework.


During the creation of the Product Backlog, an initial estimationof the amount of needed work and its value is developed. For thispurpose, every user story includes an attribute called ‘‘Size’’ repre-senting the estimated amount of work required to finish the storyin comparison with the rest of the stories included in the ProductBacklog answering the question: ‘‘How difficult is to build a certainpiece of functionality?’’

Classical estimation approaches focus on calculating the totalamount of effort needed to finish some task whereas Agileapproach proposes evaluating the relative size of a feature in rela-tion to other features by means of an ideal measure unit called‘‘story point’’ [18]. To compare them, one of the user stories ofthe Product Backlog is chosen to act as a basis for comparisonand the rest of the stories are estimated taking it as reference.The most common approach [18] consists in selecting a small fea-ture to act as a basis for comparison.

It is recommended to pick a discrete universe of measurements[18] to make the estimation process easier. In this particular case,the use of a variant of Fibonacci’s sequence is proposed (0, 1, 2, 3, 5,8, 13 and 20). This framework advises to use Planning poker as anestimation technique to perform all the estimation process, keep-ing in mind that only members of the development team can esti-mate the size of each story. Users or customers work in liaison withthe development team by answering questions and clarifying anydoubt regarding the scope of the stories.

In addition to the size of each story, the attribute ‘‘Value’’ is alsoestimated in order to obtain the Product Backlog size. This attri-bute intends to answer the question: ‘‘How much value does thisfeature bring to the organization?’’ Using this attribute is veryimportant, as it integrates users opinion, which is crucial in Webdevelopment projects. The ‘‘Value point analysis’’ technique [33]is the key to address this process. It deals with assigning a differentvalue to each story from a discrete set of values, by means of differ-ent scale (the following values are proposed: 500: 1000; 2000;5000; 10,000 and 20,000). Only customers, users or their represen-tatives, depending on how the workshops are organized, can assignthis value. The total amount of value points that users can give tothe Product Backlog is limited, aiming at visualizing the existenceof limited resources in each project and ensuring the real prioriti-zation of stories.

At the end of the workshops, each of the elements of the Prod-uct Backlog has two values; size (in story points) and value (invalue points). The former represents how much the organizationhas to pay for building the feature (either by investing money orresources) and the latter represents how much value this featureimplies. Using both, a relation between cost and benefit of everyuser story can be calculated, and that value is assigned to the‘‘Return Of Investment’’ (ROI) attribute of the story [20]. The ROIwill present a measure of the cost-benefit ratio, seemed as an easyway to calculate how much the company will pay to obtain the dif-ferent features of the Product Backlog.

Using this value in the project has two main goals:

� Allow ordering the Product Backlog taking into account bothbusiness and development points of view� Avoid introducing a significant amount of overhead in the esti-

mation process that could impede frequent re-planning cycles.

The equation to calculate this value is the following:

ROI ¼ Value ðin value pointsÞ=Size ðin story pointsÞ

As previously introduced, one of the characteristics of the Prod-uct Backlog is that it is an ordered document. In this line, ROI isproposed with the aim of organizing Product Backlog items toaddress, mainly, those stories that have the highest ratio betweenthe benefits provided and the effort needed. This ordering will

provide an initial proposal for both business and technical teamsthat might be adjusted in accordance with the needs and con-straints (for instance, a feature either represents a competitiveadvantage or it is a technical constraint). The ordered ProductBacklog can be readjusted after the agreement reached by allstakeholders during the release planning discussions.

The team has an ordered list of the initial scope of the project,once this process is finished. Consequently, they are ready todevelop an initial project plan. However, it is essential to estimatetwo important magnitudes to carry it out: the team velocity andthe duration of the iteration. Velocity can be defined as a measureof a team rate of progress [18] and represents the amount of storypoints finished during the iteration. The value can be obtained bymeans of different methods, like executing test iteration, using his-torical data from previous projects or forecasting the value. Thelength of the iterations must be the same during the project toguarantee velocity value consistency. This length is establishedaccording to several factors [18] like users need of feedback orchanges, or uncertainty of functionality or technology, amongothers.

Velocity represents the number of story points that can finish inone iteration, therefore, if the total amount of story points of theProduct Backlog is divided by velocity, the result will be the initialnumber of iterations needed to finish the project:

Number of expected iterations

¼ Total number of story points=Velocity

Once the duration of the iteration has been calculated, the ini-tial estimation for the project length results from multiplying thisvalue by the number of expected iterations:

Initial duration of project ¼ Number of expected iterations

� Length of iteration

Lastly, and by means of the organized Product Backlog, the esti-mated team velocity and the established duration of the iterations,an initial release plan can be developed by allocating each story inan iteration in terms of the ‘‘Return of Investment’’ attribute valueand size [49].

To conclude, it should be reminded that estimating and plan-ning in Agile is not a phase, but a process, thus this initial planneeds to be reviewed and updated in each iteration. Everybodyknows that the continuous control and modification of the devel-opment process is the key for success in empirical processes [51].To guarantee this, the Product Backlog needs to be updated (some-times deleting, adding or modifying stories or some others reas-sessing their value or size, for example), the real velocity of theteam needs to be tracked, and depending on the changes, the pro-ject plan also needs to be updated. Next section further explainsthe proposed techniques to achieve this goal.

4.3. Managing Web projects

This sub-section presents our advised management process forWeb development projects, describing at the beginning the man-agement workflow, how the Agile triangle helps to guide the pro-ject to succeed, how to use Agile EVM techniques and how to useand present the quantitative data to better manage the project.Our management process is a value-guided quantitative process,which proposes tools to quantitatively identify the most relevantfeatures of the customer so as to deliver them firstly. Ours is aScrum-based iterative approach, consequently, it will meet thespecific Web development requirements in three ways.


� Increasing feedback loops: The iterative approach will provide‘‘check-points’’ at every Sprint. This will allow adjusting theplanning, based on new priorities coming from any of the stake-holders, by means of the value or the size of each story.� Improving adaptation to changes: The proposed framework will

allow quick adaptation to changes, providing the tools to reas-sess and re-plan at Sprint level.� Reducing ‘‘time-to-market’’: As a consequence of the foregoing,

the right features will be built in the right moment, with shortdevelopment cycles.

4.3.1. Management workflowThe management process starts when the initial planning fin-

ishes and lasts until the end of the project. Fig. 11 illustrates themanagement workflow.

As Fig. 10 shows, after developing the initial project estimationand plan, the senior management team of the organization sup-porting the project must approve it. The general approach for thistask deals with making a project charter. It is an executive docu-ment, as defined by the Project Management Institute [53], wherethe main elements of the project are presented, including the costthat carrying out the project implies as well as the main risks.

Developing a project charter is considered a valuable tool in theAgile world, even though its development entails considering the‘‘Barely sufficient’’ approach proposed by Alistair Cockburn [16].This project charter may include at least the initial project plan, itsmain risks and costs as well as the main involved stakeholders. Thisapproach meets properly the characteristics of Web projects, as theyare normally short projects with short ‘‘time-to-market’’ require-ments, not having time to be spent in long documentation phases.

Once approved, the project starts running iteration-by-itera-tion, trying to keep a sustainable and continuous developmentrhythm. However, first of all, a special ‘‘Sprint 0’’ is executed [74]in order to establish, among others, the rules and criteria for qual-ity management. This ‘‘Sprint 0’’ is a time-boxed iteration, lastingthe same period of time as the rest of Sprints, where the teamestablishes the general basis and rules for the project: qualityassurance, project data management, ‘‘definition of done’’ and pro-ject measurement and analysis. When it concludes, the normaliteration-guided lifecycle of an Agile project begins with consider-ing that all quality and management data established during‘‘Sprint 0’’ is gathered during ordinary Scrum meetings.

4.3.2. Usage of Agile triangleAs before stated, the traditional approach to project manage-

ment usually takes into account cost, schedule and scope

Fig. 11. Management workflow.

restrictions by means of a tool known as ‘‘Iron triangle’’. Neverthe-less, achieving the goals over those variables successfully does notguarantee that the most valuable features are delivered to custom-ers and final users. This fact is even more important in Web devel-opment projects, as they are normally short projects carried out bysmall teams, where developing or not the right feature implies thesuccess or failure of the project.

We previously referred that the Agile triangle recommends theusage of Value, Quality and Constraints as guides to asses the suc-cess of a project. Our framework proposes a set of techniques thathelps to track the goals achieved during the project to successfullymanage these variables. The necessary data is gathered during thepreviously cited meetings (Sprint Planning Meeting, Daily Scrum,Sprint Review and Sprint Retrospective) in order to reduce theoverhead as much as possible. They are estimated at iteration levelto allow the team to modify the process with the aim of improvingresults.

The first of the proposed restrictions is ‘‘Value’’. Each user storyis evaluated during the initial planning and assigned a value givenin value points by means of ‘‘Value Points analysis’’. Then, a‘‘Return of Investment’’ of each story is calculated through thisvalue as well as the size of each story. The present framework pro-poses both to structure the Product Backlog in a descending orderto that of the ‘‘Return of Investment’’ attribute and develop thosestories that have the closest relation between value given and costneeded. This approach provides teams with a quantitative measureof the value that each feature assumes for users, not only at thestart of the project, but also through all Sprints. In consequence,the team can focus on the right feature in the right moment, whatrepresents one of the key success factors in Web developmentprojects.

Tracking the percentage of achieved business value at the end ofeach Sprint is suggested as an additional tool. This figure is calcu-lated by means of the following formula:

% Delivered Value

¼X

Value of finished stories=XValue of all stories on Product Backlog

A chart representing its evolution trough iterations is used tooffer this value. Fig. 12 shows an example of this chart.

This tool tends to help the team deliver most of the projectvalue during the initial iterations of the project. The case study pre-sented afterwards will confirm that the calculations of these met-rics do not imply significant overhead to the project. On thecontrary, they constitute a useful tool to guide the development.

The second variable proposed by the Agile triangle is ‘‘Quality’’.It has to be mentioned that within Agile philosophy ‘‘Quality has tobe built-in along the project’’ [51]. When talking about quality, itstwo faces must be reminded: product quality and process quality,

Fig. 12. Evolution of the value delivered through iterations.


being necessary to measure and manage both during theproject. Agile proposes several techniques, most of them basedon ‘‘Test-first’’ philosophy, such as Test-Driven-Development[9,10], to guarantee product quality. However, the detaileddescription of these techniques is out of the scope of this paper.With regard to quality, it has to be added that only the story pointsof stories that pass the defined test and customer acceptance counton team velocity.

Retrospective meetings are the main tools to direct the processquality. During these meetings, the established development pro-cess is checked through a systematic analysis by means of innova-tion games and other techniques, with the aim of tracking the realroot causes of the identified problems and impediments. Somemetrics, dealing with discussions on these meetings, are intro-duced in the following section. Our framework recommends col-laborative tools, like Wiki pages, in order to collect theconclusions of each retrospective meeting and track the solutionalong the following iteration of the project.

The last magnitude included in the Agile triangle is ‘‘Con-straints’’. This magnitude involves all classical project constraints,like scope, cost and schedule. Our framework suggests a variationon the Agile-based approach to EVM techniques proposed bySulaiman et al. [67] to follow the evolution of this variable. Asit is known, these techniques allow measuring the relationshipamong cost, scope and schedule in the course of a project [53].EVM proposes two main indexes to track the relations amongconstraints. These indexes are Cost Performance Index (CPI) andSchedule Performance Index (SPI). CPI appraises the relationbetween estimated cost and real cost and SPI determines the rela-tion between estimated schedule and real schedule. If the valuesof these indexes are close to 1, it means the project is evolving asit was conceived.

Our framework advises measuring both indexes at the end ofeach Sprint and analyzing them in the retrospective meeting. Thatwould help to identify problems or obstacles early during the pro-ject execution and correcting them in the forthcoming iterations.

4.3.3. Agile Earned Value ManagementThe following data is calculated at the beginning of the projects,

with the aim of obtaining Agile EVM values:

� Estimated average cost of the team per hour: It represents theaverage labor cost of team per hour.� Estimated number of hours per iteration: It is obtained when

planning the first iteration, and it varies depending on the num-ber of team members and the hours they work in the project.� Estimated average cost per iteration: It is the result of multi-

plying the estimated average cost per hour by the estimatednumber of hours per iteration.

Estimated average cost per iteration

¼ Estimated average cost per hour of team

� Estimated number of hours per iteration

� Budget At Completion (BAC): It refers to the total estimatedcost of the project. It is the result of multiplying the numberof expected iterations by the estimated average cost periteration.

BAC ¼ Estimated average cost per iteration

�Number of expected iterations

The following parameters are calculated at the end of the itera-tion, based on previous values:

� Expected Percent Completed (EPC): It is obtained by dividingthe planned completed story points at the end of the iterationby the total amount of story points of the Product Backlog.

EPC ¼X

Planned completed story pointsat iteration i=XTotal stories on Product Backlog

� Actual Percent Completed (APC): It is obtained by dividing thereal completed story points at the end of certain iteration by thetotal amount of story points of the Product Backlog.

APC ¼X

Real completed story points at iteration i=XTotal stories on Product Backlog

� Planned Value (PV): It is obtained by multiplying EPC at certainiteration by the BAC of the project.

PV ¼ EPC at iteration i � BAC

� Earned Value (EV): It is obtained by multiplying APC at certainiteration by BAC of the project.

EV ¼ APC at iteration i � BAC

� Actual Cost (AC): It is obtained by multiplying the Estimatedaverage cost of the team per hour by the real number of work-ing hours at the end of certain iteration.

AC ¼ Estimated average cost of the team per hour

� Real number of working hours at iteration i

� Cost Performance Index (CPI): It is obtained by dividing EV byAC at the end of certain iteration.

CPI ¼ EV=AC

� Schedule Performance Index (SPI): It is obtained by dividingEV by PV at the end of certain iteration.

SPI ¼ EV=PV

At the end of each Sprint, the number of finished story pointsand working hours dedicated to the project is tracked and the val-ues of the exposed magnitude are calculated. Once analyzed, cor-rective actions are performed in order to adapt the project to thenew situation.

4.3.4. Management reportsOne of the most relevant things to handle during a project con-

cerns managing expectations. This framework recommends toolslike burn-down charts, burn-up charts, velocity charts or changereports, among others [18, 62], to address the challenge and informthe relevant stakeholders on the project’s progress during theiteration.

A special report is generated to inform the relevant stakehold-ers at the end of the Sprint. This report might include the followinginformation:

� Change report: It refers to stories that have changed during theiteration (e.g. finished, non-finished, discarded, re-estimated orincluded in the Product Backlog).� Iteration result: It deals with story points and value points fin-

ished, percentage of value delivery or real number of workinghours dedicated to the project.� Agile EVM calculations: It is used to show the status of the pro-

ject in relation to the designed plans.� Iteration burn-down chart: It shows how the work progresses

during the iteration.� Project burn-down chart: It represents the evolution of the

real completed work of the project regarding the expected com-pleted work.


� Project burn-up chart: It offers the evolution of the real com-pleted work in the project in relation to the remaining workin the Product Backlog.� Evolution of delivered value through iterations: It represents

how much value is being delivered to customers and users.

This report can be generated as a printable document or can besimply posted in a collaborative tool (like VersionOne [76]) or inthe team room for consultation. It plays the role of ‘‘informationradiator’’ [15].

Most of the reports are available in tools like VersionOne, withthe exception of those regarding Agile EVM calculations and theevolution of delivered value through iterations. These can be easilygenerated with Excel spreadsheets.

4.4. Measuring productivity in Web projects

This framework proposes tracking some productivity metrics tomotivate teams to improve their performance during the projectdevelopment. As Web development teams are normally smallgroups of people, it is important to take most of them to delivervalue to customers.

One of the goals of Agile projects is to maintain a lightweightprocess, which enables adapting to changes that provide customersand users with competitive advantages. These metrics, suggestedby Downey and Sutherland [26], can be obtained without increas-ing overhead expenses of the process during the meetings, sincethe finished story points and dedicated working hours are onlyneeded to calculate them. Metrics are evaluated per Sprint andanalyzed during the Sprint retrospective. Our framework includesthe following metrics from Downey and Sutherland’s set ofmetrics:

� Team velocity (in hours): It is defined as the sum of the fin-ished story points in certain iteration multiplied by the averagenumber of hours per story point. It represents the averageamount of functionality delivered by the team per iteration.

Team velocityðhoursÞ¼XðFinished story points in iteration

�Average number of hours per story point in projectÞ

� Work capacity: It is defined as the sum of working hours spentduring certain Sprint, whether the user story in which develop-ment the hours were spent in was finished or not. It representsthe average cost spent by the team per iteration.

Work capacity ¼X

Working hours in iteration

� Focus factor: It is defined as the team velocity measured inhours divided by the work capacity. It represents the relationbetween the dedicated working hours and velocity in hours.Besides, it shows whether the team is over or under the fore-casted capacities.

Focus factor ¼ Team velocityðin hoursÞ=Work capacity

� Percentage of accepted work: It is defined as the result ofdividing the working hours dedicated to finished user storiesduring certain iteration by the total dedicated working hoursof the iteration. It represents the percentage of working hoursdedicated to deliver features to the user in certain iteration.

% of accepted work

¼XðDedicated working hours to finished stories in iterationÞ

.XðTotal dedicated working hours in iterationÞ

� Target Value Increase (TVI+): It is defined as the finished storypoints of the actual iteration divided by the average story pointsof all finished iterations. It represents the team productivityincrease in relation to the average performance throughoutthe project.

TVIþ ¼XðFinished story points in iterationÞ

.

Average story points per iteration

Tracking these metrics will uncover the number of workinghours along the iteration; the number of features that will be deliv-ered; the measure of the team’s forecasted capabilities; the per-centage of the team’s effective work that produces features todeliver and the team productivity improvement. These values willallow the team to self-manage. As it has been stated, this frame-work proposes to calculate them per Sprint and analyze them dur-ing the iteration retrospective. Guiding discussions will enhancefinding out the root of each impediment or problem and modifyingthe process to increase results.

To summarize, it has to be mentioned that our main contribu-tion is the combined use of the described productivity metricstogether with the usage of the rest of Agile techniques, mainlyAgile EVM. The aim is to propose a direct streamlined lightweightsystematic process to improve the team productivity throughSprints.

5. A practical example

This section describes a practical sample of the proposed frame-work, being our first empirical experience regarding the applica-tion of this framework. A single experience can or cannotvalidate it, although it can help to provide the first insights forour further research. Some of the results described in this sectionhave been already presented in our previous paper [75]. Thatwas focused on assessing the suitability of Agile practices in PublicAdministration and its adaptability to different kinds of projects(Web projects and infrastructure projects), giving only high-leveldetails of the project.

In this section, the full project and its results are described indepth. The example, called eBOJA project, was developed withinthe Ministry of Culture and Sports of the Regional Government ofAndalusia (Junta de Andalucía), in Spain. It is important to mentionthat all economic data included in this section are pure estimationsused for management purposes, not representing any realexpenditures.

5.1. The background of the project

Junta de Andalucía is the name of the regional government ofthe Spanish region of Andalusia, the body that has developed thisproject. The Information and Communication Technologies (ICT)Department of the Ministry of Culture and Sports has leaded theproject in liaison with other Departments belonging to the regionalgovernment.

The eBOJA project is mainly an e-Government project. The mainlegal framework to develop this kind of projects within the SpanishPublic Administration is the Law 11/2007 [41]. According to theprinciples it establishes, Junta de Andalucía has developed an e-Government framework called W@NDA [77]. W@NDA project’saxes focus on process reengineering and administrative proceduresimplification, with the aim of reducing costs and time andimproving citizens’ satisfaction. Additionally, the Official Journalof Junta de Andalucía (BOJA) is the name given to the official jour-nal of the regional government [22,23], being issued on paperalong the last thirty years. Some years ago, Junta de Andalucía


started a project to develop the necessary infrastructure to trans-form this journal into an electronic one, keeping all legal warran-ties. As a result of this initiative, the BOJA is issued electronically,with total legal validity, since May 10th 2012 [48].

Once the infrastructure is available, all institutional bodies ofJunta de Andalucía will adapt their systems and processes to useit. The ICT Department of the Ministry of Culture and Sports startedeBOJA project, to cope with that goal. Two main systems have beenused within the project: a Web application for internal communi-cations and a Web application to electronically sign official docu-ments, both of them developed under the aforementionedW@NDA project. eBOJA project included, among others tasks, thefollowing: design of administrative procedures, deployment ofapplications within the Ministry’s infrastructure, development ofsoftware APIs to protect the internal systems, interconnection ofthe applications with the general infrastructure or all the aspectsrelated to change management.

It is worth highlighting that this is a key system, because all reg-ulations approved by the Ministry shall be published through thistool. In contrast, the large number of involved stakeholders hasgiven the project an extra difficulty: managing a broad set of stake-holders, sometimes with opposed interests, as well as managing abig change in an uncertain environment, has posed a great challenge.

5.2. The technical context of the project

As it was previously introduced, eBOJA project is part of the e-Government strategy of the Ministry of Culture and Sports, alignedwith the general e-Government strategy of Junta de Andalucía.Fig. 13 shows the main components that interoperate within theproject.

Below, we will briefly describe the elements conforming thisfigure:

� A workflow engine for administrative procedures, calledTREW@. Based on it, a Web tool to operate the procedures,called eCO, which is a Web application centered on JEE architec-ture that uses Spring, Hibernate and JSF, runs as a front-end ofTREW@ (a.1). It also includes a graphic design tool to developadministrative procedures, based on AWT, where these proce-dures are designed through an XML-based language called XPDL(eXtended procedure Description language) (a.0). TREW@ con-tains an Oracle database schema and Java API to programmati-cally have access to the defined procedures.� A Web application to electronically sign documents, called

PORTAFIRMA. This Web application based on JEE architecture,using Spring, Hibernate and JSF. PORTAFIRMA compiles all thefunctionality required to electronically sign any administrative

Fig. 13. e-Government solution architecture for eBOJA.

document. It acts as the corporate multi-PKI platform front-end for authentication and electronic signature of Junta de And-alucía, called @FIRMA [77] (b.2, b.3). This platform is used by allSpanish public administrations, providing them with fast andeffective authentication and electronic signature services. POR-TAFIRMA communicates with TREW@ workflow engine via DSS-complaint Web Services (b.1, b.4).� A Web application to electronically submit documents to the

Official Journal of Junta de Andalucía, called ELECTRONICREMISSION. This application receives the documents sent byeCO through a Web Services layer (c.1, c.2). This platform hasbeen developed and it is currently maintained by the infrastruc-ture of the Ministry of Presidency. It is carried out through JEEarchitecture, using Spring and Hibernate.

5.3. The organizational environment of the project

This section summarizes some of the environmental character-istics of the project, pointing to the different elements influencingit. Fig. 14 outlines them as follows.

The most influencing element on the project is the compositionof the team. In the case of eBOJA, the project comprised a team offour members. Such a team was recognized by the followingcharacteristics:

� To be a real multi-disciplinary team, with different level ofexperience and knowledge.� To be used to work as a team and to self-organization.� To be integrated by experts in their own fields to guarantee the

knowledge of the basic tasks that they would be able to develop.

It must be highlighted that, with the exception of one member,no other team member had real experience in Agile methodologies.Before starting the project, the team members received a trainingsession regarding the grounds of Agile and during the project theywere coached by the one having Agile expertise.

Furthermore, eBOJA faced one main constraint, the schedule.That meant a strict deadline fixed by law established by the Regio-nal Government. For that reason, delivering the project on timewas one of the main goals of the team.

Additionally, eBOJA was developed thanks to the collaborativework performed among several stakeholders:

� The ICT Department System Operation teams, who were incharge of some of the necessary tasks to deliver users storieson time.� The Directors Board of the Ministry, who was responsible for

supporting the team during release and change managementprocesses.

Fig. 14. Elements having impact on the project.


� Several other Ministries, who were in charge of delivering ontime some of the components used in the project.

The large number of stakeholders (for instance the differentMinistries involved) gave this project an extra difficulty, apart fromthe technical aspects.

Lastly the project influences both internal and external users:

� From internal user perspective, eBOJA represents a great changein the Ministry’s internal users behavior since, before beingimplemented, all tasks dealing with publishing laws and regula-tions were manually issued on paper. The deployment of thesystem implies a real change of paradigm, which has to be care-fully handled.� From external user perspective, mainly citizens and companies,

the project is relevant since all laws and regulations approvedby the Ministry of Culture and Sports shall be published throughelectronic means. As examples, the system manages both, allprocurement processes and all financial aids granted to the cul-tural industry.

5.4. Results

The project started in December 2011 with the project-launch-ing phase, including the development of a project charter and twoworkshops performed in the first weeks of January, to allow theteam defining the required features in the shape of user stories.The initial Product Backlog included 14 user stories, scoring a totalof 56 story points. Four more user stories were added in the courseof the project, one of the original was deleted and some of the ini-tial estimations were updated. These facts entailed that the Prod-uct Backlog size ranged from 74 to 56. Using Agile planning andestimating framework, the next step was establishing the durationof the iteration and velocity. The iteration length was fixed in threeweeks and, because of the absence of historical data, team velocitywas forecasted taking into account the available working hours ofeach team member during the iteration. Table 1 shows the initialforecasted velocity.

The team developed an initial project plan and estimation costafter establishing the length of the iteration and team velocity.Table 2 summarizes the main aspects of the initial plan, includinga level of uncertainty suggested by the actual phase of the project[18].

Sprint-0 started in January 30th 2012, after defining the initialplan. The team was able to lay the foundation for managing theproject with this first iteration. Once finished, the normal Sprint-based lifecycle started. Table 3 represents the main figures of theproject.

As it can be observed in Table 3, the project included 5 itera-tions (4 of them lasting three weeks and one four weeks), being fin-ished in June 23rd 2012. It has to be mentioned that the Ministry ofCulture and Sports was the first body of Junta de Andalucía inadapting the systems to the new official journal. Regarding the lastSprint, it lasted an extra week, in order to include all remainingstories, avoiding a short final Sprint of one week.

As mentioned, size, composition and priorities varied during theproject and ROI was a key element to decide over Product Backlogchanges. The new features were included during Product BacklogGrooming sessions, where business representatives provided them

Table 1Initial forecasted velocity.

Available hours Estimated initial velocity Hours per story point

150 11 13.64

with value estimation. The team estimated their size and calcu-lated their ROI. If the ROI of one of the story was higher than a pre-vious one, then the Product Backlog would be reordered and theRelease Plan changed. Table 4 presents the results of the projectin relation to the values included in the initial plan.

Regarding Agile management tools used, Fig. 15 presents theburn-up chart of the project, with the evolution of the remainingamount of work in comparison with the finished amount of work,both measured in story points.

This tool shows how much work was finished attending to theremaining amount of work. It reflects the changes on the size ofthe project caused by the continuous reassessment of priorities,using value and ROI as main tools. Table 5 shows the evolutionper iteration in order to measure the delivered value.

Based on these data, Fig. 16 lets us know the accumulated busi-ness value delivered per iterations.

As it was already mentioned, the main goal of this tool is help-ing the team deliver most of the value at the beginning of the pro-ject. It can be noticed that this goal was only partially achieved dueto several reasons, such as the necessity of delivering first sometechnical stories not providing clear value to the customer or theinitial fluctuation in velocity (as the value delivered was linkedto the finished stories and during the first three iterations the valuechanged).

Moreover, Table 6 shows EVM calculations for each of the pro-ject’s iterations as an Agile management tool in order to control theproject constraints.

Table 6 also reveals how the different indicators used to controlproject constraint were evolving through Sprints. It is worth point-ing out that these indicators were used as a main tool to learnthrough the project in the Sprint retrospective meetings. Theyproved to be very useful, and as shown, they later helped to stabi-lize the project in the last Sprints. Fig. 17 demonstrates the evolu-tion of the planned value against the earned value.

It can be observed that the team was overcommitted during thefirst three iterations. Specially, in Sprint 3, the team was not able todeliver what was committed, because of some uncertainties on therequirements definition and lack of knowledge on the team part.From iteration 4 on, the team improved their estimations. As men-tioned, this stabilization can be caused by the different projectindicators described in the proposed framework, as they were usedto re-estimate the Product Backlog during the different Sprintmeetings. Fig. 18 represents the evolution of the earned valueagainst the actual cost.

It shows that the team tended to overestimate the necessaryamount of work. This element is important, as confirms that theteam improved its work, but keeping a conservative approach dur-ing the last Sprints of the project. The team discussed about thisfact during the Sprint retrospective meetings based on the data,but it decided to keep a conservative approach during the wholeproject, probably due to either a natural tendency to self-protec-tion or, as it was its first Agile experience, some lack of knowledgeabout the context or the framework. Fig. 19 shows the evolution ofCPI and SPI indexes through the project’s iterations.

Lastly, Table 7 offers the results of the productivity metricsobtained during the project estimated at iteration level, whichallowed the team to assess and adapt its behavior throughoutthe project.

Table 8 offers the average data of each metric for the entireproject.

One relevant thing that can be highlighted is how the teamimproved its performance, as it can be seen in the average valueof TVI+, which is above 100%, even with oscillations in teamvelocity during Sprint 3. Another figure that can be emphasizedis the percentage of accepted work, which shows how the teamwas able to focus on valuable work in the project.

Table 2Summary of the initial project plan.

Value Uncertainty (± 25%) Initial estimation

Total No of story points 56 14 56 ± 14Velocity (forecast) 11 3 11 ± 3No of expected iterations 5 1 5 ± 1Length of iteration 3 weeks N/A N/ADuration of project 15 weeks 3 weeks 15 ± 3 weeksHours per iteration 150 38 150 ± 38Hours per Project 764 191 764 ± 191Cost per iteration € 3694.34 € 923.59 € 3694.24 ± € 923.59Total Project cost € 18807.55 € 4701.89 € 18807.55 ± € 4701.89

Table 3Main figures of the project.

Sprint Startdate

Enddate

Workingdays

Est.velocity

Realvelocity

Finish. storypoints

Product Backlog storypoints.

Remain. storypoints

Estimatedhours

Realhours

1 30/01 20/02 15 11 6 6 56 50 150 1172 21/02 14/03 15 14 14 20 74 54 126 793 15/03 09/04 15 12 2 22 74 52 150 884 16/04 07/05 15 10 15 37 71 34 147 1185 08/05 23/06 20 19 19 56 56 0 101 157

674 559

Table 4Project results vs initial plan values.

Results Planned

Average velocity 11.20 11 ± 3Total working days 80 N/ATotal working hours 559 764 ± 191Total story points 56 56 ± 14Hours per story point 9.98 13.64

Fig. 15. Burn-up chart of the project.

Table 5Delivered value by Sprint.

Sprint Value of finished stories (estimated) Value of finished storie

1 13,000 11,0002 24,000 24,0003 6500 10004 11,000 11,5005 20,500 20,500


Another important aspect that can be extracted from the pre-sented values is that they show an overview of the team perfor-mance along the project. This fact clarifies whether somenegative values are only isolated cases or are related to underlyingproblems. Fig. 20 represents the relation between team velocityand work capacity.

It is realized that during the project, the relation between thesetwo magnitudes was primarily above 100%. Normally, a valuehigher than 80% reveals a tendency to overestimation [26]. Thisfact is aligned with the results obtained from EVM calculations.Fig. 21 confirms the evolution of the percentage of accepted work.

This value is between 80% and 95%, which means the teammainly focused on finishing work concerning delivered featuresat the end of each Sprint. Fig. 22 outlines the evolution ofTVI + during the project.

The team improved its capacity to deliver a value to the cus-tomer in each Sprint, with the exception of Sprint 3.

To sum up, the correct use of this value can help any team knowits situation, and within the Sprint retrospectives, adapt andimprove estimations and performance.

5.5. Discussion on the results obtained

In this section we will analyze the results presented in the pre-ceding section, in order to help us come to general conclusions. Asmentioned, both quantitative (through project metrics) andqualitative (through project observation) data have been gathered,and this section will address both of them separately.

5.5.1. Quantitative data analysisThis section will analyze the project metrics presented in Sec-

tion 5.3 with the aim to obtain meaningful information.

s (real) % Delivered value (%) % Delivered value (accumulated)

16.18 16.1835.29 51.47

1.47 52.9416.91 69.8530.15 100.00

Fig. 16. Delivered value per Sprint.

Table 6EVM calculations.

Sprint Plan. compl. story points EPC (%) APC (%) Estimated work hours Real no. of working hours PV EV AC SPI CPI

1 11 19.64 10.71 150 117 € 3694.34 € 2015.09 € 2880.54 0.55 0.702 14 35.71 35.71 126 79 € 6716.98 € 6716.98 € 4825.52 1.00 1.393 12 57.14 39.29 150 88 € 10747.17 € 7388.68 € 6992.08 0.69 1.064 10 57.14 66.07 147 118 € 10747.17 € 12426.42 € 9897.24 1.16 1.265 19 100.00 100.00 101 157 € 18807.55 € 18807.55 € 13762.58 1.00 1.37

Fig. 17. Planned and earned value per iteration.

Fig. 18. Earned value and actual cost per iteration.


As an initial thought, Fig. 17 states that the framework allowsthe team to improve estimation regarding scope through Sprints,becoming a valuable learning tool. This framework also providesfrequent feedback loops that are very suitable for Web develop-ment projects requirements. As the data presented in Table 5 dis-plays, modifications on the Product Backlog (stories added, deletedand adjusted) are guided by the value that users and customersassign. This enables reassessing and reordering the Product Back-log and Release Plan throughout the project, and being able to deli-ver not only the planned system, but also the desired system by theend of the project.

The framework executes project plan and estimation as a con-tinuous task along the project, not exclusively in the initial phase,as it can be observed in Tables 3 and 6. This approach helps teamsto deal with Web projects with changing requirements that some-times are unknown at the beginning. This fact, together with thepossibility of having an initial project plan, could allow organiza-tions to make medium and long-term decisions in relation to theirproject portfolios. The process of continuous planning is shown in

Table 2 and the subsequent table, which illustrate the initial planand how it evolves throughout the project.

In addition, project estimation and planning do not depend onlate product size (such as lines of code or number of pages), whichleads towards ‘‘requirements-freeze’’, but on relative estimations,including periodic milestones, which reflect and adapt estimationsto the new circumstances, as Tables 2 and 3 show. For this purpose,value estimations and ROI calculations proved to be a very usefulasset, as they headed the process of re-estimation and the decisionof what must be developed and when.

Nevertheless, the ‘‘Size’’ attribute of user stories jointly evalu-ates risk, complexity and uncertainty elements of each attribute.This fact simplifies the estimation process as well as makes estima-tions oscillate during the project, as Table 3 represents. Forinstance, the estimation of the ‘‘Size’’ attribute of certain story usu-ally decreases when risks are reduced or uncertainties are resolved.

Fig. 19. Evolution of CPI and SPI per iteration.

Table 7Productivity metrics by Sprint.

Sprint Real velocity Average Hour/Story point Team velocity (in hours) Work capacity Focus factor (%) Accepted work % Of accepted work TVI+ (%)

1 6 19.5 117 117 100 96.25 82.26 1002 14 9.79 137.03 78.75 174 69.75 88.57 1403 2 12.86 25.73 87.25 29.49 70.5 80.80 27.274 15 10.83 162.47 117.75 137.98 110.75 94.06 162.165 19 9.96 189.15 156.75 120.67 131.75 84.05 169.64

Table 8Productivity metrics for the project.

Number of Sprints 5

Team velocity (in story points) 11.2Hours/story point 9.96Team velocity (in hours) 126.27Team capacity 111.5Focus factor 111.25%% Of accepted work 85.95%TVI+ 119.82%

Fig. 20. Team velocity vs work capacity.

Fig. 21. Percentage of accepted work.

Fig. 22. Target value increase per Sprint.


There is also an element to take into account: some technicalnon user-valuable stories need to be developed first, in order toprovide technical architecture. The presented data confirm thatthis fact partially impeded the goal of delivering the higher valuefirst, as Table 5 outlines, and should require some adaptations tothe framework.

Tools such as charts representing the evolution of the deliveredvalue like Fig. 16, based on the ‘‘Value’’ attribute of user storiesgiven by customers, let the team know whether it is deliveringmost valuable stories first and then, if it focuses on costumersneeds. Tracking this value through iterations enables quick adapta-tion and correction of any non-desired effects. However, asreferred, the goal consisting in clearly delivering up-front valuehas not totally been met due to technical constraints and the initialfluctuation of velocity. A mechanism to solve this problem andspread the business value of the business stories to the technicalones could be put in place, and it could constitute the object of afuture proposal.

5.5.2. Qualitative data analysisThis section will expose and discuss the different qualitative

project observations. As an initial one, and regarding Plan and Esti-mate phases, we can point out that the framework involves allmembers of the team in planning the project. Everybody discussesand achieves a common estimation of the feature by means ofPlanning poker, which was particularly chosen for being one of


the estimation techniques that can better meet the characteristicsof Web development, as explained in Section 4.2. It was proven tobe an efficient and collaborative tool to estimate the work both atproject and Sprint levels. Additionally, introducing value estima-tion in each user story makes users and customers be engaged withthe estimation process and also enables identifying the real needs.

Besides, the proposed framework allows the team to learn andimprove how to estimate during the project as well as visualizeits own capacity of delivering finished features to the customer.Using productivity metrics and EVM calculations to guide Sprintretrospectives, as shown in the project data, helped to achieve thisgoal. As the management phase concerns, Agile teams can intro-duce aspects like provided value and quality, that are also crucialfor the project to succeed thanks to the proposed change in the clas-sical ‘‘Iron triangle’’. The double check on quality (product and pro-cess) ensures that quality is built-in during the project. Sprintreview meetings, where the team showed the developed featuresfulfilling the testing criteria, additionally became key tools.

Teams can control project constraints, like schedule or cost,through EVM calculations based on the Agile approach. These calcu-lations, obtained at a Sprint level, can be appreciated as useful toolsduring iteration retrospectives, so as to obtain indications on poten-tial problems in development processes. The ‘‘Status report’’ consti-tutes a useful tool to provide all project stakeholders withinformation such as changes on the Product Backlog, features andvalue delivered or constraints status, for example. This can enhancethe feeling of openness and teams and stakeholders’ degree of com-mitment throughout the project. Using this tool was very importantin a project like this, with a large number of different stakeholders, asit helped to handle the different expectations. In Measure and Adaptphases, EVM calculations and productivity metrics, together withinnovation games, can offer a very practical tool to identify potentialproblems and obstacles and locate their roots causes. The correct useof this data can implement the capacity of the teams to lead projectsto the correct direction.

It is important to mention that having a member of the teamfully experienced in the use of Agile, acting as a coach of the restof his/her colleagues, appeared to be very valuable, too. This col-league facilitated the transition from a classical to an Agileapproach, even if elements like self-protection on time estimationsdid not completely disappear.

Finally, it has to be added that the size of the project (a smallproject of 4 persons during 6 months) can also influence theresults. On the one hand, the reduced size favored the capabilityof quick adaptation as well as communication among the team.The short duration helped the team focus on the main goals (todeliver on time, which, as mentioned, was the main restriction ofthe project), but maybe it also constrained the learning process(as shown in the continuous overestimation tendency of the team).As a main conclusion, it can be stated that the framework proved tocorrectly behave in this type of projects.

6. Conclusions and future work

This section states the research conclusions, stressing thestrengths and weaknesses of the proposed Agile framework,including relevant information resulting from applying it to a realproject. These conclusions can provide valuable insights to guidethe future work, although further research is needed.

As previously stated in Section 2, the general research questionis structured into 3 research questions. Based on the results, somearguments can be provided to answer them, as shown below:

� RQ1: What are the suitable existing techniques for estimating,planning and managing Agile Web projects? To answer this ques-tion, it must be commented that we have assessed and a set of

suitable practices (as Planning Poker, Value points estimation,ROI calculation or Agile EVM, among others) suitable to esti-mate, plan and manage Agile Web projects in Sections 3 and 4.� RQ2: Can business value help estimation, plan and management of

Agile Web projects? With regard to this research question, Sec-tion 4 explains a way to identify business value linked todesired features, named ROI, and how to use it as guidancefor re-estimation and adaptation in Agile Web projects.� RQ3: Can the identified techniques be integrated into an Agile com-

mon framework, appropriate for Web projects needs? To answerthis research question it must be clarified that some of theseidentified practices have been adapted and modified in orderto integrate them into a coherent framework, presented in Sec-tion 4, with the goals of better suiting the characteristics of Webprojects and keeping their agility.

As a conclusion, and with the aim of answering our main ques-tion about the feasibility of an Agile approach to plan, estimate andmanage Web projects guided by value, we have identified and inte-grated an existing set of Agile practices in a framework and theyhave been tested in a real-world project. As analyzed, the proposedframework focuses on a continuous Plan and Estimate–Manage–Measure–Adapt cycle, using different techniques in each phase tohelp teams handle Web projects. It is worth pointing out that theproposed framework seems to fit well the special characteristicsof Web projects described in Section 1.

The preceding sections have introduced an Agile approach toWeb development project estimation, plan and management, aswell as the result of an empirical example dealing with the prac-tical application of the framework. The proposed framework triesto address the main characteristics of estimation and manage-ment in Web development projects, by offering a balancebetween agility and ability to change and medium and long-termcapability to plan and control project constraints. The frameworkalso includes some techniques and metrics that provide teamswith objective data, for their processes of continuously improvingtheir performance.

Although the proposed framework has been designed with theaim of covering all specific characteristics of Web projects, as men-tioned before, some of them are shared with non-Web system.Even though the scope of our research is Web environments, thesuitability of the framework to more general projects could beassessed as part of future research.

The results of the practical experience, based on the framework,are very encouraging and evidence that this kind of approach willbe very appropriate to Web development projects. However, fur-ther research is needed to clarify how the business value isassigned to user stories and their interdependencies.

Extending the proposed framework, including other Agile prac-tices and methods, by creating one that would be successfullyassessed against a maturity model like CMMI would be very inter-esting. This extended framework will supply organizations withthe possibility of combining quick responses, changing capacitiesand lightweight processes with continuous improvement, system-atic use of data and advanced management practices, as well asvalidating these capabilities with a well-known and well-estab-lished model like CMMI.

Acknowledgements

This research has been supported by the MeGUS Project(TIN2013-46928-C3-3-R) of the Ministry of Economy and Compet-itiveness, Spain. We would like to thank the Regional Ministry ofCulture and Sports of Junta de Andalucía for allowing us to issuethis data


References

[1] A.J. Albrecht, J.E. Gaffney, Software function, source lines of codes, anddevelopment effort prediction: a software science validation, IEEE TSE SE-9(1983) 639–648.

[2] G. Alleman, et al., Making agile development work in a government contractingenvironment – using earned value to measure velocity, in: AGILE 2003,Proceedings of Agile Conference, 2003, Salt Lake City, Utah, 25–27 June 2003.

[3] Agile COCOMO II. <http://sunset.usc.edu/cse/pub/research/AgileCOCOMO/AgileCOCOMOII/Main.html> (accessed 1.9.14).

[4] S.W. Ambler, Lessons in Agility from Internet-Based Development, IEEESoftware, 2002. pp. 66–73. (March–April).

[5] S. Ambler, Generalizing Specialists: Improving Your IT Career Skills. <http://www.agilemodeling.com/essays/generalizingSpecialists.htm>, 2012 (accessed1.9.14).

[6] J. Appelo, Management 3.0. Leading Agile Developers, Developing AgileLeaders, Addison-Wesley, Boston, 2009.

[7] J. Armario, J.J. Gutiérrez, M. Alba, J.A. García-García, J. Vitorio, M.J. Escalona,Project estimation with NDT, in:, ICSOFT 2012 Proceedings of the 7thInternational Conference on Software Paradigm Trends, Rome, Italy, July 24–27 2012, pp. 120–126.

[8] K. Beck, et al., Manifesto for Agile Software Development. <http://www.agilemanifesto.org>, 2001 (accessed 1.9.14).

[9] K. Beck, Test-Driven Development by Example, Addison-Wesley, Boston, 2002.[10] K. Beck, C. Andres, Extreme Programming Explained: Embrace Change, second

ed., Addison-Wesley, Boston, 2004.[11] A. Begel, N. Nagappan, Usage and perceptions of agile software development in

an industrial context: an exploratory study, in: ESEM ‘07, Proceedings of the1st International Symposium on Empirical Software Engineering andMeasurement, Madrid, Spain, September 21–27 2007, IEEE.

[12] B.W. Boehm, Software Engineering Economics, Prentice-Hall, Englewood Cliffs,NJ, 1981.

[13] B.W. Boehm, C. Abts, A. Winsor Brown, S. Chulani, B.K. Clark, E. Horowitz, R.Madachy, D.J. Reifer, B. Steece, Software Cost Estimation with COCOMO II,Prentice-Hall,, Englewood Cliffs, NJ, 2000.

[14] S. Ceri, P. Fraternali, A. Bongio, Web modelling language (WebML): a modellinglanguage for designing web sites, Comput. Networks 33 (1–6) (2000) 137–157.

[15] A. Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams,Addison-Wesley, Boston, 2004.

[16] A. Cockburn, Balancing Lightness with Sufficiency. <http://alistair.cockburn.us/Balancing+lightness+with+sufficiency>, 2010 (accessed 1.9.14).

[17] M. Cohn, User Stories Applied: For Agile Software Development, Addison-Wesley, Boston, 2004.

[18] M. Cohn, Agile Estimating and Planning, Addison-Wesley, NJ, 2005.[19] M. Cohn, Succeeding with Agile Using Scrum, Addison-Wesley, Boston, 2009.[20] R. Corral, Gestión ágil de proyectos: experiencias prácticas y resultados

obtenidos. <http://www.ermua.es/pags/noticias/archivos/2011/20110331/presentacion_rodrigo_corral.pdf> (accessed 1.9.14).

[21] J.W. Creswell, Research Design: Qualitative, Quantitative, and Mixed MethodApproaches, second ed., SAGE, 2003.

[22] Decreto 1/1979, de 30 de julio, sobre creación y publicación del Boletín Oficialde la Junta de Andalucía. <http://www.juntadeandalucia.es/boja/1979/1/3>,1979 (accessed 1.9.14).

[23] Decreto 205/1983, de 5 de octubre, por el que se aprueba el Reglamento delBoletín Oficial de la Junta de Andalucía. <http://www.juntadeandalucia.es/boja/1983/82/1>, 1983 (accessed 1.9.14).

[24] Y. Deshpande, S. Marugesan, A. Ginige, S. Hanse, D. Schawabe, M. Gaedke, B.White, Web engineering, J. Web Eng. 1 (1) (2002) 3–17.

[25] E. Derby, D. Larsen, Agile Retrospectives. Making Good Teams Great, ThePragmatic Bookshelf, Dallas, 2006.

[26] S. Downey, J. Sutherland, Scrum metrics for hyperproductive teams: how they flylike fighter aircraft, in: HICSS 2012, Proceedings of the 45th Hawaii InternationalConference on System Science, 2012, Maui, Hawaii, USA, January 4–7 2012.

[27] T. Dybå, T. Dingsoyr, Empirical studies of agile software development: asystematic review, InfSoftwTechnol 50 (9–10) (2008) 833–859.

[28] M.J. Escalona, J. Torres, M. Mejías, J.J. Gutiérrez, D. Villadiego, The treatment ofnavigation in web engineering, Adv. Eng. Softw. 38 (4) (2007) 267–282.

[29] M.J. Escalona, G. Aragón, NDT: a model-driven approach for web requirements,IEEE Trans. Softw. Eng. 34 (3) (2008) 370–390.

[30] R. Fewster, E. Mendes, Measurement, Prediction and Risk Analysis for WebApplications, in: Proceedings of Metrics ’01, London, 2001.

[31] H. Glazer, J. Dalton, D. Anderson, M. Konrad, S. Dhrum, CMMI or agile: why notembrace both!, in: CMU/SEI-2008-TN-003, Pittsburgh, 2008.

[32] D. Gray et al., Gamestorming. A Playbook for Innovators, Rulebreakers andChangemakers, O’Reilly, CA, 2010.

[33] J. Highsmith, Agile Project Management: Creating Innovative Products, seconded., Addison-Wesley, NJ, 2009.

[34] JIRA Website. <https://www.atlassian.com/es/software/jira> (accessed 1.9.14).[35] J. Johnson (Ed.), XP2002, Report to the Third International Conference on

Extreme Programming and Agile Processes in Software Engineering, Alghero,Italy May 26–29, 2002.

[36] M. Jorgensen, Practical Guidelines for Expert Judgment-Based Software EffortEstimation, OsloUniv., Norway, 2005 (Software).

[37] Gustav Karner, Resource Estimation for Objectory Projects, Objective SystemsSF AB, 1993.

[38] N. Koch, A. Knapp, G. Zhang, H. Baumeister, UML-Based Web Engineering: AnApproach Based on Standards, Web Engineering: Modelling and ImplementingWeb Applications, Springer, 2008.

[39] N. Koch, A. Knapp, S. Kozuruba, Assessment of effort reduction due to model-to-model transformations in the web domain, in: ICWE 2012, Proceedings ofthe 12th International Conference on Web Engineering, BErlin, Germany 23–27 July 2012.

[40] C. Larman, V.R. Basili, Iterative and Incremental Development: A Brief History,IEEE Comput. 36 (6) (June 2003) 47–56.

[41] Law 11/2007, 22th June, on Citizen’s Electronic Access to Public Services.<http://www.boe.es/boe/dias/2007/06/23/pdfs/A27150-27166.pdf>, 2007(accessed 1.9.14).

[42] K.K. Lilja, K. Laakso, J. Palomki, Using the delphi method, in: PICMET11,Proceedings of the 2011 Portland International Center for Management ofEngineering and Technology Conference: Technology Management in theEnergy Smart World, Portland, Oregon, USA, July 31–August 4, 2011, IEEE.

[43] K. Logue, K. McDaid, Agile release planning: dealing with uncertainty indevelopment time and business value, in: ECBS 2008, Proceedings of the 15thAnnual IEEE International Conference and Workshop on the Engineering ofComputer Based Systems, 2008, Belfast, Northern Ireland, March 31 2008–April 4 2008.

[44] E. Mendes, S. Counsell, N. Mosley, Measurement and effort prediction of webapplications, in: Proceedings of the 2nd ICSE Workshop on Web Engineering,Limerick, Ireland, June 2000.

[45] E. Mendes, N. Mosley, Web Cost Estimation: An Introduction, WebEngineering: Principles and Techniques, IGI Global, 2005.

[46] L. Olsina, Building a Web-based information system applying the hypermediaflexible process modelling strategy, In: Workshop on HypermediaDevelopment Processes, Methods and Models (Hypertext 98), Pittsburgh,USA, 1998.

[47] OMG – Object Management Group, IFML: The Interaction Flow ModelingLanguage. <http://www.ifml.org/?page_id=99> (accessed 1.9.14).

[48] Orden de 23 de abril de 2012, por la que se regula la inserción de documentosen el Boletín Oficial de la Junta de Andalucía. <http://www.juntadeandalucia.es/boja/2012/86/1.html>, 2012 (accessed 1.9.14).

[49] S.R. Palmer, A Practical Guide to Feature-Driven Development, Prentice-Hall,NJ, 2002.

[50] M. Pikkarainen et al., The Impact of Agile Practices on Communication inSoftware Development, Empirical Software Engineering, Springer, 2008 (May2008).

[51] M. Poppendieck, T. Poppendieck, Lean Software Development. An Agile Toolkit,Addison-Wesley, Boston, 2003.

[52] R.S. Pressman, What a Tangled Web We Weave, IEEE Software, 2000 (January–February 2000).

[53] Project Management Institute, Project Management Book Of Knowledge(PMBOK) Guide, fourth ed., Project Management Institute, 2008.

[54] Z. Racheva, M. Daneva, K. Sikkel, A. Herrmann, R. Wieringa, Do we knowenough about requirements prioritization in agile projects: insights from acase study, in: Proceedings of the 18th IEEE International RequirementsEngineering Conference, Sydney, Australia, September 27 2010–October 12010), RE 2010.

[55] H. Ran, et al., Agile web development with web framework, in: WiCOM ’08,Proceedings of the 4th International Conference on Wireless Communications,Networking and Mobile Computing, Dalian, China, October 12–17 2008, IEEE,2008.

[56] Redmine Website. <http://www.redmine.org/> (accessed 1.9.14).[57] D.J. Reifer, Web Development: Estimating Quick-to-Market Software, IEEE

Software, 2000 (November–December 2000).[58] M. Register, T. Golding, Using agile for buy vs build decisions, in: AGILE 2008,

Proceedings of the Agile Conference, 2008, Toronto, Canada, 4–8 August 2008.[59] P. Runeson, M. Höst, Guidelines for conducting and reporting case study

research in software engineering, Empirical Softw. Eng. 14 (2009) 131–164.[60] K. Schwaber, Scrum development process, in: OOPSLA ’95, Proceedings of the

10th Annual ACM Conference on Object Oriented Programming Systems,Languages and Applications, Austin, TX, USA 15–19 October 1995. ACM.

[61] K. Schwaber, Controlled Chaos: Living on the Edge, American Programmer,1996. April.

[62] K. Schwaber, Agile Project Management with Scrum, Microsoft Press,Redmond, 2004.

[63] M. Shepperd, C. Schofield, B. Kitchenham, Effort Estimation Using Analogy,Dept. of Comput., Bournemouth Univ. Software Engineering, 1996.

[64] J. Shore, S. Waden, The Art of Agile Development, O’Really Media, CA, 2008.[65] P. Solomon, R. Young, Performance-Based Earned Value, Wiley-IEEE Computer

Society, 2006.[66] G. Smith, A. Sidky, Becoming Agile in an Imperfect World, Manning

Publications, CT, 2009.[67] T. Sulaiman, B. Barton, T. Blackburn, AgileEVM – Earned Value Management in

Scrum Projects, in: AGILE 2006, Proceedings of the Agile Conference, 2006,Minneapolis, Minnesota, 23–28 July 2006.

[68] K. Sungjoo, C. Okjoo, B. Jongmoon, Model-based dynamic cost estimation andtracking method for agile software development, in: IEEE/ACIS ICIS 2010,Proceedings of the 9th Computer and Information Science Conference,Kaminoyama, Japan 18–20 August 2010.

[69] J. Sutherland, A. Viktorov, J. Blount, N. Puntikov, Distributed scrum: agileproject management with outsourced development teams, in: HICSS 2007,

http://refhub.elsevier.com/S0950-5849(15)00014-2/h0395



http://sunset.usc.edu/cse/pub/research/AgileCOCOMO/AgileCOCOMOII/Main.html

http://sunset.usc.edu/cse/pub/research/AgileCOCOMO/AgileCOCOMOII/Main.html




http://www.agilemodeling.com/essays/generalizingSpecialists.htm

http://www.agilemodeling.com/essays/generalizingSpecialists.htm




http://www.agilemanifesto.org

http://www.agilemanifesto.org


















http://alistair.cockburn.us/Balancing+lightness+with+sufficiency

http://alistair.cockburn.us/Balancing+lightness+with+sufficiency








http://www.ermua.es/pags/noticias/archivos/2011/20110331/presentacion_rodrigo_corral.pdf

http://www.ermua.es/pags/noticias/archivos/2011/20110331/presentacion_rodrigo_corral.pdf

http://www.juntadeandalucia.es/boja/1979/1/3




















https://www.atlassian.com/es/software/jira










http://www.boe.es/boe/dias/2007/06/23/pdfs/A27150-27166.pdf




http://www.ifml.org/?page_id=99

http://www.juntadeandalucia.es/boja/2012/86/1.html

http://www.juntadeandalucia.es/boja/2012/86/1.html














http://www.redmine.org/
























Proceedings of the 40th Annual Hawaii International Conference on SystemSciences, Waikoloa, Big Island, Hawaii. 3–6 January 2007.

[70] J. Sutherland, K. Schwaber, The Scrum Guide: The Definitive Guide to Scrum:The Rules of the Game. <http://www.scrum.org/Scrum-Guides>, 2011(accessed 1.4.14).

[71] H. Takeuchi, I. Nonaka, The new product development game, Harvard Bus. Rev.64 (1) (1986) 137–146 (January–February 1986).

[72] The Standish Group International, Inc., Extreme Chaos, The StandishGroup International Inc. <http://www.standishgroup.com>, 2001 (accessed1.9.14).

[73] The Standish Group International Inc, The CHAOS Manifesto 2012. <http://www.versionone.com/assets/img/files/CHAOSManifesto2012.pdf>, 2012 (accessed1.9.14).

[74] C.J. Torrecilla-Salinas, et al., A scrum-based approach to CMMI maturity level 2in Web Development environments, in: iiWAS ’12, Proceedings of theInternational Conference on Information Integration and Web-basedApplications & Services 2012, Bali, Indonesia December 3–5 2012, ACM.

[75] C.J. Torrecilla-Salinas, et al., Agile in public administration: oxymoron orreality? An experience report, in: CAiSE ’13, Proceedings of the Industrial Trackof the Conference on Advanced Information Systems Engineering 2013,Valencia, Spain, June 21, 2013.

[76] VersionOne Website. <www.versionone.com>. (accessed 1.9.14).[77] W@NDA Project, Workflow in Andalusia Public Administration. <https://

ws024.juntadeandalucia.es/ae/descargar/2793>, 2012 (accessed 1.9.14).[78] M. Yap, Value based extreme programming, in: AGILE 2006, Proceedings of the

Agile Conference, 2006, Minneapolis, Minnesota, 23–28 July 2006.

http://www.scrum.org/Scrum-Guides



http://www.standishgroup.com

http://www.versionone.com/assets/img/files/CHAOSManifesto2012.pdf

http://www.versionone.com/assets/img/files/CHAOSManifesto2012.pdf

http://www.versionone.com

https://ws024.juntadeandalucia.es/ae/descargar/2793

https://ws024.juntadeandalucia.es/ae/descargar/2793

estimating, planning and managing agile web …gibson/teaching/teaching...estimating, planning and...

Documents