Integrating Testing into Agile Software Development Processes

R. van den Broek 1, M.M. Bonsangue 2, M. Chaudron 3, and H. van Merode 4

1JEM-id, The Netherlands2LIACS – Leiden University, The Netherlands3Chalmers – University of Gothenburg, Sweden4KLM Royal Dutch Airlines, The Netherlands

vdbroek.robin@gmail.com, m.m.bonsangue@liacs.leidenuniv.nl, michel.chaudron@cse.gu.se

Keywords:Software Processes, Agile Development, Scrum, Testing, Agile Testing

Abstract:Although Agile methodologies have grown very popular, there is a limited amount of literaturethat combines Agile software methodologies and testing, especially on how testing is integratedwith Scrum. In this paper we present an analysis of problem based on case study performed at theIT department of KLM regarding testing in a Scrum team. After having triangulated our resultswith several interviews with external topical experts and existing literature we propose a visualmodel that integrates testing activities in Scrum.

1 INTRODUCTION

Over the last years more and more organizationshave started to adopt, or already have adopted,Agile methodologies for their software develop-ment processes (VersionOne, 2011). The in-creased pressure of the Internet industry on real-izing a fast time-to-market, designing flexible pro-cesses, and responding to changing requirementsposes several challenges to organizations’ estab-lished software development approaches. Com-bined with the continuous struggle software de-velopment teams face in the dilemma of increas-ing productivity while also maintaining and/orimproving the quality of the software delivered(Lindvall et al, 2004), organizations and teamsare motivated to look out for new ways to developtheir software.

Agile methodologies build upon the valuesexpressed in the Agile Manifesto (Cunningham,2001) and aim to achieve close customer collab-oration, to deliver business value as soon as pos-sible in an incremental manner, and to respondto changing customer requirements (Barlow et al,2011; Cockburn, 2003). The methodologies cap-tured under the umbrella term ”Agile methodolo-gies” are not new concepts per se, but the com-

bination of existing best practices with new tech-niques and a new mindset make them a refreshingand stable approach towards software develop-ment. Although the benefits of Agile methodolo-gies seem to be appealing, misinterpretations ofthe Agile manifesto and/or inappropriate projectcontexts can hinder teams in achieving their goals(Barlow et al, 2011). Furthermore, various re-searchers have recognized that there is no ”onesize fits all” (Barlow et al, 2011; Boehm andTurner, 2003a; Lindvall et al, 2004) methodol-ogy for software development, and organizationsshould assess both a project’s context and the or-ganizational context when selecting a suitable de-velopment approach (Boehm and Turner, 2003b).

At the moment, the most frequently practicedAgile methodology is Scrum (VersionOne, 2011).Scrum provides a lightweight process frameworkthat can be described in terms of roles (prod-uct owner, scrum master, team), process (plan-ning, iteration, review), and artifacts (productand sprint backlogs, burndown charts) (Schwaberand Sutherland, 2009).

Testing in Agile projects is different from tra-ditional testing because of the continuous and in-tegrated nature of testing in the project lifecy-cle from the very beginning (Crispin and Gre-

gory, 2009; Lindvall et al, 2004; Talby, Keren,Hazzan, and Dubinsky, 2006). Furthermore, be-cause every iteration aims to deliver a ”poten-tially shippable” product, the developed function-ality within every iteration should be tested andvalidated in order to assure that risks are covered.Also the role of a tester in Agile projects changessignificantly with respect to the traditional role(Crispin and Gregory, 2009; Kaner, 2004; Sum-rell, 2007; Visitacion, Rymer, and Knoll, 2009).Typically Agile projects do not have extensive re-quirements, nor a complete architecture, but bothevolve (and can change) over time. As a conse-quence, testers must be able to cope with evolv-ing, incomplete requirements, architectures, andproducts. Other challenges posed to testers arethe close team-collaboration, the increased focuson test automation and regression testing, andexploratory testing. Although the differences be-tween Agile testing and traditional testing arevarious, the test types and techniques that areapplied in Agile testing are not different fromthose applied in traditional testing (van Veenen-daal, 2010; van Veenendaal, 2009). After all, thegoal of testing is still to verify that requirementsare met.

The popularity of Agile methodologies has re-sulted in a set of valuable scientific publicationsthat mostly report about case studies and expe-riences during the introduction of Agile method-ologies. However, the extent to which the focushas been specifically given to the aspects of test-ing and quality assurance remains very limited.The only exception we are aware of is (Karls-son and Martensson, 2009), where a set of briefrecommendations is proposed to the existing testprocess. Relevant aspects that testers need toknow in Agile methodologies are also mentionedin (Astels, 2003; Beck, 2003; Crispin and Gregory,2009) and (Madeyski and Biela 2007; Madeyskiand Biela 2008). However, these publications re-mains rather unclear on the way organizationsstarting with Agile development should integratetheir testing processes.

In this paper we try to cover this gap, by iden-tifying the problem areas (Section 2), outline rec-ommendations (Section 3), and incorporate thesein a model (Section 4) that can be used by or-ganizations to integrate their testing with Scrum.We have based our observations on our experienceon a project team at KLM implementing Scrumas development methodology, and by visiting twoother companies active in the transportation sec-tor. The results have been analyzed using the

grounded theory approach (Strauss and Corbin,1994), followed by the development of a modelhow testing could be integrated with Scrum andwhat practices/activities are recommended to im-plement in future projects in order to enable asmooth integration. The developed model andrecommendations have been triangulated with ex-ternal topical expert interviews, existing litera-ture, and targeted interviews with KLM employ-ees, as briefly explained in Figure 1.

2 OBSERVATIONS

In this section we describe the case and the obser-vations made during an implementation at KLMScrum as development methodology. We havecategorized our observations into five categories:project preparation, team composition, productbacklog design, sprint preparation, and productquality.

The project under observation was about thedevelopment of a mobile web-application to beused by KLM employees. The core Scrum teamselected for this project initially consisted out of 3developers, 1 tester, a product owner and a Scrummaster. In total, 7 sprints have been implementedover a period of 3 months whereby each sprintlasted two weeks. It was the first time at KLMthat test competence (CC Test Management) wasinvolved into an Agile development process. Be-sides testers, also Business, Information Manage-ment, and Operations were aligned to the projectin order to have a true cross-functional group ofstakeholders and to be able to pilot Agile throughthe whole organization (and not just the IT de-partment).

2.1 Project Preparation

Project preparation turned out to be an essentialaspect for the team to reach an efficient way ofworking which, if not done properly, can resultin a set of impediments and waiting times. It isimportant to prepare enough elements so that ateam can efficiently start sprinting.

In the project we observed there was no ac-cess to test and acceptance environments beforethe first iteration started. The unavailability ofthese environments has caused several problemsdue to a delay of 5 weeks before resolving. First,the tests that were actually executed were ex-ecuted from a developer environment, thus de-manding time from programmers to prepare their

Figure 1: Research Methodology.

environments. Although Agile advocates collabo-ration and interaction (Cunningham, 2001), thissituation caused unnecessary delays for both thetester and the programmer. Second, developmentenvironments are not similar to test - or accep-tance environments, thus questioning the valueand quality of the tests.

While Agile promotes communication aboutteam roles and personal expectations, the projectat KLM faced initial difficulties in overcoming thetraditional viewpoint of a tester’s tasks and re-sponsibilities. In combination with the unavail-ability of test environments the added value ofthe tester was not recognized, eventually result-ing in the termination of a tester’s contract after2 iterations.

Further, incomplete and unclear system de-pendencies caused problems during the develop-ment and testing of the application. It was notknown which interactions should be tested, whattest data should be used, and what the expectedresults of the test data were supposed to be. It-eration 4 reported a change in the deploymentplatform. Although Agile methodologies are ca-pable of dealing with changing requirements (andso was the project team), platform requirementsneed some stability to avoid wasting valuabletime.

2.2 Team Composition

The project has seen the come and go of severaldifferent team members. Due to a combinationof the earlier stated issues in total, three differentpersons have been working as tester in the seveniterations of the project, leaving periods of one totwo weeks between the exit and entry of the nexttester. Furthermore, the project has employed atotal of five programmers with an average involve-ment of 2.5 programmers per iteration. Withthe absence of a tester in iteration 3, only pro-grammers and business stakeholders tested theapplication (both not possessing the specific test

skills). Furthermore, after a new tester joined theproject, the defect data-trends in iteration 4 and5 showed peaks in the number of defects found.These peaks made it difficult to complete all es-timated user stories, given that the team had notreserved time to resolve defects.

Of course, the entrance of a new memberto the team affects a team. Without previousproject knowledge new members need to orientthemselves and dig up assumptions the team al-ready has, which takes time and can hinder theflow of the team.

2.3 Product Backlog Design

Preparing the product backlog’s content forsprint planning sessions was not implemented ina proper way and caused meetings to overrunin time, created tensions in these meetings, andcontributed to the lack of overview of the busi-ness. Initially, the product backlog was not pri-oritized and user stories that were placed on theproduct backlog were too large in size. Subse-quent sprint planning meetings prioritized userstories, but they were not concrete enough forthe team to be understood. More urgent was thefact that some stories were also unclear to thebusiness stakeholders. The results were lengthysprint planning sessions, high effort estimates be-cause of uncertainty, and shifting of the story’spriorities due to the missing information.

Unclear user stories in sprint planning resultedin changing sprint backlogs during a sprint, whiledevelopment and testing had already started(which is against Agile). Occasionally, the con-tent of user stories changed which resulted in ad-ditional rework.

2.4 Sprint Preparation

Sprint preparations aim to set the scope for theupcoming iteration and to generate effort esti-mates for this scope. Although a sprint’s prepa-

ration is related to the product backlog designissues, there were more aspects that hindered theteam. Because of the peaks of defects discoveredin iteration 4 and 5 the team had to concentrateon resolving defects during sprint planning ratherthan on the user story. As a result the time spenton closing defects exceeded the time reserved andonly an average of 70% of the user stories was con-cluded. Combined with the product backlog de-sign issues, it resulted in programmers not know-ing what the business expected, and testers withdifficulties in designing and executing test casesthat reflected the business’ wishes.

To save time, those user stories of which accep-tance criteria were discussed at sprint planning,were not documented or stored in a systematicalmanner. The result was that the acceptance crite-ria become vague (or forgotten) and thus requireda re-consult of the business for re-clarification ofalready discussed issues.

2.5 Product Quality

In Agile development it is a team’s responsibil-ity to deliver increments of a ”potentially ship-pable” product that is continuously assured to beof high quality. Due to several impediments theteam we observed faced time pressures in the firsttwo iterations, which resulted in the developmentof quick and dirty solutions implying that codequality can get at risk, potentially affecting theproduct’s quality. The decision in the tradeoff be-tween resolving the problems by code refactoringand implementing new functionality was not obvi-ous. The first got the team’s preference, whereasnew functionality had the business’ preference.

The demonstration at sprint 3 revealed somedefects that were not found during the sprint.Since there was no tester present in the teamon that sprint, the overall product quality wasquestionable. Especially when the business getsto face defects that the team is unaware of, con-vincement of putting a product into productiongets harmed. Further, technical documentationappeared to be incomplete making code and tech-nical decisions harder to understand

3 RECOMMENDATIONS

Based on the observations and analysis outlinedin Section 2 we have produced a set of recom-mendations addressing the issues identified andaiming at improving the integration of testing

into Scrum-based projects. Although each rec-ommendation can be implemented on itself, thecombination of recommendations shows an inter-related set that complements each other in theareas where difficulties have been experienced.

3.1 Implement Sprint Zero

In order to deal better with preparation issues,Sprint Zero1 (also known as ’pre-iteration’, or ’it-eration 0’) can add value to the process by puttingin place enough elements for the team to achievean efficient and effective process. Although thereis ”sprint” in the name, the duration of this phaseshould not be fixed but tailored to the character-istics and demands of a project.

For teams and organizations new to Agile itis important to clearly communicate the chang-ing project roles (Sumrell, 2007) (especially thetester’s role), and to manage expectations within,but also outside, the team (Lindvall et al, 2004;West et al, 2010). It has been recognized thatalthough teams are self-organizing they require ashared mindset, values and principles (Fry andGreene, 2007). Training initiatives and organiz-ing workshops can help in attaining this goal.

Following the vision of lean development (Pop-pendieck and Poppendieck, 2003), waste in thesoftware development process should be mini-mized. Teams interact with many, and depend onsome, different organizational stakeholders andactors (Lindvall et al, 2004), that each can causedelays for the project team when the team hasfailed to contact or involve them in the project.In order to minimize the effect of organizationalstakeholders on the team and the process, possi-ble impediments they may cause should be elim-inated early on.

From a test perspective, sprint zero is an idealtime to design a suitable test strategy. Outlin-ing the overall approach towards testing focusingon the product’s characteristics and risks, butalso think of approaches regarding defect man-agement, test automation, and regression testing.Furthermore, one might need specialist testers todo non-functional testing or access to stakehold-ers to do user acceptance testing, which can bereserved/contacted from the beginning.

As preparation to the project it is a good

1Jakobsen and Johnson (Jakobsen and Johnson,2008) see Sprint Zero as a kind of CMMI (Team,2006) planning, and state that the use brings morediscipline to the project - resulting in a projection ofsuccesses in larger Agile projects

practice to arrange the test infrastructure (en-vironments and tooling) before the first iterationstarts. Not being able to thoroughly test from thebeginning hinders the true continuous and inte-grated nature testing is supposed to have in Agiledevelopment, and leads testing to lagging behindon development immediately.

Although the overall IT architecture of largeorganizations is complex, dependencies should beidentified between the system to be developed andthe existing infrastructure. If not, the result canbe long waiting times to get the specifications andaccess, but might also be implemented function-ality that is based on the wrong interactions (andthus requires rework).

During sprint zero, the Definition of Done(DoD) (Claesson, 2011) should be designed incollaboration with the project stakeholders in or-der to define when a release, a sprint, or a userstory is satisfied. Not only can the team setthe quality criteria and activities that they thinkare required for each user story, iteration, or re-lease; but other stakeholders that interact withthe project/product in later stages such as main-tenance can also contribute in, for example, casesof technical documentation.

3.2 Implement Product BacklogGrooming

Product Backlog Grooming sessions are intendedto review the product backlog before the nextsprint planning and can be used to address theproblems related to the design of the productbacklog and user stories. Furthermore, by exe-cuting it in a parallel fashion to a running sprint,upcoming sprint planning sessions are supposedto be relieved from lengthy discussions and un-ready user stories.

During the grooming sessions user stories areanalyzed regarding their size, level of detail, anddependencies. Large stories need to be sliceddown and ambiguous or incomplete stories shouldbe corrected by the Product Owner. Stories thatcontain dependencies with other stories need tobe carefully planned (e.g. preferably not in thesame sprint).

These sessions can further help the team inidentifying what extra actions should be takenbefore certain pieces of functionality can be im-plemented. Whenever new requirements havebeen designed that require additional tooling, testdata, or environments, actions can already betaken to prevent waiting time in the next sprint.

Finding out these things during a sprint planningsession may result in delays during sprints (as oc-curred in the observed project).

To reduce costs, at least the Product Owner,the Scrum Master, a tester, and one developershould be be participating in grooming the prod-uct backlog. In fact each of these participantsadd value through a different perspective, but es-pecially a tester contributes because of the criticalperspective to get user stories testable.

Given the fact that product owners and busi-ness stakeholders may have difficulties in design-ing user stories (Fry and Greene, 2007; Lindvallet al, 2004; Seffernick, 2007; West et al, 2010) theadoption of a Definition of Ready (DoR) is alsorecommended. As is the case with the DoD, theDoR aims to set quality criteria on a user storybut this time to label it as ’READY’. Previousresearch by (Jakobsen and Sutherland, 2009) hasshowed that the DoR is able to remove the wastecaused by issues related to user stories.

3.3 Use the Whole TeamApproach

Our major recommendation is to have a testerin the team and to keep the team constant forat least the duration of the project. A constantteam does not face the delays that are related topeople getting used to the team and to the projectand are thus better able to learn as a team. Fur-thermore, with a multi-functional team includinga tester, continuity of product quality is betterassured because the test efforts can be uniformlydistributed in all sprints.

Having a tester in the team contributes to theproduct backlog design issues by providing a per-spective that is focused on the customer, but alsotakes into account the technical problems thatmay occur during development. The combinationof the complementary perspectives of program-mers, product owner / stakeholders, and tester, isalso referred to as ’The Power of Three’ (Crispinand Gregory, 2009). Furthermore, testers are theones that preserve the ’helicopter view’ over aproduct and are able to assess whether the overallquality would be acceptable to the business.

Although the entire team should be responsi-ble for testing, and testing should be a task thatcan be executed by any team member (Crispinand Gregory, 2009), our experience demonstratesthat without a tester the product’s quality mightbe at risk during sprints: defect trends show animmediate increase when a tester is not present

in a team. Evans (Evans, 2009) supports the casefor the value of the specific test-skills a testerbrings in order to deliver high-quality softwarethat meets the business’ needs. Additional sup-port is provided by (Sumrell, 2007) stating thateveryone should be focused on quality and thatAgile testers are the ones infusing quality into theteam and the product throughout its lifecycle.

An often referred to model that is used forAgile testing, is ”The Agile Testing Quadrants”(Crispin and Gregory, 2009). Originally, themodel was developed by Marick (Marick, 2003) inorder to distinguish the different type of tests thateach serve different purposes. Without a testerin the team, it is unlikely that the four quadrantsare covered enough. Programmers will be pri-marily focused on the first quadrant describingtechnical tests that verify small pieces of func-tionality. But business analysts and UI designerswill mostly focus on the third quadrant describingbusiness tests focused on business scenario’s anduser interaction. This leaves two quadrants with-out enough attention, namely the one on businesstests that verify whether user story’s acceptancecriteria are met, and the other on technical teststhat verify whether the non-functional require-ments, or quality attributes, are met.

For a team to implement a good ”Whole TeamApproach” (Crispin and Gregory, 2009), it isimportant to select a tester that fits Agile de-velopment and is compatible within the team.The continuous and integrated nature of Agiletesting, the team pressure on delivering businessvalue, and the team’s responsibility for qualityare changing the role of a tester in Agile projects.A skill of testers part of a Scrum team shouldinclude knowledge about Scrum, hard skills (i.e.test [automation] skills and acceptance test drivendevelopment skills), and also good soft skills (i.e.communicative skills) because of the focus on in-dividuals, interaction, communication, and feed-back (Cunningham, 2001).

3.4 Implement Acceptance TestDriven Development

Implementing Acceptance Test Driven Develop-ment (ATDD) can help teams in developing agood input for test case design. ATDD resemblesthe practice of extracting acceptance criteria, ex-amples, scenarios, or workflows from the productowner while discussing the desired functionalityof user stories. The goal of the technique is tocollect the test basis for user stories that can be

enhanced to design test cases for these stories im-mediately. Not only does the technique result ina test basis, it also facilitates and guides the di-rections of the discussions.

From a test perspective, acceptance criteriaare important to identify given they are used tovalidate functionality. Also, having acceptancecriteria clear and documented at the beginningof an iteration reinforces a programmer’s mind-set and helps her/him in developing a properlydesigned set of unit tests through Test Driven De-velopment (TDD).

But there is more value to find in practicingATDD: it significantly speeds up the process ofdesigning test cases and also has them resemblethe ”real-world” better (Adzic, 2009). Further-more, depending on the nature of the test casesand the piece of functionality, the tests can betransformed into automated tests to be executedefficiently and requiring very little effort from atester.

3.5 Focus on Value and Risk

Having traditional testing focusing primarily onthe risk areas, Agile development stresses the im-portance of realizing business value. Althoughthere is value in covering those areas where busi-ness gets hurt most, there are also other areasthat may not be a high risk but do deliver highvalue (i.e. benefits). Testing in Agile projectsshould thus not focus only on identifying and cov-ering ’risk areas’, but should also on identifyingand covering the ’value areas’. Functionality thatdoes not deliver direct ’damage’, but can deliverdirect ’benefits’ is as important to the businessand thus should be assured of high quality.

There exist several tools to help teams indefining what is value for features. Initially, userstories were designed to model functionality thatprovides value to the business by requesting thewriter to express who the stakeholders are andwhy they want this functionality (Cohn, 2004).Adzic (Adzic, 2011) recognized that there is aneed to also have a project overview of deliver-ing value and introduced effect maps. An effectmap aims to guide a team in setting priorities tothose features that contribute most to the busi-ness goals that were set. By using this tool andcombining this with a risk-value based test ap-proach, both functionality with high risks andhigh value are covered by testing.

3.6 Start Test Automation Early

One of the first recommendations that is oftenmade to Agile teams is to start test automationearly on. Investments in the automation of testswill deliver pay back in later stages by reduc-ing the amount of time required to execute tests.Automated tests are efficient and consequent inexecution and reduce the time required for test-ing significantly. Especially regression testing canbenefit from automation given the fact that theregression test suite grows over time and willreach a stage where testing can no longer keepup with development.

However it is questionable whether test au-tomation has to be applied in every project. Forexample, for projects focussing on user interfaceon mobile devices automation is difficult. Also,when a project lifetime until maintenance is lessthan 3 months test automation could be avoided.Finally, test automation requires a significant ini-tial investment, and thus it should be consid-ered whether the Return on Investment (ROI) fortest automation is positive. Teams should con-sider the expected ROI of test automation duringSprint Zero, and question to what extent maybeonly parts of the test activities, such as preparingtest data, can be automated.

4 PROPOSED MODEL

To assist organizations (especially those new toScrum) with the integration of testing and QAactivities with Scrum and the changing role ofa tester in Agile projects, we have developed amodel that outlines the different aspects relatedto testing and QA within the Scrum framework,see Figure 2 in the appendix. The rationale be-hind the model is based on the project obser-vations, the set of recommendations outlined inSection 3, the interviews conducted, and the aca-demic literature consulted. Besides the changingrole of a tester in Agile projects, we also see achanging role for the test manager: a test man-ager will be of a supportive role during the projectwhen it comes to, for example, the arrangementof test specialists and the test infrastructure, andwill primarily be actively involved in sprint zero.

The model distinguishes between the processframework and a sub-layer of identified activities,practices, and tools that relate to steps in the pro-cess. Activities resemble the activities a team andtesters do during the development process; prac-

tices resemble the identified practices that are rec-ommended to incorporate in the Scrum process;and tools resemble means that can assist the teamin performing its work.

Starting from the initiation of a project, we gothrough an adjusted version of the general Scrumprocess framework. As can be seen from the fig-ure, Sprint Zero has been included as we see itas an essential phase for Scrum teams to be ableto implement sprints in an efficient and effectivemanner from the start. As stated before, the du-ration of Sprint Zero can be flexible as long asthe team gets prepared well enough to start thefirst sprint. Besides taking away possible imped-iments, communicating the role of testers in theproject, and arranging the required test infras-tructure, the team can develop a shared under-standing about quality by defining the overall teststrategy, designing a good definition ready and agood definition of done.

The inclusion of product backlog groomingand release planning as clear steps in the pro-cess (and as activities in sprint zero) stresses theimportance we again see in enough preparation.By eliminating waste regarding the content andcomposition of user stories in a timely manner,these activities aim to prevent the team fromunnecessary long sprint planning sessions, am-biguous user stories, unclear priorities and un-clear content. Preparing the product backlog forsprint planning sessions by timely reviewing alsotrains the business in designing their stories insuch a way that they are accepted as ’READY’by the team. Following from good product back-log grooming, the sprint planning sessions in theirturn will be more focused on what they shouldbe: finding out what it is that the business reallyneeds.

Following from a properly designed set of userstories, the implementation of ATDD provides avaluable way to extract user stories’ acceptancecriteria and examples, which together are a valu-able source of input for test case design. By driv-ing the dialogue between programmers, testers,and the business, ATDD aims to guide the teamto find out what it is that the customer reallywants (i.e. challenging the business), what his orher true acceptance criteria are, and what kind ofreal-world examples there might appear in prac-tice. Together with well designed user stories,ATDD enables a team to develop a large part oftest case design before a sprint has even started.

The above three inclusions relate to the chang-ing role we see for a tester. Being integrated with

Figure 2: The Scrum and Testing Framework.

the development team and being involved withthe project from day 0, testers can add a lot ofvalue to the team as a whole. Although the addedvalue of a tester may be clear during the executionof a sprint given the relation to traditional testing(i.e. test case design, execution of test cases, de-fect management, etc.), we especially see oppor-tunities for testers to add value to the team dur-ing sprint zero, product backlog grooming, andsprint planning sessions.

The model contains several notions of risk andvalue based testing in sprint zero, sprint planning,and product backlog grooming. We see the focuson both risks and value as an important switchcompared to the traditional risk-perspective. Inorder to create a consistent focus throughout theentire project, we state that teams should startto identify business value and priorities of desiredfunctionality in sprint zero. A tool through whichthis can be done is the effect map (Adzic, 2011),outlining the relationships between functionality,stakeholders, and business goals (i.e. the valuethe project wants to generate). Knowing thevalue that can be realized by specific pieces offunctionality, the test planning for one iterationcan map the combination of the risks and thevalue in user stories in order to set the prioritiesfor testing and in this way ensuring that the testeffort is focused on the most important aspectsfor the business.

5 CONCLUSIONS

In this paper we present an empirical studythat exposes the difficulties that were faced bya project at KLM regarding the integration oftesting with Scrum. Based on the project obser-vations, additional interviews, and existing liter-ature we have analyzed the problem areas andpropose a set of recommendations that help or-ganizations to integrate their testing into Scrum.The recommendations are maybe common senseand when seen in isolation they are not necessar-ily novel. Our contribution is in complementingthe recommendations by developing a model thatextends the common known Scrum framework inorder to map the activities, practices, and toolsthat are related to testing and quality assurance.

We have not included experimental results tovalidate our framework, because to draw gen-eral conclusions about the added value of theproposed model a large number of developmentteams in different contexts would have to be ob-served and measured. However, in order to pre-vent threats to the validity of our findings, wehave taken several measures. First, to preserveconstruct validity and internal validity, we haveconsulted multiple sources during the develop-ment of the model: literature, expert interviews,project observations, and workgroup sessions.Second, the interviews and project observationshave been transcribed, coded, and analyzed fol-

lowing the grounded theory approach (Straussand Corbin, 1994). Third, to preserve externalvalidity, two similar companies have been visited,external experts have been consulted and litera-ture was used in order to prevent any bias.

ACKNOWLEDGEMENTS

We would like to thank KLM for providing theopportunity to observe a real-life project imple-menting Scrum and all interviewees that havefreed their valuable time to help us in our study.

REFERENCES

G. Adzic. Bridging the Communication Gap: Specifi-cation by Example and Agile Acceptance Testing.Neuri Limited, 2009.

G. Adzic. Agile product management using EffectMaps. Agile Record The Magazine for Agile De-velopers and Testers, 2011.

D. Astels. Test-Driven Development: A PracticalGuide. Pearson Education Inc., 2003

J. B. Barlow et al. Overview and Guidance on AgileDevelopment in Large Organizations. Comm. ofthe Ass. for Inform. Systems, 29(1):25–44, 2011.

K. Beck. Test Driven Development: By Example.Pearson Education Inc., 2003

B. Boehm and R. Turner. Balancing Agility and Dis-cipline: A Guide for the Perplexed. Addison-Wesley Pearson Education, 2003.

R. Boehm, B. Turner. Using risk to balance agileand plan-driven methods. Computer, 36(6):57–66, 2003.

A. Claesson. Test Strategies in Agile Projects. InEuroSTAR 2011, 2011.

A. Cockburn, L. Williams Agile software develop-ment: it’s about feedback and change IEEEComputer, 36(6):39–43, 2003

M. Cohn. User Stories Applied: for Agile Soft-ware Development. Addison-Wesley Profes-sional, 2004.

L. Crispin and J. Gregory. Agile Testing: a PracticalGuide for Testers and Agile Teams. Addison-Wesley Professional, 7th ed., 2009.

W. Cunningham. Agile Manifesto, 2001.http://www.agilemanifesto.org.

D. Evans. How to Succeed in an Extreme Testing En-vironment. Cambridge University Press, 2009.

C. Fry and S. Greene. Large Scale Agile Transforma-tion in an On-Demand World. In AGILE 2007,pages 136–142. IEEE, 2007.

C.R. Jakobsen and K.A. Johnson. Mature Agile witha Twist of CMMI. In Agile Conference, 2008,pages 212–217. IEEE, 2008.

C.R. Jakobsen and J. Sutherland. Scrum and CMMI:Going from Good to Great. In Agile Conference,2009, pages 333–337. IEEE,2009.

C.J.D. Kaner. The Ongoing Revolution in SoftwareTesting. Software Test & Performance Confer-ence, Seattle, 2004.

E. Karlsson and F. Martensson. Test Processes for aScrum Team. Master’s thesis, Lund University,Sweden, 2009.

M. Lindvall, et al. Agile Software Development inLarge Organizations. IEEE Computer Society,4:26–34, December 2004.

L. Madeyski and W. Biela. Empirical Evidence Prin-ciple and Joint Engagement Practice to Intro-duce XP In Proc. of Extreme Programming andAgile Processes in Software Engineering volume4536 of Lecture Notes in Computer Science, pp.141–144, Springer, 2007.

L. Madeyski and W. Biela Capable Leader andSkilled and Motivated Team Practices to Intro-duce eXtreme Programming in Proc. of Balanc-ing Agility and Formalism in Software Develop-ment, volume 5082 of Lecture Notes in ComputerScience, pp. 96–102, Springer, 2008.

B. Marick. Exploration through Example.http://www.exampler.com/old-blog/2003/08/21/.

M. Poppendieck and T. Poppendieck. Lean SoftwareDevelopment: An Agile Toolkit. Addison-WesleyProfessional, 2003.

K. Schwaber and J. Sutherland. Scrum guide. ScrumAlliance, Seattle, 2009.

T.R. Seffernick. Enabling Agile in a large organiza-tion: our journey down the yellow brick road. InAgile Conference, 2007, pages 200–206. IEEE,2007.

A. Strauss and J. Corbin. Grounded theory method-ology: An overview. Sage Publications, 1994.

M. Sumrell. From Waterfall to Agile - How does aQA team transition? In Agile Conference, 2007,pages 291–295. IEEE, 2007.

D. Talby, A. Keren, O. Hazzan, and Y. Dubinsky.Agile Software Testing in a Large-scale Project.Software, 23(4):30–37, IEEE 2006.

C.P. Team. Cmmi for Development, Version 1.2.2006.

E. van Veenendaal. Scrum & Testing: Back to theFuture. Testing Experience, 3, 2009.

E. van Veenendaal. Scrum & Testing: Assessing therisks. Agile Record, 3, 2010.

VersionOne. State of Agile Survey 2011 - The Stateof Agile Development. 2012.

M. Visitacion, J. R. Rymer, and A. Knoll. A FewGood (Agile) Testers. Forrester Research Inc.,2009.

D. West and et al. Overcoming the Primary Chal-lenges Of Agile Adoption. Forrester ResearchInc., 2010.