an effective agile testing framework

14
An Effective Agile Testing Framework Anko Tijman

Upload: jess

Post on 10-Jan-2016

46 views

Category:

Documents


1 download

DESCRIPTION

An Effective Agile Testing Framework. Anko Tijman. AN AGILE TESTING FRAMEWORK. 1. 2. 3. 4. 5. ACCEPTANCE TEST DRIVEN DEVELOPMENT. ATDD. Add test cases to requirements User stories Team Product owner + designer + developers + end user To ‘mould’ the thinking about requirements - PowerPoint PPT Presentation

TRANSCRIPT

PowerPoint Template 2011

An Effective Agile Testing FrameworkAnko Tijman

Hi, Im Anko Tijman. Welcome to the first Eurostar Virtual Conference. My session is about creating an Effective Agile Testing Framework. With a framework I mean: a set of practices that when used together provide complete coverage of the testing domain.Its about how testing can be organised effectively within the iteration and how your agile team members from can benefit from testing.

1

AN AGILE TESTING FRAMEWORK

12534Audio: Framework_slide2-pt1.wav

This is the central image of our vision. The building blocks are 5 seperate practices within the testing domain. Each of these practices can and should be tailored to a specific environment. But: you should have all 5 of them, like the three legs of a triathlon. Using this framework, you can rest assured that your team has covered all of the testing duties

Lets start with the vertical axis, it has 3 components: Moulding, Checking and Testing. These 3 components show 3 different sides of the testing practices. The Moulding is about having a joint understanding on the right system. For the productivity of the team it is very important that everyone knows what is important for the customer. And for the customer it is very important that he knows what system is going to be developed. This can be accomplished through testing practices. That is why Moulding is inmportant.The second element is Checking: validating test cases about the regular behaviour of the system. It is about test driven development and automated testing. Its about executing automated test cases. The 3rd element is Testing, and I mean the real testing of unexpected behaviour. As you know, software and users both are very inpredictable. So only executing automated checks , which is veryfying expected system behaviour, wont do! Testing is about making sure that the system cant be broken, is stable under heavy circumstances.

I will discuss the 5 practices Acceptance Test Driven Development, Unit and Integration Testing, Non Functional Testing, Exploratory Testing and Continuous Acceptance now in detail.

2ACCEPTANCE TEST DRIVEN DEVELOPMENTAdd test cases to requirementsUser storiesTeamProduct owner + designer + developers + end user To mould the thinking about requirementsStart of sprint

ATDD3The first of the 5 practices central to the framework is Acceptance Test Driven Development. Lets shorten that to ATDD In ATDD, test cases are added to the requirements. In most agile projects these requirements will be used in the form of user stories. You do ATDD together as a team, so with the product owner, the designer, the developers and an end user. As a team, this is an real good exercise to mould your thoughts. You perform ATDD at the start of the sprint.

This process description is from Elisabeth Hendrickson. First, as a team you discuss the story. The Product Owner explains why the feature is important and what business process it supports. Then the team can ask questions about the acceptance criteria: when is the feature done? The results will be Distilled in a table format, which is very powerful, and so that you can use FitNesse. Then it will be Developed and Demo-ed. That doens;t have the be the formal Scrum Demo, ofcourse small implementations of the story can be demoed to the team earlier. After all, agile is about creating short feedback loops. After demoing the working software, a new discussion might rise, then then the whole process will restart, hopefully for a smaller bit of the implementation of the user story.

ATDD is a very powerful technique to mould your thoughts, as a team, about the understanding of the system and value for the customer. Just like Stephen Covey learns: you begin with the end in mind!

ACCEPTANCE TEST DRIVEN DEVELOPMENTwww.testobsessed.comATDD4The first of the 5 practices central to the framework is Acceptance Test Driven Development. Lets shorten that to ATDD In ATDD, test cases are added to the requirements. In most agile projects these requirements will be used in the form of user stories. You do ATDD together as a team, so with the product owner, the designer, the developers and an end user. As a team, this is an real good exercise to mould your thoughts. You perform ATDD at the start of the sprint.

This process description is from Elisabeth Hendrickson. First, as a team you discuss the story. The Product Owner explains why the feature is important and what business process it supports. Then the team can ask questions about the acceptance criteria: when is the feature done? The results will be Distilled in a table format, which is very powerful, and so that you can use FitNesse. Then it will be Developed and Demo-ed. That doens;t have the be the formal Scrum Demo, ofcourse small implementations of the story can be demoed to the team earlier. After all, agile is about creating short feedback loops. After demoing the working software, a new discussion might rise, then then the whole process will restart, hopefully for a smaller bit of the implementation of the user story.

ATDD is a very powerful technique to mould your thoughts, as a team, about the understanding of the system and value for the customer. Just like Stephen Covey learns: you begin with the end in mind!

ATDD APPROACHES

Gojko AdzicCollaborative method for specifying requirements and testsBehaviour Driven DevelopmentDan North (dannorth.net)GWT-format:GIVEN WHEN THEN ELSE Specification by examplespecificationbyexample.com There are different approaches that support ATDD. There is Behaviour Driven Development by Dan North. He suggests to use the so called GWT-format: Given When Then.Acceptance criteria can be expressed in this format: GIVEN a situation, WHEN something happens, THEN I get a result. This clearly limits the scope of a user story, and that is very important. Because without limitation, we as tester would be able to analyse hundreds of test cases for a single user story and that is counterproductive for The Whole Team.What I learned using this BDD format is, that it might be useful to describe an ELSE statement as well. Because then the team knows how the system needs to behave when the criterium isnt met. It helps your team to think in system behaviour. Another approach for ATDD is Specification by Example by Gojko Adzic. This is a Collaborative method for specifying requirements and tests. This is a 7 pattern method to use examples as the driver for specifications. This really enrichens the discussions! You can check out his website for more information and his book on this subject.5UNIT AND INTEGRATION TESTING(Regression) test automationHelps refactoringDevelopers and testers work togetherDiscuss requirementsDeveloper writes some unit testsDiscuss unit test casesCode will pass tests Decide with your team who executes the Integration testsAny user input can be validated at UT/IT level Huge performance improvement for the team!6Unit and Integration testing is the 2nd practice of the framework and its essential in Agile Testing. It creates an automated regression framework for the team, so that refactoring can be done. Unit testing is done by the developers in a unittest framework, and as a tester you can really help them. Not many developers know test techniques like boundary value analysis and decision tables. But: these techniques are very useful at these lower levels! Collaboration is the key here: discuss the requirement with the developer, have the developer code some unit tests and discuss these test cases. It really helps the developer to think in terms of end states and it helps you as a tester to gain more technical knowledge on the system. Youll end up with a system that will pass the tests and that is exactly what you need. Because then the basic, expected technical and functional behavior of the system is Checked automatically and very, very frequently!For Integration Testing, you can decide with your team how to execute the tests. Some teams prefer to have this done by the tester using FitNesse, and then the developers write the fixtures. Other teams have the developers do all the Integration Testing so that it is within their IDE so they have short feedback loop at IT level as well. Youll have to decide that as a team, based on effectiveness for and competence of the team.Remember, nearly any user input can be validated at Unit test and Integration level. So that means that in System test, you only have to focus on bugs that rise at that level. This is a huge performance improver for teams! The more you test at UT and IT level, the more effective your test strategy is at higher levels of testing!UNIT AND INTEGRATION TESTINGUnit testing, regression testing and automated testing are not the sameDiscuss what to (not) automate as a teamDo more than automated checking!7UTIT 2.wavNow, what important is to realize as a team, is that there is a difference between unit testing, regression testing and automated testing .In agile projects, unit testing is always automated. Because the unittest framework is integrated in the developers IDE and they can run it as frequent as they want, it creates a short feedback loop for them. That framework creates a safety net for any changes in the code. That is good and that is how unit testing should function. But unit testing is not the same as regression testing. Regression tests are there to confirm that the functionality that wasnt supposed to change, didnt change. Not all regression tests can or should be automated, for instance when automating a test case or maintaining the fixtures is very cumbersome. It is very wise to discuss as a team what you want or dont want to automate. So, you might have some manual regression tests to execute every sprint.And of course not all automated tests are unit tests or regression tests. For instance, you might use automation tools for performance testing and some of the test cases you execute might not make it to the regression test set. So make sure that your team does more than automated checking and just running a lot of automated unit tests!NON FUNCTIONAL TESTINGFeedback on the whole systemPotentially (!) shippable productQuality attributesFeedback within the iterationIf not possible: Pareto principlesimple test techniques that manage the risks involvedSecurity: OWASP checklist (www.owasp.org) Usability testingHeuristic Evaluation (Jakob Nielsen)Add this to your Sprint Definition of Done8C:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 6The third practice of the Agile Testing Framework is Non Functional testing. As well as the other practices, it is preferred to have Non-F-T performed within the iteration. This because it creates a short feedback loop on the whole system. Remember, a Scrum team strives for developing a potentially shippable product within the sprint. So that includes proper testing on the quality attributes as well!Now some Non functional tests are quite comprehensive, and then performing all test activities necessary to create the complete feedback loop is quite a challenge. What you can do then is applying the Pareto principle to this test level: create an 80% feedback loop based on a 20% effort. For instance, its quite a challenge to have a specialist security tester within your team all the time. But if you would have this security tester to perform a workshop in your team and then use the OWASP checklist when hes mot there, then youll probably avoid the most common mistakes made in security. That also means that the security tester will find critical, rare bugs. The same principle can be used for Usability Testing as well! Youll probably dont have a usability lab available every sprint, or even for the whole release, but by applying the Heuristic Evaluation principles by Jakob Nielsen, your software will be much more user friendly, but then without the huge investments of hiring a usability lab. (dat laatste is niet opgenomen)You can add these practices to your Sprint Definition of Done. That way it is clear what you test activities are performed within the iteration and what still remains to be tested before going into production. Be aware, if you, your team or the PM take the short cut and skip N-F-T within the sprint, then you might end up with a lot of unwanted and too much feedback on the system as a whole just before releasing. And that is not Agile!QUICK REFERENCE CARDSSharing information based on the Pareto principleCondensed information about a topicThe essential parts of informationAny topic worth sharing Share the QRCs!9.wav: Framework_slide6_NFT_QRC.wavC:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 6 QRCJust to give you an idea what the Pareto principle can do for your team: we use so called Quick Reference Cards to share within and outside our team.These QRCs as we call them- contain very condensed information about one specific topic each. Heres an example of a QRC about Exploratory Testing. What we do is not only writing it down as brief as you can, but we narrow it down to the essential parts of the information. So we skip out the detailed exceptions that most of the time- include 80% of the information around a subject -- while theyre only relevant in less than 20% of the time. Applying principle supports effective knowledge transfer!You can use the QRCs for any topic that is relevant for your team or even your organisation. Whether its about applying Non-Functional techniques, or ISTQB test techniques, or the use of Fitnesse and the creation of fixtures As long as you want to share information quickly and effectively with more people, then QRCs are very useful. The whole idea is: Keep it simple, focus on the core elements that are relevant in 80% of the time, and share that in just 20% of the information!If elaboration on a topic is needed, please refer to that. That might be a book, a website or your teams wiki page. And ofcourse: share these QRCs in your organisation!EXPLORATORY TESTINGChecking TestingChecks look for expected bugsFind unexpected (!) bugsTest cases that need human intelligenceDifficult to automateChartersSharing and gaining knowledgeRisk basedFocus10Framework_Exploratory.wavC:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 7 ExploratoryWhen you have an automated Unit and Integration test framework for regression and refactoring, and have some automated tests for N-F-T, than you have CHECKED a lot. But you havent really TESTED the system yet. Checks are regulated, analysed test cases that verify whether the system works As Expected. But bugs are nasty things, and they have the talent to hide them selves deeply in the code. And they certainly wont limit themselves to areas where you expect them to be when youre analyzing test cases! So theres a great need for Exploratory testing, even if you have extensive Unit and Integration tests.ET is about the real testing: finding bugs at unexpected paces, execute test cases that really need human intelligence or that are difficult to automate. To keep yourself effective in ET and preventing yourself fro wandering around in the system, you can use charters to limit the scope and time of your ET sessionET has another great advantage: if you pair with team mates on specific areas, then you gain knowledge from their experience and you can share knowledge on how to test effectively. So theres a win-win situation. ET sessions should be risk based: youll probably automate the most critical test cases for the system. But heres also system behaviour that cant be left untested. That is where manual ET fits in. Make sure you focus on specific goals that need to be tested! CONTINUOUS ACCEPTANCEAgile: customer is nearAcceptance sessions in sync with demoUser representativesIT-SupportAutomated scripts & manual scenarioEmotional acceptance11C:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 8 ContAccThe 5th and final practice of an Effective Agile Testing Framework is Continuous Acceptance. In traditional projects, the Acceptance test is performed at the end of the project. Its a decision for the business to decide if they can work with the system. Too often, the system is not accepted and I believe that this mainly comes from the fact that the customer isnt able to give feedback during the development phase. In agile, iterative projects the customer is always near. Weve had good experiences with adding an Acceptance Test moment to the demo just before or right after it. End users and IT-support were invited to these sessions. The end users tested the system based on real life scenarios and IT-support checked whether they could support the system-as-is. The scripts we used were automated scripts, developed right from the ATDD outcomes, and manual scripts as well. In these sessions, we have encouraged users to give us feedback to improve the system. Weve added these items to the Product Backlog and discussed these with the Product Owner. Second, apart from the AT session around the demo, we have had more formal AT sessions with larger groups of end users. Wer organised these every 3 or 4 sprints. In these sessions, real live data was entered so that the users could really experience working with the new system. And again we have processed a lot of their comments to improve the system. Through this process, we have established what we called emotional acceptance the users could get used to the system and we were able to tune the system to the actual needs from the business.AN AGILE TESTING FRAMEWORKC:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 9 BigPictNow, lets go back to where we started. I hope youve seen that all 5 practices provide complete coverage of the testing domain. From the Acceptance Test Driven Development level with requirements, down to the tiny bits of code at Unit and Integration Test level, the feedback on the quality attributes in Non Functional testing, covering the unexpected bugs through Exploratory testing and finally making sure that the system is easily accepted through the Continuous Acceptance Practice: all testing activities are covered in this Effective Agile Testing Framework. It all comes down to moulding your thoughts as a team, checking a lot through test automation and testing exploratory and comprehensively. As mentioned earlier, performing these activities within the iteration will become an important feedback loop for the whole team! Having this framework will create valuable information at all levels. As youll probably realize by now, your role as a tester within the team really changes with all this. Let us take a closer look at that.12YOUR ROLE CHANGESC:\Users\Schmolle\Desktop\My Dropbox\EurostarAudio\Produced\Slide 10 RoleThe tester will become the spindle of the team when it comes to testing activities. Youll no longer be the guy or girl that does all the execution of tests at the end of the sprint but youll become the mentor and director for all testing activities within the team. Youll coordinate more and make sure that the whole team is focusing on delivering the software according to the Definition of Done. After all, your focus is on delvering a potentially shippable product, so the quality standards are high!Youll discuss more on what needs to be tested and how, with people who know less about testing than you do. But you will learn from them about why it should or shouldnt be tested.Youll coach more and share your knowledge about testing with designers, developers, end users and IT-support people. Let them become just as good at testing as you are!And youll probably execute and analyze less than youre currently used to, because youre doing all these other activities. Be aware and make sure that for some time in the week you are still the tester that analyses and execute tests. Becausethat is where your heart is and that is also where youre delivering value for your team. So keep connected with testing and lead your team!1314Im available on chat for Q&A. Come and join in!www.ordina.nl/blogswww.ordina.nl/agiletesten @agiletesternlpaper.li/AgileOrdinaA final note: implementing this framework is not easy. What I mainly did in this session, was sketching an outline of practices that you will have to work out for your team. It requires thinking skills to tailor the practices to the needs of your team. Take it easy, step by step! Second, implementing a new way of working in your team requires people skills! Listening, explaining and leadership is at the heart of any change in the way of working. So please be patient and persevering!I encourage you to pick up the challenge and improve the testing in your agile project. Testing creates feedback, and feedback is very, very addictive. So let your team become real test addicts!If you have questions or remarks, please join us right away in the chat session!Thank you for listening and I hope to see you at Eurostar!14