t-76.5613 software testingand qualityassurance
TRANSCRIPT
Automated testing in Agile SW development
Seppo Sahi
SoberIT
2.10.2006
T-76.5613 Software Testing and Quality Assurance
2
Introduction
� Agile methods have strong emphasis on practices that support good quality in the constructive work
� Pair programming, war room, on-site customer etc.
� Much of the testing activities are actually constructive work
� Test-driven development
� Less testing with destructive attide
� Agile methods have a very strong focus on quality issues
� Testing in agile environment
� Automated unit testing
� Automated acceptance testing
� Exploratory testing
3
No manual tests
� Chapter about manual tests in ”Testing Extreme Programming” (Crispin & House 2003)
� ”No manual tests.”
� Quick and minimal effort regression testing
� Keep up a good pace in software development
� Enables safe refactoring
� Suitable testing tools not always available
� Organization has to develop own tools
Agile acceptance testing with FIT
Acceptance testing
= A level of testing conducted from the viewpoint of the customer, used to establish the criteria for acceptance of a
system.
5
Requirements and acceptance testing
� Agile requirements definition
� Specifying requirement in great detail is delayed until the details are really needed
� Agile acceptance testing
� In XP Testers, customer and developers design acceptance test together – main responsibility in customer
� Acceptance tests are designed before implementation
� All acceptance tests are gradually automated
� Developers use tests as executable specifications
Acceptance testing is not a phase in the end –it is a tool to carry out a project
6
Roles
� Turning customer requirements into an automated test is a team effort
� Customer - knows what is needed
� Design and write tests together with testers and programmers
� Tester - knows how to test software and where software systems usually break
� Write more advanced tests (e.g. negative tests, use testing techniques)
� Programmers are not interested
� Customers don’t have the required skills
� Consider testing viewpoint early
� Programmer – connects tests and application
7
What is FIT and FitNesse?
� FIT (Framework for Integrated Test)
� Framework for Acceptance testing
� FitNesse
� A Wiki based UI to create, edit and run FIT tests
� Non-programmers write and run tests
� Business people, customer etc.
� A little help from testers and programmers is needed
� Standard and convenient way to write tests, log results, group tests into suites
� GUI and command-line based execution tools
� Available for many programming languages
8
Creating and modifying acceptance tests is easy
� Use Word, Excel or HTML editor
9
How acceptance tests plug-in to application (1/2)
� Simple ”glue code” a.k.a Fixture code is needed
� Framework offers a variety of ready fixtures from which programmers can inherit their own ones
� Handles parsing of test data etc.
� Separation of test data and test code
Presentation test fixtures
Application under testFixture
Test data Fixture code Application code
10
How acceptance tests plug-in to application (2/2)
Business rules
Business rule API
Client-side presentation, validation, calcualtion
Presentation API
UI
Presentation test fixtures
Business ruletest fixtures
Fixture code
Application code
11
Demo
� Writing and organizing tests
� Running tests
� Writing fixture code
12
References
� FIT homepage - http://fit.c2.com/� Download framework and tools
� Example code
� Articles about agile acceptance testing� Crispin, Lisa. 2005, ”Using Customer Tests to Drive Development”,
Methods & Tools, Summer 2005. http://www.methodsandtools.com/PDF/mt200502.pdf
� Shore, James. 2005, ”A Vision for Fit”. http://www.jamesshore.com/Blog/A-Vision-For-Fit.html
� Marick, Brian. 2004, “Driving Projects with Examples: a Handbook for Agile Teams”. http://www.exampler.com/book/
� Reppert, Tracy. 2004, ”Don’t Just Brake Software. Make Software.”, Better Software, July/August 2004. www.industriallogic.com/papers/storytest.pdf
14
Unit testing
� ”A unit is the smallest possible testable software component”
� Various interpretations exists
� Procedure / function
� Class / object / method
� Small-sized component
� Goal is to ensure that software units are functioning as intended
� E.g. a function returns correct value when exercised with valid parameters
� E.g. function behaves correctly when given an invalid input
� Automated unit test == a piece of code that exercises another piece of code
15
Unit testing - a vital testing level
� Writing software is difficult and every programmer makes mistakes� There will be defects in all software you write
� The cost of a defect is lowest when it is found in unit testing� Defects that slip to upper levels (integration, system or acceptance testing)
� More difficult to spot� Time consuming to debug
� Trivial bugs on unit level can be hard to resolve on system level� Uses organizational resources
� Writing, prioritizing and managing test and defect reports� Bug found during the use in production
� Loss of money, loss of...
� Unit testing lays the foundation for good quality software� Higher levels of testing are needed to ensure overall quality of software
� Unit testing lays foundation for effective testing on higher levels� Bad unit level quality slows down integration and system testing
16
How do you know that your code works correctly?
� Do you just trust your luck?
� Do you use “printf” outputs to see what happens in your code?
� Do you use debugger to find out how your code really works?
� Do you ask your friend/colleague to check through your code?
� Do you create small (temporary) widget to access the piece of code directly runtime?
� Do you use some ugly, cumbersome, time consuming practice that leaves trash into your code and is thrown up to the user’s face when you least expect it…
“Getting here is not possible, something is really
messed up in this code!!!”
“Fatal error 9981: Assertion failed : av_d_dx > 0”
17
Unit testing frameworks
� Convenient frameworks for writing unit tests as part of the programming task � In the same environment with the same language
� Tools of xUnit family are the most famous ones� jUnit
� CppUnit
� NUnit
� …
� The unit testing frameworks provide a convenient way of writing and executing unit tests� Standard and convenient way to write tests, log results, group tests into suites
� GUI and command-line based execution tools
� Integrated to IDEs
� Convenient report generators
� Xml, html
18
Demo
� Framework features
� IDE integration
19
Naming conventions for jUnit tests
1. import junit.framework.TestCase;
2.
3. public class TriangleTest extends TestCase
4. {
5. ...
6.
7. /**
8. * Tests that {@link Triangle#classify()} can classify an
9. * impossible triangle.
10. */
11. public void testClassify_withImpossibleTriangle() {
12. Triangle impossibleTriangle = new Triangle(8, 3, 4);
13.
14. String classification = impossibleTriangle.classify();
15. String expectedClassification = "impossible";
16.
17. assertEquals("One side longer than two others together" +18. "should be impossible.", 19. expectedClassification,
20. classification);
21. }
22.
23. ...
24. }
Naming conventionstell what class and methodis under test
20
Naming conventions for jUnit tests (2/2)
Triangle.java
classify()
isIsosceles()
...()
TriangleTest.java
testClassify()
testIsIsosceles_withEquilaterialTriangle()
test...()
testIsIsosceles_withNonIsoscelesTriangle()
test...()
CODE: TEST FOR THE CODE:
testIsIsosceles()
� Test method name
� Begins with ’test’...
� ...followed by the name of the method it tests...
� ...and side conditions of the test
21
Basic test flow
1. import junit.framework.TestCase;
2.
3. public class TriangleTest extends TestCase
4. {
5. ...
6.
7. /**
8. * Tests that {@link Triangle#classify()} can classify an
9. * impossible triangle.
10. */
11. public void testClassify_withImpossibleTriangle() {
12. Triangle impossibleTriangle = new Triangle(8, 3, 4);
13.
14. String classification = impossibleTriangle.classify();
15. String expectedClassification = "impossible";
16.
17. assertEquals("One side longer than two others together" +18. "should be impossible.", 19. expectedClassification,
20. classification);
21. }
22.
23. ...
24. }
Create test data (12)
Perform operation and fetch result (14)
Specify expected re-sult of the operation (15)
Check that result matches the expected (17-20)
Clean up
22
Simple Test class
1. import junit.framework.TestCase;
2.
3. public class VectorTest extends TestCase {
4. private Vector testVector = null;
5.
6. protected void setUp() {
7. testVector = new Vector();
8. }
9.
10. protected void tearDown() {
11. testVector = null;
12. }
13.
14. public void testIsEmpty() {
15. assertTrue("Vector is not empty", testVector.isEmpty());
16. }
17.
18. public void testAddElement() {
19. testVector.add(new Object());
20. assertEquals("Wrong size!", 1, testVector.size());
21. }
22.
23. public static void main(String[] args){
24. junit.textui.TestRunner.run(VectorTest.class);
25. }
26. }
23
Assert method summary 1/2
� assertEquals(String msg, Object expected, Object actual)
� Compares two values for equality. The test passes if the values are equal.
� Comparison methods for all primitive data types, too (int, float, long, byte, char ..)
� assertTrue(String msg, boolean b)
� Evaluates a boolean expression. The test passes if the expression is true.
� assertFalse(String msg, boolean b)
� Evaluates a boolean expression.The test passes if the expression is false.
� assertNull(String msg, Object a)
� Compares an object reference to null. The test passes if the reference is null
� assertNotNull(String msg, Object o)
� Compares an object reference to null. The test passes if the reference is not null.
24
Assert method summary 2/2
� assertSame(String msg, Object a, Object b)
� Compares two object references using the == operator. The test passes if both refer to the same object.
� assertNotSame(String msg, Object a, Object b)
� Compares two object references using the == operator. The test passes if both refer to different objects.
� fail(String msg)
� Causes the current test to fail
All asserts take a string as the first parameter� Provides documentation
� Shows up in the JUnit TestRunner interface
� Good messages can tell you what is wrong without starting up the debugger
Check possible assertXXX() methods from JUnit JavaDocs!
25
Unit testing best practices (1/3)
� Write automated unit tests rather than use “printf” or debugger
� Automated unit tests retain value over time
� Tomorrow, after five years, ...
� No need for human interaction
� Write once – run with minimal effort
� Test code should contain as little logic as possible
� Introducing more logic will introduce more errors
� Don’t use loops, branches etc.
� Unit tests should be independent
� Don’t assume a specific execution order
� Don’t assume specific execution time
� Tests have no side effects
26
Unit testing best practices (2/3)
� Write tests like they were small increments
� Each test tests very small and targeted subject
� Tests build on each other - utilize existing tests to create new ones
� Keep test methods short
� Kent Beck: Long tests indicate the likelihood of a design problem
� Refactor your code so that it can be tested in smaller parts
� JUnit is designed to stop on the first failing assert on purpose!
� Use method naming conventions
� Test name should describe functionality tested and side conditions
� E.g. testAddUserWithAlreadyExistingUser
� Document your unit tests
� Use method naming conventions
� Write JavaDocs for test methods
� Include a message in assertXXX()
27
Unit testing best practices (3/3)
� Treat test code like production code
� Remember readibility and clarity
� Remember coding conventions
� Store data in ways that can be managed by the source control system
� Test data evolves along with test cases
� If it can't break on its own, it's too simple to break � otherwise, write unit tests
� testing setX() and getX() usually ignored
� Run all your unit tests as often as possible, ideally every time the code is changed.
� For larger systems, you may just run specific test suites that are relevant to the code you're working on
� Keep all tests running
� Tests have to be maintained like application code
Agile unit testing and test-driven development
29
Document driven vs. agile unit testing
� Principle 7. ”Testing should be carried out by a group that is independentof the development group”
� Course book presents a unit testing viewpoint which appreciates this principle
� Unit testing should be separate task conducted after the development is done
� Unit test design is based on written specifications
� Modern ideas of unit testing represent slightly different viewpoint
� Unit tests are written by developer
� Automation is essential part of unit testing
� Unit tests might even be written before the actual program code
� This lecture is mostly about agile unit testing
� Read the course book Ch. 6 and understand the difference
� The same tools are used by both disciplines
30
Agile unit testing
� Test-driven development is practiced
� Writing and running tests is integral part of daily development rhythm
� Developers’ QA and programming tool
� All unit tests are automated
� Automated tests are run as often as possible
� Gather all your tests to a test suite and use it for regression testing
� Integration, nightly, milestone
� During development; after every change to the code; new code or bugfix
31
Test-driven development
� Writing test before code to be tested
� ”a little test, a little code, a little test, a little code, ...”
� Tests are added gradually during implementation – not in large lumpafterwards
� Process of writing tests drives low-level design and programming
� Tests specify what code should do
� Tests validate that code does what it should
� Actually, a design and coding practice
� One of the core practices of Extreme Programming
� Developers have been applying TDD for several decades
32
TDD episode (1/2)
� Proceeds step by step
1. Write a test.
2. Design and implement just enough to make the test pass.
3. Repeat.
� Testing and coding alternate in very smallsteps
� Duration of one cycle should be a few minutes
� Small steps – difficult to make mistake
Write test
Write code
Run tests
Fail
Pass
33
TDD episode (2/2)
� Episode is over when you can’t write a failing test anymore
� Write test for each requirement of the code
� Write test for each point that can possibly break
� One cycle at a time
� Don’t write a bunch of tests at once
� Refactor if you ever see the chance to make the design simpler
� Run all tests after finishing episode
� Make sure you did not brake anything else
34
Demo: CommandLineParameters class
� Class that represents command line parameters given
� Format should be /<param> <value>, e.g. /backup true /interval 300
35
Test-Driven Development – claimed benefits (1/2)
� Close feedback loop
� TDD cycle is very short – know if code is working right after you programmed it
� Task-orientation
� Encourage programmer to decompose problem into manageable programming tasks
� Helps to maintain focus
� Helps to measure progress and scope work
� Low-level design
� Programmer is forced to think which classes and methods to create, how they are used, how to name them, what arguments does a method take, what does a method return
36
Test-Driven Development – claimed benefits (2/2)
� Results better code
� If the test is too hard to write, the code being tested is too complicated
� Results testable code
� Programmer can’t end up with code that cannot be tested
� Effect on quality
� Testing becomes part of the development process and gets done
� Side effect of TDD is that code gets thoroughly unit tested
37
Try it!
� The only way to know!
� Personal experiences
� Good feeling about the code written
� General confidence that your code does what you have intended it to do
� Good feeling when checking your code into version control with all green
� Tests really get written when they are written beforehand
� You allways have an up-to-date regression testing suite
� TDD helps you to keep focus on the current task
� Program only what is needed to see the green light
� Promote best practices
� System.out.println is used for displaying messages for user – not for developer
� Debugger is used for debugging
38
Example of testbase growth in a project that used TDD
0
2000
4000
6000
8000
10000
12000
Pre
-pro
ject
Itera
tion
1
Itera
tion
2
Itera
tion
3
Itera
tion
4
Itera
tion
5
Itera
tion
6
Mai
nten
ance
Application Testcode Comments Comments in testcode SQL code
� 4 developers, iteration length 2 weeks