Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

TestingTechnion – Institute of Technology236700

Author: Gal Lalouche ©

1

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Why (unit) test?

• Programming is incremental by nature• We want to verify we haven’t broken anything

• Tests not only examine the code’s functionality but also its• Ease of use• API• Boilerplate & setup• Error path behavior• Coherency and Coupling issues

• Tests are the usage example of our code and are an excellent source for documentation

2

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testable Code → Good Code

• Tests are usually the first time outside code will use your production code• Is the production code hard to use?• Is object creation long and complex?• Are methods cryptic with lots of parameters?• How does your production code handle usage errors?• Have you been surprised by a method’s behavior?

• Is the production code hard to test?• Do you write 10-15 LOC before you reach the assert line?• Is it difficult to assert the desired behavior?• Do you find it hard isolating the tested feature?• Are you not even sure what you should be asserting? 3

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Test types

• Unit Tests – Test a single element of the system• Integration Tests – Verify that a few units can work together

and are wired up correctly• End-to-End Tests – Test the complete flow of interactions

throughout the system• No “mocks”/”stubs”/”fakes”

• Acceptance / Business logic Tests – Answers the question:“Did we build what we wanted?”• Usually centered around a user story

Our focus today

4

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Test types

5

http://blog.jonasbandi.net/2010/09/acceptance-vs-integration-tests.html

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing Framework – Wish list

• Better Asserts: Validating results Vs. expectations• assert(sqrt(4) == 2) Vs. assertEqual(2,sqrt(4))• assert(x != null) Vs. assertNotNull(x)

• Clear indication of Success / Failure• Meaningful message upon failure

• Ease of use – Should be easy to add new Tests• Aggregate tests together into Test Cases / Tests Suites• Continue with other tests even if a test fails / throws an

exception• No duplication of initialization code of tested objects• Side effects should not influence other tests 6

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit

• A unit testing framework for Java• http://www.junit.org

• A Test is a Method • A Test Case is a class which includes tests methods• Aggregates tests for a single tested unit

• A Test Suite is a collection of Test Cases / Test Suites (or both)• Used to create a hierarchy of tests

7

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit 4.0• A step up from JUnit 3.0 • New Assertions methods in org.junit.Assert.*• No need to extend from junit.framework.TestCase• No need to prefix test methods with ‘test’

• Test methods are annotated with @Test

• @BeforeClass – runs once before first test• Setting up the stubs objects / DB Connections

• @AfterClass – runs once after the last test• Closing DB connections

• @Before method – runs before every test• Preparing data• Could be used for advanced construction

• @After method – runs after every test• Resetting stubs / cleaning data

8

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit test case life-cycle

9

Test Case Class load Static C’tor @BeforeClass*

C’tor

@Before*

@Test

@After*

C’tor

@Before*

@Test

@After*

C’tor

@Before*

@Test

@After*

C’tor

@Before*

@Test

@After*

@AfterClass*

@A* - Running all methods annotated

with A

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit life-cycle - Example• Local resources should be initialized and cleaned before/after

every test

10

public class DbTests {

File output; @Before public void createOutputFile() {

output = new File(...);

output.createNewFile();

} @Test public void testSomething() { ... } @Test public void testSomethingElse() { ... } @After public void deleteOutputFile() {

output.delete();

}

}

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit life-cycle - Example• Expensive and global resources can be initialized and

destroyed before/after all tests

11

public class DbTests {

static DatabaseConnection database;

@BeforeClass public static void login() {

database = new DatabaseConnection(); database.establishConnection();

} @Test public void testSomething() { ... } @Test public void testSomethingElse() { ... } @AfterClass public static void logout() {

database.logout();

}

}

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit 4.0 - Example• Assuming we want to test a new Stack interface

implementation

• Smart Asserts:• assertNotEqual, assertTrue, assertSame,assertNotNull, assertArrayEqual, etc.

12

Name is testing scenario

ResultExpectation

org.junit.ComparisonFailure: expected:<[LAST]-ITEM> but was:<[FIRST]-ITEM>

Initialization

import org.junit.Assert;

import org.junit.Test;

public class StackTestCase {

@Test public void peekReturnsTheLastPushedItem() {

Stack<String> s = new ArrayListStack<>();

s.push("FIRST-ITEM");

s.push("LAST-ITEM");

Assert.assertEquals("LAST-ITEM", s.peek());

} }

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Ideal Test • A recipe for a good test:

• If (1) and (2) are needed at all (e.g., initialization in constructor/@Before), it will be as short as possible (ideally just calls to new)

• (3) will be a single method invocation• (4) will be a single assert

• Of course this ideal cannot always be achieved• But if you can’t, you should always ask yourselves why?

13

@Testpublic void testShouldNameExplicitlyStatesExpectation() { (1) – Initialize collaborators (optional) (2) – Initialize object under test (optional) (3) – Exercise behavior (4) – Assert outputs }

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Test names – bad• It should be easy pin-pointing the problem when a test fails

without looking at its code

• Consider the following test:

• How about addItemWithOutOfStockProduct?• Describes the method and the parameter• Still not a good name – does not describes the desired behavior

14

@Testpublic void addItemTest() {  Product p = new Product();  p.setAvailable(false);  ShoppingCart cart = new ShoppingCart();  cart.addItem(p);  assertEquals(0, p.numItems());}

Bad name.Specifies the name of the

invoked method but not the desired behavior

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Test names – good• It should be trivial understanding what is the expected

behavior from the name alone• We do not need to mention the name of the invoked method

• Good rule of thumb: the test name should include an explicit expectation• Use verbs such as should, does, returns, throws, etc. 15

@Testpublic void doNotCountOutOfStockItem() {  Product p = new Product();  p.setAvailable(false);  ShoppingCart cart = new ShoppingCart();  cart.addItem(p);  assertEquals(0, p.numItems());}

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Exceptions• A test that throws an exception will fail.• So how do we make sure a code throws an exception when it should?

• We would like to verify 3 things:• The right exception has been thrown• The right line has thrown the exception (sometimes)• The exception contains the right data (rarely)

16

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Exceptions• The hard way – Using the try/catch pattern

• Verifies our 3 wishes• Easy to forget the fail resulting in a test that never fails• Quite long and verbose• Test code is just as important as our “regular” code! We want to minimize

code duplication and boilerplate.

@Testpublic void badEncodeArgument() { Encoder encoder = new Encoder(“UTF-8”); try { encoder.encode(null); fail(“Exception should have been thrown”); } catch (NullPointerException e) { assertEquals(“Text should not be null”, e.getMessage()); } }

17

Based on http://alexruiz.developerblogs.com/?p=1530

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Exceptions• The easy way (JUnit 4.0) – expected value in @Test annotation

• Can’t check the exception message• Can’t verify which line has thrown the exception• Very short

• Can be “good enough” in most cases

@Test(expected = IllegalArgumentException.class)public void badEncodeArgument() { Encoder encoder = new Encoder(“UTF-8”); encoder.encode(null);}

18

Based on http://alexruiz.developerblogs.com/?p=1530

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Exceptions• The smart way (JUnit 4.7) – Using @Rule annotation

and ExpectedException

• Verifies our 3 wishes• Still quite short

• JUnit Rules offers a few additional candies

@Rule public ExpectedException thrown = ExceptedException.none();

@Testpublic void badEncodeArgument() { Encoder encoder = new Encoder(“UTF-8”); thrown.expect(NullPointerException.class); thrown.expectMessage(“Text should not be null”); encoder.encode(null);}

19

Based on http://alexruiz.developerblogs.com/?p=1530

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit - Timeout• We want to verify that our code finishes in the allotted time.

• This is criminally verbose

@Testpublic void testTimeout() {

final boolean[] done = new boolean[] { false };new Timer().schedule(new TimerTask() {

@Override public void run() {foo();done[0] = true;

}}, 0);Thread.sleep(10);if (done[0] == false)

fail(“Failed to run in the allotted time");}

20

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit - Timeout• In JUnit 4.0 – Use the timeout annotation value

• Test will fail after 10 milliseconds.

• Global timeouts using @Rule (JUnit 4.7)

@Test(timeout = 10)public void testTimeout() { // Tested code}

21

@Rule public MethodRule globalTimeout = new Timeout(10);

@Testpublic void test1() { // Tested code}

@Testpublic void test2() { // Tested code}

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Ignoring tests• You can ignore tests using the @Ignore annotation• Test will not run, but will still be visible in the test reports

• Commented out tests will usually be forgotten about• Won’t be deleted and possibly will not be fixed

• Ignored tests are easily visible in the output window

• Make sure you disable tests temporarily• Bad tests should either be deleted or fixed• Ask yourself – why are you ignoring the test in the first place?

22

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit - Running• Textual runner• Used mainly in command line environments (Scripts, Maven, Ant, CI)

• IDE Support• To run several test cases

org.junit.runner.JUnitCore.runClasses(TestCase1.class, ...);

@RunWith(Suite.class)@Suite.SuiteClasses({ TestCase1.class, TestCase2.class, subpackage.AllTests.class, …})public class AllTests {} 23

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

JUnit – Source code Organization

• Option 1: Test class in same folder as subject• A single hierarchy of folders• Locality

• Option 2: Keep tests in the mirrored directory structure of source location• A duplicated hierarchy of folders• Easy to find all tests• Easy to separate source code from its tests• Easier bytecode separation

24

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Leveraging OOP• Tests are classes too!• Can use inheritance to our advantage• Use Abstract Test classes against abstract classes / interfaces

25

A

B C

ATest

BTest CTest

Classes Tests Class

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Leveraging OOP

26

public abstract class Stack<T> { public void push(T element); public int size();}

public class ArrayStack<T> extends Stack<T> { // … implementation code}

public class LinkedListStackStack<T> extends Stack<T> { // … implementation code}

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Leveraging OOPpublic abstract class StackTest { private final Stack<Object> stack;  protected StackTest(Stack<Object> stack) { this.stack = stack; }  @Test public void sizeIsOneAfterOnePush() { stack.push(new Object()); Assert.assertEquals(1, stack.size()); }}

27

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Leveraging OOP

• Extend abstract TC to initialize the concrete test subjects

28

public class LinkedListStackTest extends StackTest {  public LinkedListStackTest() { super(new LinkedListStack<Object>()); } // specific LinkedListStackTest tests}

public class ArrayStackTest extends StackTest {  public ArrayStackTest() { super(new ArrayStack<Object>()); }

// specific ArrayStack tests }

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Order (or lack thereof)• Do not assume order between tests• There’s no order guaranties in TestCases• There is one in TestSuites

• Avoid side effects in tests

29

public class StackTest {  private static Stack<Object> s = new Stack<>();

@Test public void pushFirstAndCheckSize() { s.push(new Object()); assertEquals(1,s.size()); }

@Test public void pushSecondAndCheckSize() { s.push(new Object()); assertEquals(2,s.size()); }}

Bad !

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Resource Management• Do not load data from hard-coded locations on a file system

• Use Class.getResource() or Class.getResourceAsStream()

• Will search the file relatively to the Test Case .class file• Maven will automatically create a src & test resource folders

@Beforepublic void setUp () { InputStream inp = FileInputStream("C:\\TestData\\dataSet1.dat"); …}

@Beforepublic void setUp () { InputStream inp = getClass().getResourceAsStream("dataSet1.dat"); …}

30

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Unit Testing: What to Test• It is not practical to test everything:• State space > Not always practical• Loops in the code State space is infinite / Halting Problem• End to End and Acceptance tests will cover the state space quite well

• Cover as much of the code as possible, but be smart about it• Extensiveness of the test should be proportional to the complexity of the

implementation of the tested method• If m2() calls m1(), no need to duplicate m1() tests for m2()• But, for each variation in the flow (usually an if()) you should have a

corresponding test that exercises it• Always test public methods

• Should you test package-private and protected methods as well?• Private methods usage is important only in the context public methods

• If you find yourself wanting to test a private method, you should rethink your design, (e.g., extract the private method to a helper class)

31

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Unit Testing: Quality• Cover code usage• Check common usage errors / edge cases:

• nulls• empty strings• zeros• empty lists• single-item lists• uninitialized objects• bad configurations

• Exception handling

• Code Coverage• Function, Statement, Branch, Decision coverage• Monitor using external Code Coverage tools (e.g. EclEmma)

32

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Code Coverage - EclEmma

33

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Code Coverage• Code coverage is correlated with good testing, but does not

equate it• You can achieve very high coverage with bad tests (e.g., testing

without asserts)• You can write great tests with low coverage

• 100% coverage may look “pretty”, but should not be a goal in and off itself• You should always suspect tests that were written solely for

achieving high coverage• Do you really need to test for NullPointerException for each

of your parameters?• You can use code coverage to Spartanize your code• e.g., branches that are never taken may be dead code 34

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Do’s & Don’ts• Mutability makes code harder to test• Tests also have to take state into account

• Always place shortest, simplest tests first in a TestCase or in a TestSuite• You should always make sure the simplest cases work before you

carry on to the complicated ones• Prefer lots of short tests instead of a few long ones• Every test should test a single behavior• Speed up discovery-to-fix cycle

• Testing code should be simpler than the subject code• Complicated tests usually implies complicated production code usage• Tests are the code usage examples• Do not put if statements in your tests. Loops are quite rare.

35

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Do’s & Don’ts

• Always make sure you start Green before you start coding• This way, you know if you broke something• Unless working in TDD…

• In which case, start Green before refactoring!

• Regression testing • Make sure new additions don’t break existing code

• Run tests as often as possible• The earlier you detect the bug the sooner you solve it

and the easier it becomes• Therefore, test runs should be short, at most 10ms per

test• Never commit code that doesn’t pass all tests!

36

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Do’s & Don’ts• When you work on a bug, write a test case first• Your work is done when the test case succeeds• If you write the test after the bug is fixed, you cannot know if

the test succeeded in reproducing the bug• Helps with regression testing, i.e., ensuring the bug won’t

reappear later• You can actually debug using unit tests!

1. Write a test that fails2. If you still aren’t sure what the bug is, write a smaller unit

test that fails3. Repeat step 2 as necessary• This could be preferable to running a debugger, as the tests

remain when done

37

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Testing – Do’s & Don’ts

• When you change the subject class, add tests

• Testing is meant to improve the quality of the code• Do not write bad code to pass tests !!

• Although possible in TDD…• Do not write bad tests to pass tests !! (Make sure a test can fail)

• Hard to test? Re-think the design!

38

Auth

or: G

al L

alou

che

- Tec

hnio

n 20

15 ©

Test types – misc. (not taught this semester)• Behavior-driven development• Using Domain Specific Language (DSL) to structure tests around

stories and features• Property tests• Large randomized input to check invariants and properties of

output• Mutability testing• Changing (mutating) the source code or tests slightly, to see if our

tests fail• This ensures tight coupling between tests and source code

• Design by Contract• Asserting pre and post-conditions on class’s methods• Fails in runtime, possible to check even at compile time

39