Quality Software with Unit Quality Software with Unit TestTest

Alice YANG

Some definitionsSome definitions

• Unit testing – testing a single class to ensure Unit testing – testing a single class to ensure that it performs according to its API specs that it performs according to its API specs (i.e., its Javadoc)(i.e., its Javadoc)

• Integration testing – testing that the units Integration testing – testing that the units interact appropriatelyinteract appropriately

• Functional testing – testing that the integrated Functional testing – testing that the integrated units meet the system requirementsunits meet the system requirements

• Regression testing – testing that changes to Regression testing – testing that changes to code have not (re-)introduced unexpected code have not (re-)introduced unexpected changes in performance, inputs, or outputschanges in performance, inputs, or outputs

• The Brass Ring: We generally want our The Brass Ring: We generally want our code to be as free of bugs as is code to be as free of bugs as is economically feasibleeconomically feasible

• Testing is the only way to know how bug-Testing is the only way to know how bug-free your code isfree your code is

• All four kinds of testing mentioned can be All four kinds of testing mentioned can be automated with repeatable suites of testsautomated with repeatable suites of tests

• The better your test suite, the more The better your test suite, the more confidence you can have in your code’s confidence you can have in your code’s correctnesscorrectness

• Junit is the most commonly used way to Junit is the most commonly used way to automate unit tests for Java codeautomate unit tests for Java code– Suites of repeatable tests are commonly built Suites of repeatable tests are commonly built

up over timeup over time– QA involves running these suites of tests on a QA involves running these suites of tests on a

regular basisregular basis

• Junit can also be used to do integration, Junit can also be used to do integration, functional, and regression testingfunctional, and regression testing– Integration tests theoretically should create Integration tests theoretically should create

two objects, and test their interactionstwo objects, and test their interactions– Functional tests can simulate the user Functional tests can simulate the user

interacting with the system and verifying its interacting with the system and verifying its outcomesoutcomes

– Regression testing is typically making sure Regression testing is typically making sure that changes do not introduce test failures in that changes do not introduce test failures in the growing suite of automated teststhe growing suite of automated tests

Mock ObjectsMock Objects

• Are you isolating your objects under test?Are you isolating your objects under test?– If a test uses two objects and the objects interact, If a test uses two objects and the objects interact,

a test failure can be attributed to either of the two a test failure can be attributed to either of the two objects, or because they were not meant to objects, or because they were not meant to interactinteract

– Mock objects are a common solutionMock objects are a common solution• One and only one real code object is tested – the other One and only one real code object is tested – the other

objects are “mock objects” which simulate the real objects are “mock objects” which simulate the real objects for test purposesobjects for test purposes

• Allows test writer to simulate conditions that might be Allows test writer to simulate conditions that might be otherwise difficult to createotherwise difficult to create

– This problem is well-known and amply addressed This problem is well-known and amply addressed by several products (e.g., EasyMock)by several products (e.g., EasyMock)

Code CoverageCode Coverage

• Do you have enough tests? What’s tested Do you have enough tests? What’s tested and what isn’t?and what isn’t?– Well-known problem with numerous tools to help, Well-known problem with numerous tools to help,

such as Emma, Jcoverage, Cobertura, and such as Emma, Jcoverage, Cobertura, and Clover. These tools monitor which pieces of code Clover. These tools monitor which pieces of code under test get executed during the test suite. under test get executed during the test suite.

– All the code that executed during the test is All the code that executed during the test is considered covered, and the other code is considered covered, and the other code is considered uncovered.considered uncovered.

– This provides a numeric measurement of test This provides a numeric measurement of test coverage (e.g., “Package x has 49% class coverage (e.g., “Package x has 49% class coverage”)coverage”)

JUnit Fallacy # 1JUnit Fallacy # 1

• ““The code is just fine – all our tests pass”The code is just fine – all our tests pass”• Test success does not mean the code is Test success does not mean the code is

finefine• Consider the following test:Consider the following test:

public void testMethod() {public void testMethod() {;// Do absolutely ;// Do absolutely

nothingnothing}}

This test will pass every time.This test will pass every time.

What the world What the world reallyreally needs needs

• Some way of measuring how rigorous Some way of measuring how rigorous each test iseach test is– A test that makes more assertions about the A test that makes more assertions about the

behaviour of the class under test is behaviour of the class under test is presumably more rigorous than one that presumably more rigorous than one that makes fewer assertionsmakes fewer assertions

– If only we had some sort of measure of how If only we had some sort of measure of how many assertions are made per something-or-many assertions are made per something-or-otherother

“Assertion Density”

• Assertion Density for a test is defined by the equation shown, where– A is the assertion density– a is the number of assertions made

during the execution of the test– m is the number of method calls

made during the execution of the test

• Yep, I just made this up

Junit Fallacy #2Junit Fallacy #2

• ““Our code is thoroughly tested – Cobertura Our code is thoroughly tested – Cobertura says we have 95% code coverage”says we have 95% code coverage”

• Covered is not the same as testedCovered is not the same as tested

• Many modules call other modules which Many modules call other modules which call other modules. call other modules.

Indirect TestingIndirect Testing

• Class A, Class B, and Class C all execute as Test A runs

• Code coverage tools will register Class A, Class B, and class C as all covered, even though there was no test specifically written for Class B or Class C

Test A

Class A Class B Class CCallsCalls

Tests

What the world What the world reallyreally needs needs

• Some way of measuring how directly a Some way of measuring how directly a class is testedclass is tested– A class that is tested directly and explicitly by A class that is tested directly and explicitly by

a test designed for that class is better-tested a test designed for that class is better-tested than one that only gets run when some other than one that only gets run when some other class is testedclass is tested

– If only we had some sort of “test directness” If only we had some sort of “test directness” measure…measure…

– Perhaps a reduced quality rating the more Perhaps a reduced quality rating the more indirectly a class is tested?indirectly a class is tested?

Testedness

• Testedness is defined by the formula shown, where– t is the testedness– d is the test distance– nd is the number of

calls at test distance d

• Yep, I made this one up too