![Page 1: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/1.jpg)
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Budapest University of Technology and EconomicsFault Tolerant Systems Research Group
Development testing
Zoltan Micskei
1
Software and Systems Verification (VIMIMA01)
![Page 2: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/2.jpg)
Main topics of the course
Overview (1)
o V&V techniques, Critical systems
Static techniques (2)
o Verifying specifications
o Verifying source code
Dynamic techniques: Testing (7)
o Developer testing, Test design techniques
o Testing process and levels, Test generation, Automation
System-level verification (3)
o Verifying architecture, Dependability analysis
o Runtime verification2
![Page 3: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/3.jpg)
Example: what/how/where to test?
3
![Page 4: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/4.jpg)
Example: what/how/where to test?
4
Tests through GUI(~ system test)
![Page 5: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/5.jpg)
Example: what/how/where to test?
5
Tests through API(~ integration test)
![Page 6: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/6.jpg)
Example: what/how/where to test?
6
Module/unit tests
![Page 7: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/7.jpg)
UNIT TESTING
7
![Page 8: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/8.jpg)
Learning outcomes
Explain characteristics of good unit tests (K2)
Write unit tests using a unit testing framework (K3)
8
![Page 9: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/9.jpg)
Module / unit testing
Module / unit:
o Logically separable part
oWell-defined interface
o Can be: method / class / package / component…
Call hierarchy (ideal case):A
A1 A2
A31
A311
A3
A32 A33
A312 A313
A3 A31 A311 A312A
…
9
![Page 10: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/10.jpg)
Why do we need unit testing?
Goal: Detect and fix defects during development (lowest level)o Can integrated later tested modules
o Developer of the unit can fix the defect fastest
Units can be tested separatelyo Manage complexity
o Locate defects more easily, fix is cheaper
o Gives confidence for performing changes
Characteristics if unit testso Checks a well-defined functionality
o Defines a “contract” for the unit
o Can be used an example
o (Not neceserraly automatic)
10
![Page 11: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/11.jpg)
Unit test frameworks
Run unit tests frequently
o During development (e.g. refactoring)
Need to be fast
Good tool support
o Frameworks (JUnit, xUnit, TestNG, …)
o Support in IDE (Eclipse, VS, ...)
Usually simple functionality
o Define test sequences and checks
o Run tests
o Display results (red-green)
11
![Page 12: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/12.jpg)
Example: simple JUnit test
public class ListTest{
List list; // SUT
@Before public void setUp(){
list = new List();
}
@Test public void add_EmptyList_Success(){
list.Add(1);
assertEquals(1, list.getSize());
}
}
Preparing test
Calling SUT
Checking12
![Page 13: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/13.jpg)
Annotation Description
@Test Defining a test method
@Before Executed before each test, e.g. setting environment
@After Executed after each test
@BeforeClassExecuted only once before all tests(caution, tests will be not independent!)
@AfterClass Executed after all tests, e.g. global cleanup
@Ignore Do not execute given test
@Test(expected=IllegalArgumentException.class)
Test passes if code throws this Exception
@Test(timeout=100) Limits execution time of test
JUnit annotations (excerpt)
Others: creating Suite, Category, Parametrized…
13
![Page 14: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/14.jpg)
Good unit tests (guidelines!)
Simple, reliable
o No complex logic (e.g. loops, try/catch)
One test does one thing
o (not necessarily one assert statement)
This is also quality code
o No duplicated code
o Easy to understand and maintain
Tests are independent
Checks are not overspecified
…14
![Page 15: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/15.jpg)
Coding conventions
Name of test class: [Unit_neve]Test
Name of test method:
o Method_StateUnderTest_ExpectedBehavior
o MethodName_DoesWhat_WhenTheseConditions
o [feature being tested]
Structure of test:// Arrange
// Act
// Assert
// Given
// When
// Then
15
![Page 16: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/16.jpg)
Guidelines from “xUnit Test Patterns”
Source: Gerard Meszaros, http://xunitpatterns.com/
16
![Page 17: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/17.jpg)
Test-Driven Development (TDD)
17
![Page 18: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/18.jpg)
UNIT ISOLATION
18
![Page 19: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/19.jpg)
Learning outcomes
Explain why and when isolation of dependencies is recommended in unit tests (K2)
Use an isolation framework to write isolated unit tests (K3)
19
![Page 20: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/20.jpg)
Isolation in unit tests
Tests unit separately in isolation
Needed: test drivers and test doubles
20
A
A1 A2
A31
A311
A3
A32 A33
A312 A313
Test driver
Unit under test
Test double
Test double
Test double
![Page 21: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/21.jpg)
Problem: Handling dependencies
What is a dependency?
Anything that
o is collaborating with the SUT,
o but does not belong to (could not control it)
Examples:
o Other modules
o File system, network call
o Using current date/time
o …
21
![Page 22: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/22.jpg)
Example: Hard to test codepublic class PriceService{
private DataAccess da = new DataAccess();
public int getPrice(String product)
throws ProductNotFoundException {
Integer p = this.da.getProdPrice(product);
if (p == null)
throw new ProductNotFoundException();
return p;
}
}
Hard to isolate,how to specify a
test version?
22
![Page 23: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/23.jpg)
Example: Making the SUT testable
public class PriceService{private IDataAccess da;
public PriceService(IDataAccess da){this.da = da;
}
public int getPrice(String product)throws ProductNotFoundException {
Integer p = this.da.getProdPrice(product);if (p == null)
throw new ProductNotFoundException();return p;
}}
Passing implementation in constructor
23
![Page 24: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/24.jpg)
Example: Unit tests for the SUTpublic class PriceServiceTest{
@Before public void init(){DataAccessStub das = new DataAccessStub();das.add("A100", 50);ps = new PriceService(das);
}
@Test public void SuccessfulPriceQuery(){int p = ps.getPrice("A100");assertEquals(50, p);
}
@Test(expected = ProductNotFoundException.class)public void NotExistingProduct(){
ps.getPrice("notExists");}
}
Using a stub in the test
24
![Page 25: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/25.jpg)
Stub, Mock, Dummy, Fake… objects
Many techniques for substituting dependencies
o Different names (see xUnit Patterns)
o Common name: Test double
Stub
o Checks the state of the SUT
Mock
o Check the interactions of the SUT
Dummy
o Not used object (filler)
Fake
o Working, but not the real one
26
![Page 26: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/26.jpg)
Isolation frameworks
Tedious to create mocks and stubs
o Lots of repeated, plumbing code
o Time-consuming to maintain
Framework functionality
o Input: interface or class description
o Output: generate stub/mock dynamically
Example:
o JMock, Mockito, Rhino Mocks, Typemock…
27
![Page 27: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/27.jpg)
Example: Using mocks (Mockito)
public class PriceServiceTest{
@Before public void init(){
DataAccess mockDA = mock(DataAccess.class);
ps = new PriceService(mockDA);
}
@Test public void SuccessfulPriceQuery(){
// Arrange
when(mockDA.getProdPrice("A100")).thenReturn(50);
// Act
int p = ps.getPrice("A100");
// Assert
verify(mockDA, times(1)).getProdPrice("A100")
}
…
}
Create a compatible test double
Specify rules
Check interactions
28
![Page 28: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/28.jpg)
SUMMARY
29
![Page 29: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/29.jpg)
Is it worth to create such unit tests?
Calculations on an example project:
Source: Roy Osherove, The Art of Unit Testing, 2009.
30
![Page 30: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/30.jpg)
Data on a safety-critical projects
31
fs – functional specification des – design ut des – unit test design int – integration test
fs rev – fs review des rev – review ut run – ut execution sys – system test
Source: The Economics of Unit Testing, ESE 11: 5–31, 2006
![Page 31: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/31.jpg)
“Is TDD Dead?”
32
What is a unit?
Test-induced design damage.
Self-testing code
Is separate QA needed?
Costs and benefits of unit testing
if I screw up this line of code is a test going to fail?
See: http://martinfowler.com/articles/is-tdd-dead/
![Page 32: Zoltan Micskei - inf.mit.bme.hu³ria/education/... · System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2. Example: what/how/where](https://reader031.vdocument.in/reader031/viewer/2022041218/5e07d9df143d363e0101a88e/html5/thumbnails/32.jpg)
Further information
Martin Fowler: Mocks Aren't Stubs, 2007 o URL: http://martinfowler.com/articles/mocksArentStubs.html
Martin Fowler: UnitTest, 2014o http://martinfowler.com/bliki/UnitTest.html
Roy Osherove: The Art of Unit Testing: With Examples in .Net.o Manning Publications; 1st edition (June 3, 2009)
o URL: http://artofunittesting.com/
- Brett L. Schuchert: Mockito.LoginServiceExample- URL: http://schuchert.wikispaces.com/Mockito.LoginServiceExample
33