![Page 1: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/1.jpg)
Testing
MSO 08/09, WP
![Page 2: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/2.jpg)
Why testing ?
Obvious.
Still, how many of you actually test your assignment before submitting them?
How do you know if you have tested enough??
How much testing really cost?
Seemingly a lot, since there are plenty of software testing companies in NL; they seem to make good business!
2
![Page 3: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/3.jpg)
Some numbers
US financial sector suffers 1.5 billion USD (2002) due to inadequate testing (US Nat. Inst. of Standards and Tech.)
Testing cost 30 – 70% of total development cost! (various studies)
Effort to find a bug (Ellims et all 2004, JTest)
at system testing 12 – 20 hrs (600 Eur per bug!)
at unit testing 0 – 3 hrs (90 Eur per bug)
3
![Page 4: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/4.jpg)
Why is it so hard ?
Size
At the system level you can be dealing with millions LOCs!
OO adds a multitude of other complications.
Concurrency
4
![Page 5: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/5.jpg)
Plan
Unit test, integration test, system test OO issues, concurrency issues Testing with JUnit Adequacy of your tests Assignment-7 Specification-based testing Regression testing
5
Dennis et al only discuss testing very globally
![Page 6: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/6.jpg)
Taxonomy of testing
According to the "scope" : Unit testing Integration testing System testing
According to ... "other things" : Functional testing GUI testing Acceptance testing Performance testing Regression testing
6
![Page 7: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/7.jpg)
Unit Testing
Testing each "program unit" in isolation.
Manageable size; finding bugs are easier at this level.
Widely supported by xUnit tools.
Automated tools e.g. T2.
We'll take a look at Junit.
7
![Page 8: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/8.jpg)
Unit Testing
What is "unit" ? Options : method class package
Typically people take method as "unit"
But you should be aware that in OO methods can influence each other through their side effects on objects' states.
This implies that there are methods that should not be tested in isolation.
![Page 9: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/9.jpg)
OO issues complicates testing
Intra-class side effect makes it difficult to test each method in isolation:
9
class Subscription { int price ; int n = 12 ;
public Subscription(p) { price = p }
public int pricePerMonth() { return price/n } public cancel() { price=0; n=0 }}
![Page 10: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/10.jpg)
OO issues
We fix Subscription and find no more error. So, now we're sure that every client can safely call "pricePerMonth".
Watch out!! Inheritance adds another complication
10
class client {
m(Subscription s) { s.cancel() ; return s.pricePerMonth() }
class EvilSubscription extends Subscription {
public int pricePerMonth() { return 0/0 } ...
This implies that whenever you override a method, you need to test if you still respect the spec of the original method (Lipkov substitution).
![Page 11: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/11.jpg)
Integration test
Test if a set of "units" (that logically belong together) functions expectedly.
But what is a "set of units" ? Class or package
Class level testing. Given a class C, check that
Lots of work to do by hand; use automated tool (T2).
11
Various sequences of calls to C's methods and updates to C's exposed fields do not violate C's class invariant and the specs of each method.
![Page 12: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/12.jpg)
Integration test
Package level testing.
12
Subscription
price...
Customer
SubsctiptionPackage
classinv price 0Service
Potential problem: if there are fields in Subscription which are open to package-level access, but their values are constrained by a class invariant.
Implying you have to test that every method in the package repects the class invariant of every class in the package.
![Page 13: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/13.jpg)
Concurrency Concurrent threads multiply the number of possible
scenarios to consider:
This implies you need lots of tests to cover all scenarios expensive!!
But even that is not possible without special tools: We can't control JVM to do a specific scheduling. This also implies that you can't replay an error in a
concurrent setup!
13
thread 1 : { price++ ; price++ }thread 2 : { price = 0 }
Will this satisfy : 0 price 2 ?
And multi-cores are coming! (shudder)
![Page 14: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/14.jpg)
The famous V model of testing
14
Image is taken from Wikipedia.
![Page 15: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/15.jpg)
V model of testing
15
This approach assumes a water-fall SDLC model.
SystemRequirement
SystemSpecifications
Package/classlevel
specifications
Implementation unit testing
integration testing
system testing
acceptancetesting
But more generally phases can run parallel.
![Page 16: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/16.jpg)
System test System testing tests if a software as a whole functions as
expected.
Problem: at this level a software often has to be interacted through its (complex) GUI.
Testing through GUI is HARD to write and hard to automate!!
Alternative: split system testing to Functional system testing Separate GUI testing
16
![Page 17: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/17.jpg)
Functional System Testing
You need a well defined interface (set of methods) through which we can directly access a software functionality (without going through the GUI).
We will test through this interface.
Ideally this interface is prepared all the way from your design phase.
17
Test Interface
GUI
Business Logic
![Page 18: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/18.jpg)
Test Plan
For a large application you (obviously) need a test plan.
Because it involves lots of activities (and huge investment).
IEEE 829, test plan is a document:
1. describing scope, approach, resources, and schedules of testing activities.
2. identifying test items, features to test, testing tasks, who will do the tasks, risks, and contingency plans.
18
![Page 19: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/19.jpg)
Test Plan
IEEE 829
c) Approach d) Test items e) Features to be tested; features not to be tested g) Pass/fail criteria h) Suspension and resumption criteria i) Test deliverables
Oh well, a test plan essentially: Defines the scope of the testing Explains how we do the testing Estimates the resources needed
19
![Page 20: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/20.jpg)
(Unit) Testing with Junit
20
C Ctest
Junit
report
Java
public class Subscription {
private int price ; private int length ; ... public double pricePerMonth() { double r = (double) price / (double) length ; return r ; }}
![Page 21: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/21.jpg)
Testing with Junit
21
C Ctest
Junit
report
Java
import org.junit.* ;import static org.junit.Assert.* ;
public class SubscriptionTest {
@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }
@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }}
Read the Tutorial, linked from Opdracht-7.
![Page 22: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/22.jpg)
JUnit with Eclipse
22
![Page 23: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/23.jpg)
JUnit with Eclipse
23
![Page 24: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/24.jpg)
Have we tested enough ?? In principle, no way to know that.
Still, we can use "coverage" as indication:
The precentage of "parts" of your code that are executed during your tests.
Keep in mind: 100% coverage does not imply absence of bugs!!
But low coverage is obviously not good.
24
![Page 25: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/25.jpg)
Various concepts of "coverage"
Very abstract: Class coverage (have we tested all classes?) Method coverage (have we tested all methods?)
Code level: Line coverage Branch coverage Path coverage
25
![Page 26: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/26.jpg)
Line, branch, and path coverage
Line coverage: percentage of lines executed by your tests. E.g. a test (1x) with
s.period = 0 s.price = 0
will give you 100% line coverage. Very weak. But in practice is used most
26
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
![Page 27: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/27.jpg)
Line, branch, and path coverage
Branch coverage: percentage of branches (of "if", "while" etc) executed by your tests.
E.g. the tests (2x)
s1.price=0 s1.period=0s2.price=2000 s2.period=2000
gives 100% branch coverage.27
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
s.period > 12
s.price > 1000
But you still miss the "red" case!!
![Page 28: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/28.jpg)
Line, branch, and path coverage
Path coverage: percentage of paths in the "control flow graph" that your tests pass through.
There are 4 paths. You will need 4 tests to cover them all. Much stronger. Complication with loop & recursion take e.g. "prime
paths".
28
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
s.period > 12
s.price > 1000
![Page 29: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/29.jpg)
Measuring Coverge in Eclipse
29
Read the Tutorial, linked from Opdracht-7.
![Page 30: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/30.jpg)
Assignment 7 : Foo Subscription System
30
![Page 31: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/31.jpg)
Foo architecture in more detail
31
![Page 32: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/32.jpg)
Foo architecture in more detail
32
![Page 33: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/33.jpg)
Discount Abstract class DiscountStrategy
description() calcDiscount(customer, activeDiscountStrategies)
Two concrete implementations: Discount_5top Discount_10pack
TASK 1: test these two classes (unit testing) Verify against the specifications (of the above two methods) 100% (emma) coverage!
33
![Page 34: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/34.jpg)
Problem with traditional tests
We need concrete data as inputs and oracles need manual effort.
This is very fragile; if we change the functionality a bit, you'll have to re-calculate them.
34
@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }
@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }
![Page 35: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/35.jpg)
Alternative: specification-based testing We'll write specifications in-code (in Java). Alternative: use JML or OCL. You'll need additional tools. Your need to train your programmers in them.
TASK 2 : (optional, 1.5 pt)
Write specifications for ApplicationLogic, and do specification-based testing.
We'll limit to the methods addCustomer and removeCustomer of ApplicationLogic.
35
![Page 36: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/36.jpg)
Architecture
36
ApplicationLogic
ApplicationLogic_SPEC
addCustomerremoveCustomerclassinv
ApplicationLogic_SPEC_test
app int addCustomer(String name, String email) {
... // pre-condition
int result = app.addCustomer(name,email) ;
.... // post-condition
return result
}
![Page 37: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/37.jpg)
Example of in-code specification : spec-1
37
Suppose this is the spec of addCustomer:
"Name and email should not be null nor empty (pre-cond). If name does not occur in the system, then add the customer, return +1. Else don't add, return -1."
int addCustomer(String name, String email) {
assertTrue(name != null && email != null && ! name.equals("") && ... )
boolean occur = false ; for (String n : getCustNames()) if (n.equals(name)) { occur = true ; break }
int result = app.addCustomer(name,email) ;
if (occur) assertTrue(result == 1) else assertTrue(result == -1) ;
return result }
![Page 38: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/38.jpg)
Example of in-code specification : spec-2
38
ApplicationLogic
ApplicationLogic_SPEC
addCustomerremoveCustomerclassinv
app
Suppose we want to impose this class invariant on the application logic:
"The value of the variable NextCustId should be non-negative."
boolean classinv() {
assertTrue(app.nextCustID 0) ;
return true ;
}
![Page 39: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/39.jpg)
Examples of the tests
39
import org.junit.*;import static org.junit.Assert.*;
public class ApplicationLogic_test {
@Test public void test1() { ApplicationLogic.reset() ApplicationLogic_SPEC applogic = new ApplicationLogic_SPEC()
assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) }}
![Page 40: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/40.jpg)
Advatanges of spec-based testing
No (fragile) oracle! But we still have to manually write the test sequences and inputs. Can be
automated! e.g. with T2 You must combine this with pair-programming. Disadvantage:
writing specs takes time Incure maintenace cost.
40
public void test1() { ... assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) }
![Page 41: Testing MSO 08/09, WP. Why testing ? Obvious. Still, how many of you actually test your assignment before submitting them? How do you know if you](https://reader035.vdocument.in/reader035/viewer/2022062719/56649ef35503460f94c06328/html5/thumbnails/41.jpg)
Regression test
The re-testing of a software that has been modified. Is a serious problem for large software may take days! Between versions, the changes are usually very limited (to
some modules). However, the impact may be global due to interactions
between modules. This forces you to test the entire software. That is, you need
a test set that will deliver full coverage over your entire software.
But if we just re-execute the whole set of tests collected so far, this will take too long!!
Challenge: how do we identify a good minimum set?
41