functional testing - northeastern university

FUNCTIONAL TESTING

F. Tip and M. Weintraub

Thanks go to Andreas Zeller for

allowing incorporation of his

materials

A TESTER WHO IS REALLY INTO THE JOB

HOW TO TELL IF A SYSTEM MEETS EXPECTATIONS?

Two options:

1. Testing: execute parts of the program and observe if unexpected behaviors occur

2. Formal Verification: exhaustively enumerate all states of the system, and try to prove that properties to be verified hold in each state.

THE FIRST COMPUTER BUG (1947)

From https://www.facebook.com/navalhistory/photos/a.77106563343.78834.76845133343/10153057920928344/

𝑖𝑛𝑝𝑢𝑡 𝑖,𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑒𝑛𝑣 𝑗

WHAT TO TEST?

Configurations

𝑖𝑛𝑝𝑢𝑡 𝑦,𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑒𝑛𝑣 𝑧


DIJKSTRA’S CURSE

Configurations


Program testing can be used to

show the presence of bugs, but

never to show their absence!

Edsger Dijkstra

FORMAL VERIFICATION

Configurations

Le

vel o

f A

bstr

action

Design or Specification Level

High Level Framework

Code Level

Assembly

OS

Hardware

FORMAL VERIFICATION

Configurations

Le

vel o

f A

bstr

action

Design or Specification Level

High Level Framework

Code Level

Assembly

OS

Hardware

Verification can only find the

absence of errors, but never their

presence


SO HOW CAN WE COVER AS MUCH BEHAVIOR AS POSSIBLE?

Configurations


FUNCTIONAL TESTING – AKA BLACK BOX TESTING

WHITE BOX TESTING IS WHERE YOU TEST BASED ON KNOWING WHAT’S INSIDE THE MODULE

/**

* Created by Michael

*

* Strings are immutable in JAVA. Each time a string

concatentation occurs, a

* new String Object is created. After, memory is allocated for

the new string:

* the concatenation of <em>result</em> and the string being

added to <em>result.</em>

* The old string is sent to the trash, which eventually will be

collected.

*/

import java.lang.String;

public class anticipation {

public static void main(String[] args) {

int n = 1000000;

String result = "";

String phrase = "anticipation is making me wait";

long startTime = System.currentTimeMillis ();

long estimatedTime = 0;

long previousTime = 0;

double percentChange = 0.0;

for (int i = 0; i < n; i++) {

result += phrase;

IF WE CANNOT KNOW THE CODE INSIDE, AGAINST WHAT DO WE WRITE TESTS?

Specifications

TESTING TACTICS

Tests based on spec

Test covers as much specifiedbehavior as possible

Functional/

Black Box

Structural/

White Box

Tests based on code

Test covers as much implemented behavior as possible

WHY DO FUNCTIONAL TESTING?

Tests based on spec


Functional/

Black Box

1. Program code not necessary

2. Early functional test design has benefits

1. Reveals spec problems

2. Assesses testability

3. Gives additional explanation of spec

4. May even serve as spec, as in XP

WHY DO FUNCTIONAL TESTING?

Tests based on spec


Functional/

Black Box

Best for missing logic defects where some program logic was simply forgotten

Applies at all levels of testing

1. unit tests

2. integration tests

3. system tests

4. regression tests

HOW TO TEST?

RANDOM TESTING

Pick possible inputs uniformly, but then this treats all inputs as equally valuable

Avoids designer bias

A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person)

And removes any perception that the tests are rigged to pass (or fail)

Angle

Force

Two Inputs

Can you test all combinations

of angle and force (trajectories)?

INFINITE MONKEY THEOREM

If you put enough monkeys in front of typewriters and give them enough time, you eventually will get Shakespeare

Youtube

Angle

Force

232 = 4,294,967,296

different values

232 = 4,294,967,296

different values

TOTAL NUMBER OF TRIALS = 232 * 232 (264)

264 = 18,446,744,073,709,551,616

1B monkeys would still take ~35 Years

THE ALTERNATIVE: COMPUTER SCIENCE APPROACHES

Computer scientists are smart, and they can systematically test and analyze programs.

Functional

specification

Independently

testable feature

Representative

valuesValues

Test case

specifications

identify derive

identify

derive

Test Case

generate

SYSTEMATIC FUNCTIONAL TESTING

WE ARE HUNTING BUGS

Reliability estimation requires unbiased samples for valid statistics.

To estimate the proportion of bugs to lines of code (or whatever code model), sample randomly

But that’s not our goal!

Our goal is to find bugs and remove them from the code

We need look systematically (and perhaps non-uniformly) for bugs

Unless there are a lot of bugs in the code, a random sample will not be effective at finding any of them.

We need to use everything we know about code, what is it trying to achieve?

Adapted from Pezze and Young, Software Testing and Analysis, Ch. 10.

Reliability estimation requires unbiased samples for valid statistics.

To estimate the proportion of bugs to lines of code (or whatever code model), sample randomly

But that’s not our goal!

OTHER WAYS TO FIND THAT NEEDLE

Borrowed from http://hpperformancecenter.blogspot.com/, an

HP blog on performance testing

https://www.nbcbayarea.com/news/local/Hay-Destroyed-

Fire-Contra-Costa-County-Brentwood-Antioch-

432664033.html

http://hpperformancecenter.blogspot.com/

Functional

specification

Independently

testable feature

Representative

valuesValues

Test case

specifications

identify derive

identify

derive

Test Case

generate

SYSTEMATIC FUNCTIONAL TESTING

Functional

specification

Independently

testable feature

identify

TESTABLE FEATURES

Decompose system into independently testable features (ITF)

An ITF need not correspond to units or subsystems of the software

For system testing, ITFs are exposed through user interfaces or APIs

WHAT ARE THE INDEPENDENTLY TESTABLE FEATURES?

class Roots {

// Solve ax2 + bx + c = 0

public roots(double a, double b, double c)

{ … }

// Result: values for x

double root_one, root_two;}

EVERY FUNCTION IS AN INDEPENDENTLY TESTABLE FEATURE

Consider a multi-function calculator.

What are the independently testable features?

Independently

testable feature

Representative

valuesValues

identify derive

DECIDING THE SET OF USEFUL VALUES

Try to select inputs that are especially valuable

Usually by choosing representatives of equivalence classes that are apt to fail often or not at all

LIKE FINDING NEEDLES IN A HAYSTACK

To find bugs systematically, we need to find out what makes certain inputs or behaviors special

THE PARTITION PRINCIPLE

Use knowledge to choose samples more likely to include trouble-prone regions of the input space.

Partition testing: separates the input space into classes

The Ideal Situation

Each fault leads to failures that are dense in some class of inputs.

Sampling each class in the quasi-partition selects at least one input that leads to a failure, revealing the fault

Seldom guaranteed; we depend on experience-based heuristics

Adapted from Pezze and Young

Failure (valuable test case)

No failure

SYSTEMATIC PARTITION TESTINGFailures are sparse in

some regions of

possible inputs ...

... but dense in other

If we systematically test some cases

from each part, we will include the

dense parts

Functional testing is one way of drawing lines to isolate regions with likely failures

Th

e s

pa

ce o

f p

oss

ible

inp

ut

va

lue

s(t

he

ha

yst

ack

)

EQUIVALENCE PARTITIONING

Input condition Equivalence classes

rangeone valid, two invalid

(larger and smaller)

specific valueone valid, two invalid

(larger and smaller)

member of a set one valid, one invalid

boolean one valid, one invalid

Defining equivalence classes comes from input conditions in the spec.

Each input condition induces an equivalence class – valid and invalid inputs.


-1 0 2 3-2-3 1

valid invalidinvalid

Range


-1 0 2 3-2-3 1

valid invalidinvalid

Specific Value


valid

invalid

Set

BOUNDARY ANALYSIS –FINDING ERROR AT THE EDGES

Test Possible test case

at lower range (valid and invalid)

at higher range (valid and invalid)

at center

REPRESENTATIVE VALUES EXAMPLE: ZIP CODE

Input: 5-digit ZIP code

Output: list of cities

EXAMPLE: ZIP CODE



What are representative values to test?

1. With 0 cities as output

(0 is boundary value)

2. With 1 city as output

3. With many cities as output

EXAMPLE: ZIP CODE




4. 5 characters

5. 1–4 characters(4 is boundary value)

6. 6 characters(6 is boundary value)

7. Empty input

EXAMPLE: ZIP CODE




7. Very long input

8. No digits

9. Non-character data

“SPECIAL” ZIP CODES

How about a ZIP code that reads

12345‘; DROP TABLE orders; SELECT * FROM zipcodes WHERE ‘zip’ = ‘

A ZIP code with 65536 characters…

Independently

testable feature

Representative

valuesValues

Test case

specifications

identify derive

derive

USING MODELS TO DERIVE VALUES

Use a formal model that specifies software behavior

Models typically come as

▪ Finite State Machines

▪ Decision Structures

identify

STATE-BASED TESTING

Protocols (e.g., network communication)

GUIs (sequences of interactions)

Objects (methods and states)

EXAMPLE: PRODUCT MAINTENANCE

FSM

EXAMPLE: PRODUCT MAINTENANCE

FSM

0

1 2 3

4 5 6

7 8

9

OPTIONS FOR COVERING FSM’S

State Coverage

Every state should be visited by at least one test case.

A minimum testing criterion

0

1 2 3

4 5 6

7 8

9

OPTIONS FOR COVERING FSM’S

Transition coverage

Every edge should be traversed by at least one test case.

Typically, a good coverage criterion to target.

0

1 2 3

4 5 6

7 8

9

0

1 23

4 5 6

7 8

9

EXAMPLE

Each test case covers a set of transitions

Here, there are five needed to cover each transition once

one color = one test case

PATH COVERAGE

Tests cover every path

Not feasible in practice

Cycles create infinite paths

Acyclic graphs can still have an exponential number of paths

0

1 2 3

4 5 6

7 8

9

PATH COVERAGE

Tests cover every path

But there are approximations

Single State Path Coverage

Single Transition Path Coverage

Boundary Interior Loop Coverage

0

1 2 3

4 5 6

7 8

9

DECISION TABLES

Some specifications define decision tables, decision trees, or flow charts.

Type of PurchaserEducational

Purchaser

Individual

Purchaser

Education account T T F F F F F F

Current purchase >

Threshold 1– – F F T T – –

Current purchase >

Threshold 2– – – – F F T T

Special price <

scheduled priceF T F T – – – –

Special price < Tier 1 – – – – F T – –

Special price < Tier 2 – – – – – – F T

Outcome Edu discountSpecial

priceNo discount

Special

priceTier 1

discount

Special

priceTier 2

discount

Special

Price

PARETO’S LAW

Approximately 80% of defectscome from 20% of modules

LEARNING FROM THE PAST

Functional

specification

Independently

testable feature

Representative

valuesValues

Test case

specifications

identify derive

identify

derive

Test Case

generate

DERIVING TEST SPEC’S

COMBINATORIAL TESTING

Windows

Linux

OracleMySQL

Apache

IIS

OSServer

Database

MANAGING THE COMBINATIONS

1. Eliminate invalid combinations

2. Cover all pairs of combinations

3. Combinations typically generated automatically and – hopefully –tested automatically, too

Windows

Linux

OracleMySQL

Apache

IIS

OSServer

Database

GENERATING TEST CASES FROM TEST SPEC’S

62

Test case

specificationsTest Case

generate

GENERATING TEST CASES FROM TEST SPEC’S

Generation creates concrete, executable test cases from test case specifications

Test design often yields test case specifications, rather than concrete data

E.g. a large positive number, not 3219765

E.g: “a sorted sequence, length > 2”, not {Curly, Larry, Moe, Shemp}

Other details for execution may be omitted

63

AUTOMATING TEST EXECUTION

Designing test cases and test suites is takes skill, judgment, and intuition.

Executing test cases should be automatic and mechanical.

SCAFFOLDING

Code produced to support development activities (especially

testing)

Not part of the “product” as seen by the end user

May be temporary (like scaffolding in construction of buildings

Includes

Test harnesses, drivers, and stubs

Adapted from Prezze and Young© Glen Laker using creative commons license

TEST COMPONENTRY

Test driver

Drives program under test through test cases

Test stubs

Substitutes for called functions/methods/objects

Test harness

Substitutes for other parts of the deployed environment

RUNNING A TEST

A test case…

▪ sets up an environment for the test

▪ tests the unit

▪ tears down the environment again

Organize tests into suites

RECAP: SYSTEMATIC FUNCTIONAL TESTING

Functional

specification

Independently

testable feature

Representative

valuesValues

Test case

specifications

identify derive

identify

derive

Test Case

generate

functional testing - northeastern university

Documents