black box testing - adam's note · pdf fileblack box testing attempts to find errors in...

Black Box Testing

Adam Hendra Brata

2

Agenda

Testing Strategies

Black Box Testing– Introduction

– Techniques Equivalence Partitioning

Boundary Value Analysis

3

Testing Strategies

4

Testing Strategies

White Box

– White Box Testing is a

software testing method in

which the internal structure/

design/ implementation of

the item being tested is

known to the tester.

Black Box

– Black Box Testing is a

software testing method in

which the internal structure/

design/ implementation of

the item being tested is

NOT known to the tester

5

Test-to-pass and Test-to-fail

Test-to-pass

– assures that the software minimally works

– does not push the capabilities of the software

– applies simple and straightforward test cases

– does not try to “break” the program

Test-to-fail

– designing and running test cases with the sole

purpose of breaking the software

– strategically chosen test cases to probe for common

weaknesses in the software

6

Black Box TestingIntroduction

7

Black Box Testing

Testing software against a specification of its external

behavior without knowledge of internal

implementation (source code) details

– Can be applied to software “units” (e.g., classes) or to entire

programs

– External behavior is defined in API docs, Functional specs,

Requirements specs, etc.

Because black box testing purposely disregards the

program's control structure, attention is focused primarily

on the information domain (i.e., data that goes in, data

that comes out)

The Goal: Derive sets of input conditions (test cases) that

fully exercise the external functionality

8

Black Box Testing

interface

Black Box

(Component or System)outputinput

operation

9

Black Box Testing

Black box testing also known as specification-

based testing

Black box testing refer to test activities using

specification-based testing methods and criteria to

discover program errors based on program

requirements and product specifications

The major testing focuses:

– specification-based function errors

– specification-based component/system behavior errors

– specification-based performance errors

– user-oriented usage errors

– black box interface errors

10

Black Box Testing

Black box testing attempts to find errors in the

external behavior of the code in the following

categories :

1. incorrect or missing functionality

2. interface errors

3. errors in data structures used by interfaces

4. behavior or performance errors

5. initialization and termination errors

Through this testing, we can determine if the

functions appear to work according to specifications

11

Black Box Testing Problems

So exactly, what’s the problem ?

12


Domain is all possible inputs to the function

Range is all possible outputs of the function

“there are so many combination of inputs and various outputs,

so what should I do ?”

13


“..and there are millions of possible test cases to run, what do I do ?”

14

Black Box Testing TechniquesEquivalence Partitioning

15

Equivalence Partitioning

Typically the universe of all possible test cases is so

large that you cannot try them all

You have to select a relatively small number of test

cases to actually run

Which test cases should you choose?

Equivalence partitioning helps answer this question

Goal: Reduce number of test cases by equivalence

partitioning:

– Divide input conditions into equivalence classes

– Choose test cases for each equivalence class. (Example: If

an object is supposed to accept a negative number, testing

one negative number is enough)

16


17


Partition the test cases into "equivalence classes"

Each equivalence class contains a set of

"equivalent" test cases

Two test cases are considered to be equivalent if we

expect the program to process them both in the

same way (i.e., follow the same path through the

code)

If you expect the program to process two test cases

in the same way, only test one of them, thus

reducing the number of test cases you have to run

18


Steps in equivalence testing:

– identify classes of inputs with same behavior

– test on at least one member of each equivalence class

– assume behavior will be same for all members of

class

Criteria for selecting equivalence classes:

– coverage : every input is in one class

– disjointedness : no input in more than one class

– representation : if error with 1 member of class, will

occur with all

19


First-level partitioning : Valid vs. Invalid test cases

Valid Invalid

20


Partition valid and invalid test cases into equivalence

classes

21


Create a test case for at least one value from each

equivalence class

22


When designing test cases, you may use different

definitions of “equivalence”, each of which will

partition the test case space differently

Example: int Add(n1, n2, n3, …)

– Equivalence Definition 1: partition test cases by the number

of inputs (1, 2, 3, etc.)

– Equivalence Definition 2: partition test cases by the number

signs they contain (positive, negative, both)

– Equivalence Definition 3: partition test cases by the

magnitude of operands (large numbers, small numbers,

both)

– Etc.

23


When designing test cases, you may use different

definitions of “equivalence”, each of which will

partition the test case space differently

Example: string Fetch(URL)

– Equivalence Definition 1: partition test cases by URL

protocol (“http”, “https”, “ftp”, “file”, etc.)

– Equivalence Definition 2: partition test cases by type of file

being retrieved (HTML, GIF, JPEG, Plain Text, etc.)

– Equivalence Definition 3: partition test cases by length of

URL (very short, short, medium, long, very long, etc.)

– Etc.

24

Equivalence Partitioning : Oracle

If an oracle is available, the test values in each

equivalence class can be randomly generated.

This is more useful than always testing the same

static values.

– Oracle: something that can tell you whether a test passed or

failed

– An oracle is a mechanism for determining whether the

program has passed or failed a test.

Test multiple values in each equivalence class.

Often you’re not sure if you have defined the

equivalence classes correctly or completely, and

testing multiple values in each class is more

thorough than relying on a single value.

25

Equivalence Partitioning Example

26


Let's consider the behavior of tickets in the Flight

reservation application, while booking a new flight.

Ticket values 1 to 10 are considered valid & ticket is

booked, while value 11 to 99 are considered invalid

for reservation

Here is the test condition

– Any Number greater than 10 entered in the reservation

column (let say 11) is considered invalid.

– Any Number less than 1 that is 0 or below, then it is

considered invalid.

– Numbers 1 to 10 are considered valid

– Any 3 Digit Number say -100 is invalid.

27


We cannot test all the possible values because if

done, the number of test cases will be more than

100.

– To address this problem, we use equivalence

partitioning hypothesis where we divide the possible

values of tickets into groups or sets as shown below

where the system behavior can be considered the

same

28


The divided sets are called Equivalence Partitions or

Equivalence Classes. Then we pick only one value

from each partition for testing.

– The hypothesis behind this technique is that if one

condition/value in a partition passes all others will also

pass. Likewise, if one condition in a partition fails, all

other conditions in that partition will fail.

29


30


Input Valid Equivalence Classes Invalid Equivalence Classes

A integer N such that:

-99 <= N <= 99 ? ?Phone Number

Area code: [200, 999]

Prefix: (200, 999]

Suffix: Any 4 digits

? ?

Another cases…..

31




-99 <= N <= 99

[-99, -10]

[-9, -1]

0

[1, 9]

[10, 99]

< -99

> 99

Malformed numbers

{12-, 1-2-3, …}

Non-numeric strings

{junk, 1E2, $13}

Empty value

Phone Number

Area code: [200, 999]

Prefix: (200, 999]


? ?

32




-99 <= N <= 99

[-99, -10]

[-9, -1]

0

[1, 9]

[10, 99]

< -99

> 99

Malformed numbers

{12-, 1-2-3, …}

Non-numeric strings

{junk, 1E2, $13}

Empty value

Phone Number

Area code: [200, 999]

Prefix: (200, 999]


555-5555

(555)555-5555

555-555-5555

200 <= Area code <= 999

200 < Prefix <= 999

Invalid format 5555555,

(555)(555)5555, etc.

Area code < 200 or > 999

Area code with non-numeric

characters

Similar for Prefix and Suffix

33

Black Box Testing TechniquesBoundary Value Analysis

34


Boundary value analysis

– Testing conditions on bounds between classes of inputs

Why is it useful to test near boundaries?

– likely source of programmer errors (< vs. <=, etc.)

– language has many ways to implement boundary checking

– requirement specs may be fuzzy about behavior on

boundaries

– often uncovers internal hidden limits in code

– example: array list must resize its internal array when it fills

capacity

35


When choosing values from an equivalence class to test,

use the values that are most likely to cause the program

to fail

Errors tend to occur at the boundaries of equivalence

classes rather than at the "center"

– If (200 < areaCode && areaCode < 999) {//valid area code

– If (200 <= areaCode && areaCode <= 999) {//valid area code}

– Testing area codes 200 and 999 would catch this error, but a

center value like 770 would not

In addition to testing center values, we should also test

boundary values

– Right on a boundary

– Very close to a boundary on either side

36


Testing boundary conditions of eq. classes is more

effective i.e. values directly on, above, and beneath

edges of eq. classes

Choose input boundary values as tests in input eq.

classes instead of, or additional to arbitrary values

Choose also inputs that invoke output boundary values

(values on the boundary of output classes)

Example strategy as extension of equivalence

partitioning:

– choose one (n) arbitrary value in each eq. class

– choose values exactly on lower and upper boundaries of eq.

class

– choose values immediately below and above each

boundary (if applicable)

37


Create test cases to test boundaries of equivalence

classes

38

Boundary Value Analysis Example

39


In our earlier example instead of checking, one value

for each partition you will check the values at the

partitions like 0, 1, 10, 11 and so on. As you may

observe, you test values at both valid and invalid

boundaries.

– Boundary Value Analysis is also called range checking

40


Input Boundary Cases

A number N such that:

-99 <= N <= 99 ?Phone Number

Area code: [200, 999]

Prefix: (200, 999]


?

Another cases…..

41


Input Boundary Cases

A number N such that:

-99 <= N <= 99

-100, -99, -98

-10, -9

-1, 0, 1

9, 10

98, 99, 100

Phone Number

Area code: [200, 999]

Prefix: (200, 999]


Area code: 199, 200, 201

Area code: 998, 999, 1000

Prefix: 200, 199, 198

Prefix: 998, 999, 1000

Suffix: 3 digits, 5 digits

42


Imagine we are testing a Date class with a

daysInMonth(month, year)method

– What are some conditions and boundary tests for this

method?

Possible answers

– check for leap years (every 4th yr, no 100s, yes 400s)

– try years such as: even 100s, 101s, 4s, 5s

– try months such as: June, July, Feb, invalid values

43


Another Example of Equivalence Partitioning and

Boundary Value Analysis– http://www.guru99.com/equivalence-partitioning-boundary-value-

analysis.html

http://www.guru99.com/equivalence-partitioning-boundary-value-analysis.html

44

Thank You

black box testing - adam's note · pdf fileblack box testing attempts to find errors in...

Documents