efficient techniques for software testing

36
Efficient Techniques for Software Testing Jeff Lei CSE@UTA April 13, 2007

Upload: khalil

Post on 28-Jan-2016

69 views

Category:

Documents


0 download

DESCRIPTION

Efficient Techniques for Software Testing. Jeff Lei CSE@UTA April 13, 2007. Outline. Software Testing Concurrency Testing Testing Concurrent Programs Testing Synchronization Components T-Way Interaction Testing Other Work Model Checking, Protocol Validation, Pervasive Computing. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Efficient Techniques for Software Testing

Efficient Techniques for Software Testing

Jeff Lei

CSE@UTA

April 13, 2007

Page 2: Efficient Techniques for Software Testing

Outline

• Software Testing• Concurrency Testing

– Testing Concurrent Programs– Testing Synchronization Components

• T-Way Interaction Testing• Other Work

– Model Checking, Protocol Validation, Pervasive Computing

Page 3: Efficient Techniques for Software Testing

Software Engineering

• Software has become pervasive in modern society– Directly contributes to quality of life– Malfunctions cost billions of dollars every year,

and have severe consequences in a safe-critical environment

• All about building better software in better ways, especially for large-scale development– Requirements, design, coding, testing,

maintenance, configuration, documentation, deployment, and etc.

Page 4: Efficient Techniques for Software Testing

Software Testing

• A dynamic approach to detecting software faults– Alternatively, static analysis can be performed,

which is however often intractable

• Involves sampling the input space, running the test object, and observing the runtime behavior

• Perhaps the most widely used approach in practice– Labor intensive, and often consumes more than

50% of development cost

Page 5: Efficient Techniques for Software Testing

Research Problems

• How to detect as many faults as possible spend with as little effort as possible?– How to generate a good set of test inputs?– How to perform actual test executions?– How to evaluate test outcomes?

Page 6: Efficient Techniques for Software Testing

Outline

• Software Testing• Concurrency Testing

– Testing Concurrent Programs– Testing Synchronization Components

• T-Way Interaction Testing• Other Work

– Model Checking, Protocol Validation

Page 7: Efficient Techniques for Software Testing

Concurrent Programs

• Consists of multiple threads of execution that proceed in parallel– multithreaded programs, multi-process (or

distributed) programs

• Advantages: – Better resource utilization, increased computation

efficiency, and providing simplified solutions for many problem domains

• Notoriously difficult to test due to their non-deterministic behavior

Page 8: Efficient Techniques for Software Testing

What Is Unique

• Non-determinism– Multiple executions with the same data input might

display different behaviors

• Synchronization– How to ensure that concurrent events are

exercised in an expected order?

• Communication– How to exchange information between different

threads?

Page 9: Efficient Techniques for Software Testing

Concurrency Testing Strategies

• Non-deterministic testing - execute the same program with the same input for multiple times:– easy, but inefficient, and some errors cannot be detected

• Deterministic testing - select a set of test sequences and then force them to be exercised:– can detect more subtle errors, but requires additional effort

for test sequence selection and runtime control

• State exploration – explore the state space in a systematic manner– suffers the well-known state explosion problem– state representation is difficult for programs written in a full-

fledged programming language

Page 10: Efficient Techniques for Software Testing

Reachability Testing

• Combines non-deterministic and deterministic testing– Generating test sequences dynamically, without

constructing any static model– Dealing with partial orders directly, and thus no

redundant interleavings!

• Can also be considered as a state exploration technique– Exercising every possible synchronization

sequence of a program with a given input

Page 11: Efficient Techniques for Software Testing

The Framework

1. Execute a program P with a given input non-deterministically to collect a trace Q

2. Identify the race conditions in Q and compute its race variants

3. For each variant, conduct a prefix-based test run to collect a new trace Q’

4. Repeat 2, 3, and 4 for each newly collected trace Q’.

Page 12: Efficient Techniques for Software Testing

Example

Page 13: Efficient Techniques for Software Testing

Main Results

• Developed three reachability testing algorithms – Exhaustive RT, Combinatorial RT, and Random

RT

• Built a reachability testing tool called RichTest– Portable implementation, applicable to most

common synchronization constructs

• Conducted an empirical evaluation– Performed significantly better than VeriSoft (from

Bell Labs)

• Publications: STVR07a, TSE06, ISSRE05

Page 14: Efficient Techniques for Software Testing

Dinning Philosophers

Page 15: Efficient Techniques for Software Testing

Outline

• Software Testing• Concurrency Testing

– Testing Concurrent Programs– Testing Synchronization Components

• T-Way Interaction Testing• Other Work

– Model Checking, Protocol Validation

Page 16: Efficient Techniques for Software Testing

Synchronization Components

• Synchronization logic is often encapsulated into one or more components

• Synchronization components need to be tested separately– Obtain necessary confidence prior to integration– helps to reduce complexity by incremental testing

• A different problem than testing complete programs– The ways in which a component could be used is

typically unknown

Page 17: Efficient Techniques for Software Testing

Java Monitor

• A high-level synchronization construct that supports data encapsulation and information hiding– A Java class that defines one or more

synchronized methods

• Mutual exclusion automatically enforced by the Java runtime

• Conditional synchronization implemented by the programmer, using wait and notify operations

Page 18: Efficient Techniques for Software Testing

Java Monitor Semantics

• Consists of three components: an entry queue, a critical section, a waiting queue

• A thread must enter the CS before it executes a synchronized method; it has to wait in the entry queue if the CS is not empty

• The wait and notify operations can only be executed inside the CS– wait: puts the current thread in the waiting queue– notify: awakens one thread in the waiting queue

Page 19: Efficient Techniques for Software Testing

Example

class BoundedBuffer { private int fullSlots=0; private int capacity = 0; private int[] buffer = null; private int in = 0, out = 0; public BoundedBuffer(int bufferCapacity) {1. capacity = bufferCapacity; 2. buffer = new int[capacity]; } public synchronized void deposit (int value) {3. while (fullSlots == capacity) {4. try { wait(); } catch (InterruptedException ex) {} } 5. buffer[in] = value;6. in = (in + 1) % capacity;7. if (fullSlots++ == 0) {8. notifyAll(); } } public synchronized int withdraw () {9. int value = 0;10. while (fullSlots == 0) {11. try { wait(); } catch (InterruptedException ex) {} } 12. value = buffer[out];13. out = (out + 1) % capacity;14. if (fullSlots-- == capacity) {15. notifyAll(); }16. return value; } }

Page 20: Efficient Techniques for Software Testing

Graphic View

Critical Section

Entry Queue

W

Condition Queue

D

Assume that the buffer is initially empty.

Page 21: Efficient Techniques for Software Testing

Research Questions

• What is unique to testing Java monitors?– Used to synchronize, and thus typically accessed

by, multiple threads simultaneously

• Can existing work on OO testing apply?– OO testing methods typically assume a single-

threaded test driver

• How to effectively test Java monitors?

Page 22: Efficient Techniques for Software Testing

Technical Challenges

• To simulate possible scenarios in which a Java monitor can be accessed, multiple threads need to be created– Some faults can only be detected by a certain

minimum number of threads. But how many?– Multiple threads can display non-deterministic

behavior. How to describe and control thread behavior?

Page 23: Efficient Techniques for Software Testing

A State Exploration-Based Approach

• Explores the state space of a monitor as an open system– Threads introduced on-the-fly, and as needed, to

simulate race conditions– State abstraction used to ensure termination and

to control state explosion– Thread behavior is controlled at finer granularity

than method calls

Page 24: Efficient Techniques for Software Testing

Main Results

• Developed an algorithm that implements the state exploration-based approach

• Built a prototype tool, called MonitorExplorer, and conducted an empirical evaluation– Empirical results suggest that this approach can

be very effective

• Ongoing work: Extend to general monitors, and general synchronization components

• Publication: ISSRE06

Page 25: Efficient Techniques for Software Testing

Outline

• Software Testing• Concurrency Testing

– Testing Concurrent Programs– Testing Synchronization Components

• T-Way Interaction Testing• Other Work

– Model Checking, Protocol Validation, Pervasive Computing

Page 26: Efficient Techniques for Software Testing

Combinatorial Testing

• Creates tests by combining different parameter values– The input space of a test object is represented as

a set of parameters and their values

• Advantages: light specification, test input generation can be fully automated, requires no access to source code

• How to deal with the combinatorial explosion problem?

Page 27: Efficient Techniques for Software Testing

T-Way Interaction Testing

• Motivation: Not every parameter contributes to every fault– Many faults can be exposed by interactions

involving a few parameters

• Given any t parameters, every combination of values of these parameters be covered by at least one test– Allows faults triggered by at most t parameters to

be detected

• Makes an excellent trade-off between test effort and test coverage

Page 28: Efficient Techniques for Software Testing

T-Way Testing - Example

P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 11 0 01 0 11 1 01 1 1

P1 P2 P3 0 0 0 0 1 1 1 0 11 1 0

Three parameters, each with values 0 and 1

2-way testing

exhaustive testing

Page 29: Efficient Techniques for Software Testing

State of the Art

• Greedy construction: Each step tries to cover as many combinations as possible– Involves explicit enumeration of all possible

combinations

• Algebraic Construction: test sets are constructed using pre-defined rules

• Most approaches focus on 2-way (or pairwise) testing

Page 30: Efficient Techniques for Software Testing

Strategy In-Parameter-Order

• Input: A set of parameters and values; Output: a t-way test set

• Construct a t-way test set for the first t parameters

• Extend the test set to cover each of the remaining parameters one by one– Horizontal growth - extends each existing test by

adding one value for the new parameter– Vertical growth – adds new tests, if needed, to

make the test set complete

Page 31: Efficient Techniques for Software Testing

Example

(a)

P1 P2 P30 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1

(b)

P1 P2 P3 P40 0 0 0 0 0 1 10 1 0 20 1 1 01 0 0 1 1 0 1 2 1 1 0 01 1 1 1

P1 P2 P3 P40 0 0 0 0 0 1 10 1 0 20 1 1 01 0 0 1 1 0 1 2 1 1 0 01 1 1 11 0 1 0 0 1 0 1 0 0 1 2 1 1 0 2 * 0 0 2 * 1 1 2

(c)

Horizontal growth Vertical growth

• Four parameters: P1, P2, P3, and P4• P1, P2, and P3 have 2 values• P4 has 3 values

3-way test set

Page 32: Efficient Techniques for Software Testing

Main Results

• Developed two algorithms, IPOG and IPOG-D, for general t-way testing– Has the lowest algorithmic complexity among the

existing algorithms

• Built a t-way testing tool called FireEye, and conducted an empirical evaluation– Performed significantly better than existing tools,

e.g., ITCH, TConfig, Jenny, TestVector

• Publications: TSE02, ECBS07, STVR07b

Page 33: Efficient Techniques for Software Testing

TCAS Example

t-way

FireEye ITCH Jenny TConfig TVG

Size Time Size Time Size Time Size Time Size Time

2 100 0.8 120 0.73 108 0.001 108 >1 hour 101 2.75

3 400 0.36 2388 1020 413 0.71 472 >12 hour 9158 3.07

4 1361 3.05 1484 5400 1536 3.54 1476 >21 hour 64696 127

5 4219 18.41 NA >1 day 4580 43.54 NA >1 day 313056 1549

6 10919 65.03 NA >1 day 11625 470 NA >1 day 1070048 12600

TCAS: Seven 2-value parameters, two 3-value parameters, one 4-value parameter, two 10-value parameters

Page 34: Efficient Techniques for Software Testing

Outline

• Software Testing• Concurrency Testing

– Testing Concurrent Programs– Testing Synchronization Components

• T-Way Interaction Testing• Other Work

– Model Checking, Protocol Validation, Pervasive Computing

Page 35: Efficient Techniques for Software Testing

Other Work

• Model checking: How to deal with the state explosion problem?– True concurrency model vs interleaving-based concurrency

model– Publication: FM05

• Protocol validation: A more specific problem than model checking– Allows multiple threads to proceed simulatenously during

state exploration– Publication: CJ06

• Pervasive computing– Context-awareness: How to derive high level events from a

sequence of low level events?– Power-constraints: How to reduce the power consumption

on small mobile devices?

Page 36: Efficient Techniques for Software Testing

References

• [STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for Concurrent Programs. Accepted for publication in Software Testing, Verification, and Reliability, 2007.

• [STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic strategies for multi-way software testing. Software Testing, Verification, and Reliability, pending review.

• [ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi-way software testing. Int’l Conf. on Engineering of Computer-Based Systems, 2007.

• [ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration-Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering, 256-265, 2006.

• [CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking-Based Approach to Protocol Validation, The Computer Journal, 49:541-553, 2006.

• [TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs, IEEE Transactions On Software Engineering, 32(6):382-403, 2006.

• [FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication Protocols, Proc. 13th Intl. Symp. on Formal Methods, pp. 334-349, 2005.

• [TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing", IEEE Transactions on Software Engineering, Vol. 28, No. 1, pp. 109-111., Jan. 2002.