tao xie xie@csc.ncsu.edu north carolina state university supported by cacc/nsa related projects...

Regression Testing and Programming

Tao Xiexie@csc.ncsu.edu

North Carolina State University

Supported by CACC/NSARelated projects supported in part by ARO, NSF, SOSI

Why Automate Testing?

Software testing is important Software errors cost the U.S. economy about $59.5

billion each year (0.6% of the GDP) [NIST 02] Improving testing infrastructure could save 1/3 cost

[NIST 02] Software testing is costly

Account for even half the total cost of software development [Beizer 90]

Automated testing reduces manual testing effort Test execution: JUnit, NUnit, xUnit, etc. Test generation: Pex, AgitarOne, Parasoft Jtest, etc. Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.

Example – Credit Card# Validator

Example – Credit Card# Validator

Credit Card# Validator – Behind the Scene

http://www.beachnet.com/~hstiles/cardtype.html

LUHN Formula (Mod 10)

Automatic Generation of Valid Credit Card#

MASTERCARD = 0; VISA = 1; DISCOVER = 2; AMEX = 3;

http://research.microsoft.com/pex

Target Problem

Versioni

Version i + 1

Security Functionality Performance …

Testing/retesting is expensive

fixing faultsimproving performance/designadding new features…

Project Goal

Versioni

Versioni + 1

Our Tool

Test inputs/conditions causing different

outputs

Programming Tutoring Tool

Our Tool

Test inputs/conditions causing different

outputs

Instructor solution

Student solution==

?

Validation of security patch Does the patch fix ALL attacks of the same

type (more types) beyond given specific attacks?

Attack generation based on patch [Song et al. S&P08] Can we exploit patches to other systems?

Revalidation of medical device software versions at FDA There no complete software package available

Generation of tests/regression tests DURING code development

…

Subproblems

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

Loops Fitnex [Xie et al. DSN 09]

Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]

Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]

Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri et al. AST 09]

Opportunities Regression testing [Taneja et al. ICSE 09-nier] Developer guidance (cooperative developer testing)

Challenges of DSE

Loops Fitnex [Xie et al. DSN 09]

Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]

Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]

Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri et al. AST 09]

Applications Test network app at Army division@Fort Hood, Texas Test DB app of hand-held medical assistant device at FDA

Army Telemedicine and Advanced Technology Research Center (TATRC)

NCSU Tech Transfer

Pex on MSDN DevLabs Power tool for Visual Studio 2010

Download counts (20 months)(Feb. 2008 - Oct. 2009 )

Academic: 17,366 Devlabs: 13,022 Total: 30,388

Summary

Developers make changes to a software system Bug fixes, refactorings, addition of new

features…

Developers need to make sure that the changes Introduce intended effect not introduce unintended side effect

Developed upon Dynamic Symbolic Execution, a practical, powerful recent technique in academia/industry

Good impact to software industry and agencies/defense mission

Thank you

http://pexase.codeplex.com/https://sites.google.com/site/asergrp/http://research.microsoft.com/pex

Supported by CACC/NSARelated projects supported in part by

ARO, NSF, SOSI