tao xie [email protected] north carolina state university supported by cacc/nsa related projects...
TRANSCRIPT
Regression Testing and Programming
North Carolina State University
Supported by CACC/NSARelated projects supported in part by ARO, NSF, SOSI
Why Automate Testing?
Software testing is important Software errors cost the U.S. economy about $59.5
billion each year (0.6% of the GDP) [NIST 02] Improving testing infrastructure could save 1/3 cost
[NIST 02] Software testing is costly
Account for even half the total cost of software development [Beizer 90]
Automated testing reduces manual testing effort Test execution: JUnit, NUnit, xUnit, etc. Test generation: Pex, AgitarOne, Parasoft Jtest, etc. Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.
Example – Credit Card# Validator
Example – Credit Card# Validator
Credit Card# Validator – Behind the Scene
http://www.beachnet.com/~hstiles/cardtype.html
LUHN Formula (Mod 10)
Automatic Generation of Valid Credit Card#
MASTERCARD = 0; VISA = 1; DISCOVER = 2; AMEX = 3;
http://research.microsoft.com/pex
Target Problem
Versioni
Version i + 1
Security Functionality Performance …
Testing/retesting is expensive
fixing faultsimproving performance/designadding new features…
Project Goal
Versioni
Versioni + 1
Our Tool
Test inputs/conditions causing different
outputs
Programming Tutoring Tool
Our Tool
Test inputs/conditions causing different
outputs
Instructor solution
Student solution==
?
Validation of security patch Does the patch fix ALL attacks of the same
type (more types) beyond given specific attacks?
Attack generation based on patch [Song et al. S&P08] Can we exploit patches to other systems?
Revalidation of medical device software versions at FDA There no complete software package available
Generation of tests/regression tests DURING code development
…
Subproblems
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
Loops Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri et al. AST 09]
Opportunities Regression testing [Taneja et al. ICSE 09-nier] Developer guidance (cooperative developer testing)
Challenges of DSE
Loops Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri et al. AST 09]
Applications Test network app at Army division@Fort Hood, Texas Test DB app of hand-held medical assistant device at FDA
Army Telemedicine and Advanced Technology Research Center (TATRC)
NCSU Tech Transfer
Pex on MSDN DevLabs Power tool for Visual Studio 2010
Download counts (20 months)(Feb. 2008 - Oct. 2009 )
Academic: 17,366 Devlabs: 13,022 Total: 30,388
Summary
Developers make changes to a software system Bug fixes, refactorings, addition of new
features…
Developers need to make sure that the changes Introduce intended effect not introduce unintended side effect
Developed upon Dynamic Symbolic Execution, a practical, powerful recent technique in academia/industry
Good impact to software industry and agencies/defense mission
Thank you
http://pexase.codeplex.com/https://sites.google.com/site/asergrp/http://research.microsoft.com/pex
Supported by CACC/NSARelated projects supported in part by
ARO, NSF, SOSI